Sesame Street, Letter Blocks, and Augmented Reality: Increasing Engagement and Learning

3 Sesame Street scenes – a: Ernie and Bert talking on a banana phone; b: Grover singing; c: 12 Ladybugs arranged around a number 12. — Figure 1. a: Ernie and Bert play pretend with a banana phone; b: Grover demonstrates near and far; c: 12 Ladybugs. (Credit: Sesame Workshop)

Do you remember when Ernie and Bert played pretend with a banana, used it to call Gladys the elephant, practiced conversation skills, and learned about the power of imagination? Or when Grover played with you as he ran exuberantly and exhaustingly near and far, simultaneously learning and illustrating those spatial concepts? How about when ladybugs played games, ran sack races, and played jump rope, to learn about the number 12?

For young children, as for Muppets, playing is learning. Play is how children make sense of the world, develop physical, cognitive, social, emotional, and moral capabilities, as well as build self-esteem. Play is enjoyable, voluntary, and intrinsically motivated. Children learn through play by actively engaging in activities like exploring, creating, problem solving, and communicating. For example, when you were little, playing pretend helped you to develop perspective-taking, empathy, and cooperation, while playing active games such as Simon Says helped you learn self-regulation. Playing board games, you learned turn-taking as well as counting, and while swinging and sliding at the playground you learned some basic physics.

The task for experience designers devoted to helping young kids learn is to inspire meaningful play experiences. To design for learning is to choreograph play that puts kids at the center of the experience, continually empowers them, sustains engagement, opens and deepens worlds, and propels curiosity.

Physical toys such as balls, sandboxes, dolls, and blocks engage sensorimotor learning, while digitally enhanced toys can facilitate exploration beyond children’s everyday lives and provide contingent feedback tailored to the player’s ability. Contingent feedback is powerful in that it provides scaffolded support, like the kinds of support adults naturally provide when playing with kids in the form of suggestions and hints to guide the child figure out the solution. According to Lindsay Lipscomb, Janet Swanson, and Anne West, when adults (or interactive toys) provide this kind of “scaffolding” support, it enables children to succeed at tasks they couldn’t accomplish on their own, which leads to learning.

A challenge in designing for learning is to leverage the strengths of both classic toys as well as interactive technology to inspire rich engagement. How might technology-enhanced toys combine the strengths of physical toys with those of digital media to achieve the positive outcomes associated with both? User experience designers working in the field of design for learning will do well to rigorously consider the essential qualities of physical playthings that should be preserved and the benefits of introducing digital enhancements.

Helllo Everybodeee!! It is Your ol’ Pal Grover!

Sesame Workshop is the nonprofit educational organization behind Sesame Street, which reaches 156 million children across more than 150 countries. Delivered through a variety of platforms, including television programs, digital experiences, books and community engagement, its research-based programs are tailored to the needs of the communities and countries they serve. The Workshop’s mission is to use the educational power of media to help children everywhere grow smarter, stronger, and kinder.

In 1969, Sesame Street was a new kind of show for many reasons: it was based on the idea that if you could engage children you could educate them. It was the first TV show developed using formative research, and it was committed to diversity in its casting and production hires. In addition, and for all of the above reasons, it intentionally and effectively engaged parents in their children’s learning. Parodies and celebrities were specifically designed to attract and retain parents. In 1984 Robert Reiser, Martin Tessmer, and Pamela Phelps (at Sesame Street) and in 1977 Gavriel Salomon (in Israel) studied the effect of adult engagement on kids experience with the show, and researchers found that kids learn more if they watch with adults.

However, more than 45 years later, attracting and retaining parents, especially with the rise of electronic toys and media that can serve as virtual babysitters, proves extremely challenging. And what about kids whose parents aren’t watching Sesame Street or otherwise playing with them? In recent years, Sesame Workshop has had a small team of designers and researchers who were inspired by the above research and play practices working on new ways to play, tell stories, and learn. This team has focused on the following questions:

How might we design experiences that leverage these findings for kids whose parents don’t tend to engage in their play and learning?
Could a Muppet fill the role of an engaged parent or beloved teacher/mentor? If so, could a Muppet acting this way positively impact child learning outcomes? Could a Muppet that models how to scaffold their child’s play inspire and impact adult caregivers’ engagement?
How do you develop an emergent, dynamic, adaptive parasocial learning experience?

In one project exploring these questions, Sesame Workshop partnered with Qualcomm to design meaningful, rich, and deep experiences using Qualcomm’s Vuforia augmented reality technology. The result is Grover’s Block Party, a Sesame Street letter block set and an adaptive, tablet-based personalized play experience with Grover.

3 photos of a child playing with Grover’s Block Party: one of Grover on the tablet screen, one with child arranging blocks, and one with child and blocks represented on the tablet screen. — Figure 2. Kid playing with Grover’s Block Party. (Credit: Sesame Workshop)

In Grover’s Block Party as seen in Figure 2, kids play with wooden letter blocks. Via machine vision and augmented reality, Grover “sees” what the kid is doing with the blocks, and—like an engaged parent—provides “guided play,” a term defined by Deena Skolnick Weisberg, Kathy Hirsh-Pasek, and Roberta Michnick Golinkoff, by responding to and promoting experimentation with building, pattern-making, pretend play, and letter exploration. Grover intuits, follows, comments on, and extends (rather than directs) the child’s natural curiosity and play.

For example, Grover may comment that the child made the word “cat” and play a Sesame Street short film about cats, or recognize that she spelled her name. The child can show any letter block to Grover, and Grover talks and asks questions about that letter and its letter sound.

Grover may also comment on the structures the child builds. He may count that a tower is four blocks high and encourage the child to build higher. Grover may see something that looks like a pyramid, or block formations that look like a rocket ship, robot, or castle, and an on-screen animation plays out those scenes as he imagines them. Grover comments on new blocks the child adds to her structure, or what he thinks the child is building, and if he senses from the child’s block play that he didn’t quite guess right, Grover comments on that too and makes another guess. Grover reacts with excitement and drama each time the child topples what she built.

Designing Playful Learning Experiences

What follows is an overview of what we feel the user experience and learning design community may find relevant about what we learned designing Grover’s Block Party. Our hope is that readers will learn strategies for designing playful learning experiences for learners of any age.

Experience design

The design team embarked on designing Grover’s Block Party excited by classic playthings, tactile play, tangible interfaces, embodied cognition, and the social invitation tangible toys afford. We had recently re-imagined traditional playsets using augmented reality to enhance pretend play, and decided for our next collaboration to focus on letter block play.

The design challenge we set for ourselves centered on the hypothesis that we could enrich block play and propel learning if we could extend play times using an adaptive, intelligent agent, such as a Muppet, to simulate parental engagement in child play. We were also interested to see if the intelligent agent could inspire parents to further engage with their child.

It was critical to us that the player felt that Grover could ”see” and ”understand” the player and was a ”fun playmate”. The play experience needed to feel as deep and varied as each child’s play, to adapt to each player, and to be easy and intuitive for young kids to use by themselves and in social play.

We founded our design and learning principles in the research literature (see the box at the end of the article for the references). For example, we know from Dimitri Christakis, Frederick Zimmerman, and Michelle Garrison’s research that kids learn from block play, especially if an adult is actively present to play with them. Research done by Charles H. Wolfgang, Laura L. Stannard, and Ithel Jones found that block play improves math skills. Katrina Ferrara, Kathy Hirsh-Pasek, Nora S. Newcombe, Roberta Michnick Golinkoff, and Wendy Shallcross Lam also found that playing with blocks improves children’s spatial language abilities.

Block play is primarily based around building and imaginative play, particularly make-believe, which provides rich opportunities to use language and literacy; this was discovered in research conducted by Lynn Cohen and Joanna Uhry, as well as, Ljubica Marjanovič Umek, Anja Podlesek, and Urška Fekonja.

The experience design was informed by findings from early rounds of play testing, as well as block play research literature. For example, Pamela Phelps and Laura Stannard found that children tend to progress through stages of play:

Construction – Beginning with “gathering and dumping”
Building – Horizontal and vertical; repetition of building patterns (stacks and towers)
Bridging & Enclosures – Children create a bridge to connect; they experiment, perseverate, observe, and assess varieties of block combinations. They enjoy placing blocks in such a way that they enclose a space.
Imaginative Play – Play becomes more elaborate in designs, incorporates patterns and balance into their constructions, and explores symmetry. Children don’t name their structures yet, but towers turn into realistic representations.
Dramatic Play – Using blocks to represent objects, places, and people they know (for example, zoo, farm, roads, cars, and the like.)

We began to define and capture the experience design by describing two main system states, one where we felt the system could detect logic and hypothesize player’s activity and intentions (for example, “You have built a very tall tower! Please do not let it fall.”), and another where we felt the system would see and could react to “chaotic” activity (e.g. the system would not guess the player’s intention, but could see and comment on the letter K in her structure). We created storyboards to map these two states including both high-level player behaviors (e.g. player moved a block close to Grover, player makes block topple, player does nothing) and key block formations (e.g. towers, walls, words, pyramids, patterns, words) within the states. We charted the detailed range of player behavior and block formations in spreadsheets that correlated to the storyboards. We then built table-based scripts to define the logic and dialog that correlated to both the storyboards and spreadsheets. That way Grover was able to react to any movement that the child chose to make with the blocks.

Curriculum design

In parallel, the education and research team developed the project curriculum to guide the experience design, system logic, and script development. The curriculum included the following components:

Receptive language
Phonological awareness
Alphabet knowledge
Early writing
Number concepts and relations
Measurement and comparison
Geometry and spatial sense
Reasoning and problem solving
Patterns
Executive functioning

See the More Reading section for Sesame Workshop’s published “Framework for School Readiness” for more information on these skills.

Iterative design & playtesting cycles

A series of research studies were conducted to ensure that Grover’s Block Party was effectively engaging children in terms of understanding, potential for learning, fun, and sustained engagement. The Workshop’s education and research team had children, accompanied by their parents, iteratively playtest a range of commercial block sets and in-progress builds. We conducted foundational research to define block play patterns and parent prompts, as well as iterative design research and a use study over time. For the summative evaluation, we deployed a randomized experimental study with 24 families in Colorado for one week. Half of the families received the final Grover’s Block Party prototype, which included an analytics log, and the other half received a popular commercially available letter block set. Both groups were instructed to use Dscout, a research app, to capture videos of their children playing with the blocks throughout the week, and took part in a 45-minute initial play session as well as pre- and post- interviews. For each round of testing the education and research team built protocols, had a researcher guide the play session, coded the protocol and video data, and produced a report.

What We Learned

Augmented reality (AR) is a real-world environment whose elements are supplemented by computer-generated images, sound, and/or other data. In developing Grover’s Block Party, a Sesame Street AR-enhanced letter block set, we learned the following lessons which we hope will be useful and inspiring to other practitioners exploring augmented reality as a medium and play as a driver for learning:

Mixing an intelligent agent with classic playthings, such as letter blocks was appealing, engaging, and promoted social play.

We found that the length of the average play session for families using Grover’s Block Party was double that of the control group families, which were given a popular commercially available traditional letter block set. Parents of children who played with Grover’s Block Party reported 30-45 minutes of play per day and analytics showed an average of 30 minutes of play per day. 87% of the participants who had Grover’s Block Party reported that the augmented experience made the classic toy more appealing, with some reporting Grover’s Block Party app-usage drop off and increased blocks-only play.

One mother of 2-year-old girl reported, “She started out being extremely focused on just the app but now she’s building with the blocks even when the app isn’t open.” We found the classic wooden blocks felt inherently educational and that families liked their feel. Typical play behaviors for Grover’s Block Party were building towers and walls, knocking down structures, finding letters, looking at themselves on the tablet screen, and playing with parents, siblings, and friends.

Ensure user is in control, that control is intuitive, and provide immediate feedback.

Our research reinforced the cornerstones of what we know to be important in design, especially for young (curious, impatient, experimenting) children. For instance, in all of our work we aim to give the child as much control as possible, ensure that control is instinctual, and always provide immediate feedback. In Grover’s Block Party, block frame-marker registration technology was limited and issues stemming from light quality, as well as children’s perception of field of view, proved frustrating to kids. Addressing issues pertaining to the feeling of control and instinctual play through technical improvements, as well as user experience, is critical to success.

Consider dynamically generated content

Children bring with them a feeling of warmth and trust of the Muppets, and it is critical to build on this relationship to create an environment of extended engagement and dialogue between the child and the character on screen. Grover’s Block Party was especially rich and deep in part because it used dynamically generated content. We know from research done by Alexis Lauricella, Alice Ann Howard Gola, and Sandra Calvert that parasocial extended dialogue around curricular topics enables learning to take place.

Grover’s Block Party employed procedural animation built from “mood” metatags to dynamically generate appropriate mood animations per each of Grover’s lines, and programmatically generated Grover lip movements from the wave form of the audio file. We considered dynamic text-to-speech generation (for example, building an artificial Grover voice from a huge range of Grover recordings) but it wasn’t in scope for this project. Especially for well-known brands and characters, our advice would be to approach dynamic content in isolated tests. We were unsure how children might react to a procedurally animated Muppet, and while we found that they enjoyed it, evaluating its success without the added complexity of integrated dynamic speech felt more within scope.

Create experiences of kids in which kids can see themselves

Sesame Workshop creates places and experiences where all kids can see themselves and appreciate others. In our work we model diversity; in Grover’s Block Party, as well as in other projects, we found “seeing themselves” to be literally true—kids loved seeing themselves on screen, sharing the screen with Grover and their ”living” block constructions. Augmented reality, in mixing real and virtual worlds, affords many creative ways to leverage kids’ seeing and interacting with themselves, other people (both co-located and remote), and virtual characters for engagement and learning.

Machine vision requires designing for variable quality of light

From earlier work we knew that a tablet is too bulky for young children to freely move and manipulate. We therefore designed the tablet in Grover’s Block Party to be in a stand to make the play activity and environment more comfortable. Rather than a kid watching the tablet, and having to reach over it to play with the blocks, we utilized the front facing camera to reflect play. Front-facing cameras were less powerful than standard cameras, and required even more light to enable machine vision and augmentation, and both proved problematic during the in-home study. Front-facing cameras are becoming more powerful, but until they meet machine vision requirements, it is important for user experience designers to proactively design for a range of light and camera quality when designing augmented-reality systems.

Conveying experiences that use emerging technologies is difficult

In our research as well as in our project meetings, we found that it was difficult to convey the full-concept and scope of the adaptive system. We found that kids sometimes struggled with field-of-view issues, and perhaps because Grover’s understanding of player behavior felt so natural to them, they sometimes expected Grover to “see” things he couldn’t. We struggled with this issue and initially resisted using a large mat to define the play space, but in the end included it for the in-home testing. We found that the mat was cumbersome, and addressing the field-of-view issues warrants redesign.

Another area we felt we’d like to further explore is how to optimally communicate the richness of the experience. For instance, how might Grover best reflect to the player what he sees and that he’s responsively following your lead so as to enhance the player’s relationship with him and engagement in the experience? Since Grover’s animations were dynamic, he might pivot and focus his eyes on the player or might point to a specific block. We also found parents didn’t fully understand the range of block play the system supported and the breadth of content. Experience designers, especially those incorporating emergent technology, should carefully consider how the product itself, and the user experience—as well as the surrounding product description and marketing—can best telegraph the concept and scope to users.

3 Sesame Street scenes – a: Singer from Feist holding the number 4; b: Band One Direction holding letter U with Bert and word “unibrow”; c: Grover Waiter delivering food at a restaurant. — Figure 3. a: Learning about the number 4 with Feist; b: What makes U useful with One Direction; c: Grover Waiter delivering an incomplete sandwich. (Credit: Sesame Workshop)

Today’s article was brought to you by the number 4 and the letter U

At the Workshop we’ve been experimenting with playful, kid-centered, silly and empowering learning design for over 45 years. In Figure 3, from learning about the number 4 with Feist and what makes U useful with One Direction to practicing visual discrimination as Grover Waiter delivers an incomplete sandwich, Muppets continue to play on Sesame Street and designers as well as researchers play with how kids, their families, and Muppets might play and learn using emergent technology. Mixing computer support with traditional play and playthings provides an extremely compelling and productive direction for design for learning.

We know that kids of all ages are attracted to, motivated by, and spend significant amounts of time with technology. Even very young kids spend a lot of time with electronic toys. We also know that classic play and playthings such as blocks, provide uniquely meaningful, child-led, socially interactive, sustained engagement, and positive learning outcomes. By remixing classic play modes with computer support, designers open up unique modalities as well as reinvigorate interest and engagement in the classic plaything (for instance. non-augmented blocks) that can pave the way to new audiences as well as new forms of play and learning.

[bluebox]

User Experience