notes & things

Singing in Whale

by lia on July 27, 2013, no comments

“Sing Whale” was my Master’s Thesis, and an ongoing project. I took a break from working on it during my residency at ITP, but now seems like good time to start looking at it again (It also takes about that long to get over post-thesis exhaustion, to be honest).

sw -title

Sing Whale is an iPad storybook where small children can make friends with a whale — by speaking (or singing!) in his language. Children can sing in “whale” by following visual guides and singing accordingly — kind of like tonal connect-the-dots or language-learning karaoke. Here’s the project page on my portfolio site.



It tells the story of a certain whale. The more you ask, the more you learn about a character. This particular whale is a young calf, and he’s got personality.

For instance, let’s ask him: Do you like tacos?




In a survey of picture books, I found some kindred spirits.

Charters that were illustrated simply yet were expressive appealed to me. Using just the right amount of pictures and just the right amount of words allowed children to fill in the blanks with their imagination.


Great picture books contain as much emotional truth as any good literature: In Sesame Streets’
The Monster at the End of this Book, Grover desperately begs the reader to stop turning the pages, as someone has told him that there is a monster at the end of the book!


Of course the reader can’t stop, much to grover’s despair.


I hate to ruin the ending, but it turns out there was nothing to be afraid of, as the only monster at the end of the book was grover himself.


Mo Willems is one of the most prolific writers of children’s books, and you may recognize his work: there’s Knuffle Bunny, Cat the Cat, and Don’t Let The Pigeon Drive The Bus, featuring an almost manic pigeon that you need to keep under control, mostly by shouting “NO!” at the book. Indeed, Willems has said that he prefers his books were played, not read.


In my personal favorite starring Elephant and Piggie, things get existential when the two main characters find out the truth: that they are in a book! They are being read!


If they are in a book, then all books must end and and all stories must finish. It is a picture book that is as sad as it is funny.


Here is a video of the basic interaction of the prototype: ipad_demo_2 – Wi-Fi

You (1) choose what you want to say to our whale; (2) learn how to sing it via an audio and graphic guide; (3) sing it; (4) see what happens!


Choose what you say. This prototype has a few options: “Do you like Tacos?”; “Shark! There’s a shark!”; “I have a gift for you”; “I like stars”; “Do you believe in ghosts?”; “Where is your mom?”


Select one, and you are shown (and told) how to sing it. A child’s voice is heard while a waveform-like shape appears to illustrate the song.


Now you try … the mic switches on and you generate your own waveform to compare to the original one.


If you don’t get it right, the whale reacts. He can say something standard, like, “I didn’t get that”, but sometimes its a surprise, like, “I’m sorry, did you just call me fat?”


If you do get it, you’ll get the answer to your question. And since we asked, “Where is your mom?” He proceeds to call her, and in the last frame, she appears (she’s huge!).


Different branches can lead to smaller narratives, for instance, telling the whale that it is time for bed
prompts him to ask you for a bedtime song.


Other questions can lead to different answers every time, like the one where we give him a gift. Now it is a hat,
the next time might be a framed photo of his mom.


It’s about learning a language — or at least, about the suggestion of learning a language.

… Making unfamiliar sounds. The whale song to me was the perfect one to start with, not only because of popular fascination for it, and not only because it is fun, but because its very nature lends to a very large handicap with regards to pitch: low, medium and high is all you need to know to play the game.

… Being silly.Firstly, I wanted to provide a space where children and their parents could be silly together. Whale songs sung by humans are funny. Your mom or dad making whale songs with you is also really funny. Embarrassment, after all, is a learned behavior, and children haven’t gotten around to being embarrassed quite yet.

… Hearing your own voice. I wanted to create something where children could speak up and hear their own voices in a playful environment.

Finally, it’s about how we are not all the same. There is a personal ulterior motive to this project: in playfully teaching children how to speak in “whale”, my hope was not only to suggest the idea of other languages, but that there are creatures, beings, and people out there that may not look like you or speak like you, but you can get to know them if you are curious enough. It is about what it means to make friends with someone who is from somewhere else, somewhere entirely alien to you.

While in the real world, scientists are still trying to decode what whales are saying to each other, in this make-believe one that I have created, perhaps if we can learn how to speak their language, we can be friends.


Narrow down an age range.  In most cases, you would first decide what age you are targeting, and then tailor the product after that. In my case, I dove right into the project — with a vague idea that I was making it for 6-8 year olds. After 2 months of development, I was able to test it with a wide range of ages, and found the reactions to be very different.


Children under 3 were just happy to touch things that would make funny noises. It was nice to see that I had enough funny noises in my app to keep them occupied for a while (the little girl above, Luna, was extra adorable), but that is not really an achievement.

The Pre-K class that I was able to spent the most time with enjoyed the activity thoroughly — learning how to speak to a whale and seeing how the whale reacted — but had to be directed on how to use the app. In most cases, my friend had to control the iPad for them, and they would do the whale sounds together. It was great to see them excitedly telling their parents as they were picked up how much fun they had that day: “We learned how to speak in whale!”  The documentation video I have up on my site is of them.

Screen Shot 2013-07-26 at 6.44.35 PM

At was also able to test with some older children, 7- 9 years old. This was the most satisfying, as they were able to navigate through the interface by themselves, and were genuinely curious about exploring all the things they could say to the whale. This one kid at the MoCCA Convention stayed at our booth for at least half and hour, and even asked his mom to sing with him.

When making things for children, as with most of design, setting a wide net to design for is a bad idea. Younger children need a lot more prompting, big big icons, and a voice to read everything out loud. This type of handholding would make older children impatient. Besides, younger children are more likely to try older kids’ toys than the other way around.

The Story.


The prototype was a success, I think, for two reasons: one, it proved that software-wise, this project is doable, even by me; and two, I saw enough children (and adults) like it enough for me to think it deserves a shot. But, like I said in the beginning, the story and the characters are the most important part. What good is a novel interaction if the story has no “emotional truth” (as Mo Willems likes to say), or at the very least, fun?

The Interface. There are a lot of things I would change about the interface. In fact, that would probably be another blog entry. I would change the scrollwheel, where you choose what to say via some representative pictures, into something less limiting. The visualization of the whale song needs a lot of work. A nice tutorial in the beginning would be good too.



I have to mention a little of the technical part, as it took up most of my time during this entire process. I programmed the whole thing with openFrameworks for iOS, which is still buggy, but nonetheless great. There were a few technical hurdles, which included integrating the microphone and processing the audio, which I analyzed using FFT (Fast-Fourier Transform).



So that’s a recap of where I am with this project, and where I’m going to go from here. Hello, old thesis, we meet again.


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">