notes & things

I’m on the Microsoft website!

by lia on September 5, 2015, no comments

Microsoft just redesigned and put a few of our smiling faces up on it. You have to scroll down to find me, but there I am:

The photographer told me to 'pose like a lion'

They told me to pose like this.

I’m also pretty proud that the Red Burns quote made it through. That’s my shout out to ITP and our great leader.

We took a photo for MakerU and the Makerspace as well, that should be coming out in a week or so :)

Hi mom!

Leap + My Quantified Broken Wrist

by lia on August 20, 2013, no comments

This is a post about something new that I have been working on with the Leap Motion — an application that lets me track my progress with regards to regaining my range of motion at the wrist.


Let me explain.


This is a picture of me and my partner, William, in Japan. I have a sling on because I few days earlier, we were in a motorcycle accident in an island paradise in the Philippines. Long story short, I broke my wrist. Upon our return to the USA weeks later, I was informed by my doctor that the wrist was set wrong by the truly lovely clinic in said island paradise, and I would need surgery to re-break the bone and set it right.


That’s me, post-surgery, happily accepting a get-well-soon panda from my friends (Panda has a broken wrist too!).

I was in a cast or splint for a very, very long time (3 months), and am still currently in the middle of physical therapy. Recovering is a pain, because it is so s-l-o-w. I am the kind of person that constantly wants to know how much further along I am now than I was a moment ago — I like to watch progress bars, yes — but my physical therapist insisted that we only take measurements every few weeks or so. I suppose he said this because he wants a more dramatic reveal every time, but knowing how well/ badly I was doing daily was an itch I had to scratch.

Enter the Leap Motion. Essentially, it is a tiny device that uses infrared to see where your hands are, what they are pointing at, and to some degree, what angle your palm is facing — the bit that is most important to me as that gets rotation values. When I ordered it in February, it was one of the first released. It isn’t technically made to accurately measure the angle of rotation of your palm — when you rotate one way and your thumb or one of the fingers disappear from the Leap’s field of view for instance, it stops being a complete “hand” and the data disappears. It is really more for use as a pointing interface, but I thought I would give it a try anyway.

(At some point you might ask — why not use a regular old measuring device? This guy did a great job of it. It looks like someone even tried to mount an IR sensor on a wrist. All worthwhile efforts, of course. But my goal was twofold:
1. How do I make it as painless as possible for patients recovering from injury to measure and track their range of motion daily?
2. What cool thing can I make with the Leap Motion?

I would create a web app that would connect to the Leap Motion and allow the user to measure and track their daily range of motion from the wrist. This data would be collected and visualized so that the user can see her progress. Other factors that can affect progress can be tracked as well, such as daily exercises, activity, and mood.The web app will store your data, so that your doctor/ therapist/ parents can see how you are doing too.

The great thing about using a device like the Leap is that it is extremely simple to use — stand in front of your computer, face the camera, and bend/ rotate your wrist as much as you can.

The not-so-great thing about the Leap Motion is that it’s not very accurate, at least for my purposes. It wasn’t made for this. There are some solutions, such as: doing it a few times a day so that you get a better average, filtering out the extreme values, and smoothing. These will have to be tested.

1. Feasibility testing
2. Interface sketches
3. Prototype
4. Test
5. Revise Interface
6. Circle back to #3 a few times
7. Visual Design & Fabrication

… Using Processing, the Control P5 library, and the Leap for Processing Library. The code can be found here.

As I mentioned above, it was tricky getting accurate rotation information from the Leap. I had to make some compromises, and I’m hoping to get cleaner values after some software filtering. I did, however, get to a point where I was confident that I could get “good enough” readings for me to progress to making a prototype. I was also, to be honest, getting better, so if I wanted to test this on my own progress I knew I had to hurry.


I consulted my therapist and my doctor, and we agreed that the easiest, and most important to measure were Flexion, Extension, Pronation and Supination, as seen above.

I also decided to do away with measuring in actual recognizable units — degrees or radians. Instead, I thought I would measure in percentages against the range of motion of the OTHER wrist. So in when the program first sets up, the user measures the range of the good wrist in all angles, and that is what we use as a baseline for the healing wrist. It frees me from having to accurately read degrees or radians, and also is I think a more friendly and intuitive way to measure your progress.

I created this first prototype so I could begin testing on myself. Below is a screenshot of the entire app, and then I’m going to go into the details of what I have so far.


The tricky thing with interfaces where your hand floats in the air is that you don’t get any real feedback, at least not right away. The first thing you think when you raise your hand over sensing device is: “where am I?” So the first order of business was to visualize the hand.


There are important status updates that happen here as well, such as telling the user when no hand is found (then obviously no recording can be made) or when there are multiple hands detected.

I initially added the camera for the same reason as the hand visualizer: I needed to see where I was on the screen. But then I saw that it was valuable to be able to take a snapshot at the same time that you are measuring the range of motion — it places an image to the number.

Also, it makes a neat stop motion after even a few records (see below).

When you hit ENTER to start recording, a little red circle starts blinking.
This is the bit that I like: as you bend/ rotate your wrist, your current rotation is overlaid on your previous one. That way, you get to compare any difference you may have made since you last recorded.


If you want to see your progress, you can scroll backwards. This is a precursor to a much more robust visualization to come.


1. Stay in the air. When you are using the Leap as an interface, stay on the Leap. Don’t use the keyboard. I would like to be able to use my good hand — the right — to control the app while the other one is being measured.
2. Measure against last reading, this morning’s reading, last week’s reading, etc. It is always nice to know how much progress you have made.
3. Visualize the data. More attractive charts and graphs to track one’s progress. Add goals?
4. Put it in the browser. Create log-ins for users.
5. Create a physical mount so that your wrist is laying on the same spot every time. It might be unnecessary as the Leap can get its measurements anyway, but it makes better-looking photos for the timelapse.
6. Create an intro/ setup segment where you can learn how to use the app as well as calibrate it by setting the range of motion of your good wrist as the baseline.

There are plenty of little fun things that can be tweaked in the future, but for now these are the broad strokes improvements I can think of. Comments welcome.



Singing in Whale

by lia on July 27, 2013, no comments

“Sing Whale” was my Master’s Thesis, and an ongoing project. I took a break from working on it during my residency at ITP, but now seems like good time to start looking at it again (It also takes about that long to get over post-thesis exhaustion, to be honest).

sw -title

Sing Whale is an iPad storybook where small children can make friends with a whale — by speaking (or singing!) in his language. Children can sing in “whale” by following visual guides and singing accordingly — kind of like tonal connect-the-dots or language-learning karaoke. Here’s the project page on my portfolio site.



It tells the story of a certain whale. The more you ask, the more you learn about a character. This particular whale is a young calf, and he’s got personality.

For instance, let’s ask him: Do you like tacos?




In a survey of picture books, I found some kindred spirits.

Charters that were illustrated simply yet were expressive appealed to me. Using just the right amount of pictures and just the right amount of words allowed children to fill in the blanks with their imagination.


Great picture books contain as much emotional truth as any good literature: In Sesame Streets’
The Monster at the End of this Book, Grover desperately begs the reader to stop turning the pages, as someone has told him that there is a monster at the end of the book!


Of course the reader can’t stop, much to grover’s despair.


I hate to ruin the ending, but it turns out there was nothing to be afraid of, as the only monster at the end of the book was grover himself.


Mo Willems is one of the most prolific writers of children’s books, and you may recognize his work: there’s Knuffle Bunny, Cat the Cat, and Don’t Let The Pigeon Drive The Bus, featuring an almost manic pigeon that you need to keep under control, mostly by shouting “NO!” at the book. Indeed, Willems has said that he prefers his books were played, not read.


In my personal favorite starring Elephant and Piggie, things get existential when the two main characters find out the truth: that they are in a book! They are being read!


If they are in a book, then all books must end and and all stories must finish. It is a picture book that is as sad as it is funny.


Here is a video of the basic interaction of the prototype: ipad_demo_2 – Wi-Fi

You (1) choose what you want to say to our whale; (2) learn how to sing it via an audio and graphic guide; (3) sing it; (4) see what happens!


Choose what you say. This prototype has a few options: “Do you like Tacos?”; “Shark! There’s a shark!”; “I have a gift for you”; “I like stars”; “Do you believe in ghosts?”; “Where is your mom?”


Select one, and you are shown (and told) how to sing it. A child’s voice is heard while a waveform-like shape appears to illustrate the song.


Now you try … the mic switches on and you generate your own waveform to compare to the original one.


If you don’t get it right, the whale reacts. He can say something standard, like, “I didn’t get that”, but sometimes its a surprise, like, “I’m sorry, did you just call me fat?”


If you do get it, you’ll get the answer to your question. And since we asked, “Where is your mom?” He proceeds to call her, and in the last frame, she appears (she’s huge!).


Different branches can lead to smaller narratives, for instance, telling the whale that it is time for bed
prompts him to ask you for a bedtime song.


Other questions can lead to different answers every time, like the one where we give him a gift. Now it is a hat,
the next time might be a framed photo of his mom.


It’s about learning a language — or at least, about the suggestion of learning a language.

… Making unfamiliar sounds. The whale song to me was the perfect one to start with, not only because of popular fascination for it, and not only because it is fun, but because its very nature lends to a very large handicap with regards to pitch: low, medium and high is all you need to know to play the game.

… Being silly.Firstly, I wanted to provide a space where children and their parents could be silly together. Whale songs sung by humans are funny. Your mom or dad making whale songs with you is also really funny. Embarrassment, after all, is a learned behavior, and children haven’t gotten around to being embarrassed quite yet.

… Hearing your own voice. I wanted to create something where children could speak up and hear their own voices in a playful environment.

Finally, it’s about how we are not all the same. There is a personal ulterior motive to this project: in playfully teaching children how to speak in “whale”, my hope was not only to suggest the idea of other languages, but that there are creatures, beings, and people out there that may not look like you or speak like you, but you can get to know them if you are curious enough. It is about what it means to make friends with someone who is from somewhere else, somewhere entirely alien to you.

While in the real world, scientists are still trying to decode what whales are saying to each other, in this make-believe one that I have created, perhaps if we can learn how to speak their language, we can be friends.


Narrow down an age range.  In most cases, you would first decide what age you are targeting, and then tailor the product after that. In my case, I dove right into the project — with a vague idea that I was making it for 6-8 year olds. After 2 months of development, I was able to test it with a wide range of ages, and found the reactions to be very different.


Children under 3 were just happy to touch things that would make funny noises. It was nice to see that I had enough funny noises in my app to keep them occupied for a while (the little girl above, Luna, was extra adorable), but that is not really an achievement.

The Pre-K class that I was able to spent the most time with enjoyed the activity thoroughly — learning how to speak to a whale and seeing how the whale reacted — but had to be directed on how to use the app. In most cases, my friend had to control the iPad for them, and they would do the whale sounds together. It was great to see them excitedly telling their parents as they were picked up how much fun they had that day: “We learned how to speak in whale!”  The documentation video I have up on my site is of them.

Screen Shot 2013-07-26 at 6.44.35 PM

At was also able to test with some older children, 7- 9 years old. This was the most satisfying, as they were able to navigate through the interface by themselves, and were genuinely curious about exploring all the things they could say to the whale. This one kid at the MoCCA Convention stayed at our booth for at least half and hour, and even asked his mom to sing with him.

When making things for children, as with most of design, setting a wide net to design for is a bad idea. Younger children need a lot more prompting, big big icons, and a voice to read everything out loud. This type of handholding would make older children impatient. Besides, younger children are more likely to try older kids’ toys than the other way around.

The Story.


The prototype was a success, I think, for two reasons: one, it proved that software-wise, this project is doable, even by me; and two, I saw enough children (and adults) like it enough for me to think it deserves a shot. But, like I said in the beginning, the story and the characters are the most important part. What good is a novel interaction if the story has no “emotional truth” (as Mo Willems likes to say), or at the very least, fun?

The Interface. There are a lot of things I would change about the interface. In fact, that would probably be another blog entry. I would change the scrollwheel, where you choose what to say via some representative pictures, into something less limiting. The visualization of the whale song needs a lot of work. A nice tutorial in the beginning would be good too.



I have to mention a little of the technical part, as it took up most of my time during this entire process. I programmed the whole thing with openFrameworks for iOS, which is still buggy, but nonetheless great. There were a few technical hurdles, which included integrating the microphone and processing the audio, which I analyzed using FFT (Fast-Fourier Transform).



So that’s a recap of where I am with this project, and where I’m going to go from here. Hello, old thesis, we meet again.


Radio – Eavesdropping on America

by lia on July 26, 2013, no comments

In early 2012, my friend Merche Blasco an I worked on a project called Radio. In the most basic sense, it is a large wooden map of America that you can use to listen to the different radio stations broadcasting across the country by navigating a little knob on the surface. As another friend put it, it’s like “eavesdropping on America.” I never got a chance to post about it, so I will now.



Our interest started with the idea that much of what a local culture is can be revealed to us by what is being broadcast on its radio local frequencies. America is a large country with a diverse people: the song playing in Massachusetts will be different from what the residents of New Orleans listen to. The differences between advertisements, talk shows, game shows, etc. will be interesting as well. With Radio, you can tune in by isolating an area or by mixing different radio stations together and hearing the bigger picture.


Screen Shot 2013-07-26 at 2.22.12 PM

Because we wanted to keep the geographical component of the project was so important, the interface obviously had to be a map, so that part was easy.

We also wanted to evoke the feeling of an old-fashioned radio, so a nice hardwood surface came to mind. Naturally, time and cost restrictions came into play, so we settled some with the leftover material from the woodshop, and used a CNC to cut out a shape of America and carve out small holes to mark the cities.

You can barely make it out in the picture above, but there’s a crack in the middle where we used to connect the two cut wooden parts together like puzzle pieces — it runs along where the Mississippi is. A nice little nod to Radio history, where stations that are west of the Mississippi have a call sign starting with W, and East have call signs starting with K.


For the prototype, we relied on camera vision to detect where the knob was on the map at any given time. We placed infrared LED in the knob and a security camera mounted above to track the invisible light.

An XML file was created with some map points that corresponded to each state’s capital cities and the 5 most popular radio stations in the area that were streaming on the internet.

We used openFrameworks to read the map points from the XML file and plotted them on a grid that was calibrated to the actual physical map’s size and position. When the knob’s IR LED was within the vicinity of a map point, a random radio station from that area would begin to play. For this, we used openFrameworks to communicate with Terminal, which then triggered VLC to access the streaming radio station.


Here’s what I think: I love this project. I think it’s simple, elegant, and offers a rich experience. It’s a new way to listen to the radio, and a new way to think about the country.

When you love a project, it’s hard to see where it can be improved, but thankfully that’s what exhibits and shows are for. We discovered, while watching users interact with our piece, that most of what needed to be tweaked involved the user experience: when the user first approaches the installation — made up of two things, a wooden map laid out like a table and a metal knob on top of it — does she know how to use it? Does she know what she is listening to? And within those questions, things like: how to handle the knob; displaying city names; and of course the responsiveness of the software.


The Knob: It was important that the user DRAG the knob across the map, instead of lifting it up and moving it somewhere. There’s a technical reason for this (our software was looking for the LED around a last-known-location) but there is an romantic aspect to “traveling” from one location to another, instead of just hopping around (in my mind, the equivalent of mashing keys together to see what this or that button does). But inevitably, users were impatient and chose to pick up the knob and place it in the city of their choice.

So how then do we prevent this behavior? Some thoughts:

  • Weight. If we made the knob heavy enough, with a nice padded bottom to make dragging across the map it a smooth, sliding action, people would instinctively know to slide, not lift.
  • Pull. If there were tiny magnets under the cities, then the user would “feel” a gentle weight when it approaches a city, and release as the knob moves away — a fun detail that can’t be experienced with just dropping the knob down in any location.
  • Audio. To reward sliding it across further, the audio would get louder as you approach a city, and decrease as you move away, or doing a simple fade in/ fade out based on distance. The fun bit is that you effectively “mix” two stations together when you are in between cities, controlling their levels as you move around the board.

Name The City. In the beginning, we were quite adamant about not having any text around the map — no city names, no station names, nothing. The conceit was that the focus needed to be on the discovery of radio stations as you “travelled” across America, and text would just be clutter.

However, as more people used it, we saw that a big part of their enjoyment came from being able to identify the cities. People enjoyed tuning in to their hometown’s favorite station, or tried to guess what kind of music would be playing from a given town (and vice versa — “Let’s go find some country music.”)

So in the next version, there will be a tiny screen somewhere on the map — maybe near Florida — that will display the current city as well as the currently playing radio station.

Hardware. Using computer vision worked great as a prototyping tool, as there were pretty much no wires to run or solder, but it was also fairly unstable. Calibration took a while, and IR light can come from anywhere and disrupt your data (for example: an open window). For the next version, we’re going to give some hardwired sensors with an Arduino or Raspberry Pi a go.

Hawaii, Alaska, and Manhattan Finally, there are those areas that simply couldn’t be included in the installation — in this regard, we are still flabbergasted. If anyone has any ideas, we’d love to hear them.


by lia on June 21, 2013, no comments

Some snippets from this interview of the CEOs of Toca Boca and Sago Sago (which was just bought by Toca Boca). I’ve selected some lines that I like.

Jason: Like Toca Boca, our approach is to create beautiful apps that are modelled after toys rather than games or books.

Jason: A well-designed app for this age group works well alongside flap-books, stacking cups, a good toy piano and other toddler-friendly toys. In terms of child development theory, we look to the constructivist school of thought. We focus on making experiences that are open-ended and child-led as opposed to structured and reward-based. As adults it’s easy for us to forget that young children are natural scientists. Instead of teaching kids, we’re better off providing good materials and inviting kids to teach themselves.

Björn: We still look at the way kids play — broadly speaking. We tend not to focus too much on the app space, but rather on the underlying patterns to see what could be fun with a screen too. The same goes for the design — looking around you in adjacent spaces can often be much more inspiring than to look at other kids apps.

Toca Boca
Sago Sago

Making Things For Kids update

by lia on June 19, 2013, no comments

Unfortunately, due to the unexpected turn of events (we’re moving to Seattle!) I will be unable to teach Making Things For Kids in ITP this fall semester. I did, however, amass a rather large set of resources, and will be posting them up in list form as soon as I can get it organized. Watch this space, and I hope to see you soon anyway.


Mobile Storytelling

by lia on June 19, 2013, no comments

Below is Anna and my first lecture for our Mobile Narratives class series for ITP Camp. I’m pretty proud of it. It’s a good initial survey into the world of Digital Storytelling, and should provide you with a head start if you want to know what’s out there. Enjoy.