notes & things

Leap + My Quantified Broken Wrist

by lia on August 20, 2013, no comments

This is a post about something new that I have been working on with the Leap Motion — an application that lets me track my progress with regards to regaining my range of motion at the wrist.

rom_1

Let me explain.

brokeWrist

This is a picture of me and my partner, William, in Japan. I have a sling on because I few days earlier, we were in a motorcycle accident in an island paradise in the Philippines. Long story short, I broke my wrist. Upon our return to the USA weeks later, I was informed by my doctor that the wrist was set wrong by the truly lovely clinic in said island paradise, and I would need surgery to re-break the bone and set it right.

rom_2

That’s me, post-surgery, happily accepting a get-well-soon panda from my friends (Panda has a broken wrist too!).

I was in a cast or splint for a very, very long time (3 months), and am still currently in the middle of physical therapy. Recovering is a pain, because it is so s-l-o-w. I am the kind of person that constantly wants to know how much further along I am now than I was a moment ago — I like to watch progress bars, yes — but my physical therapist insisted that we only take measurements every few weeks or so. I suppose he said this because he wants a more dramatic reveal every time, but knowing how well/ badly I was doing daily was an itch I had to scratch.

Enter the Leap Motion. Essentially, it is a tiny device that uses infrared to see where your hands are, what they are pointing at, and to some degree, what angle your palm is facing — the bit that is most important to me as that gets rotation values. When I ordered it in February, it was one of the first released. It isn’t technically made to accurately measure the angle of rotation of your palm — when you rotate one way and your thumb or one of the fingers disappear from the Leap’s field of view for instance, it stops being a complete “hand” and the data disappears. It is really more for use as a pointing interface, but I thought I would give it a try anyway.

(At some point you might ask — why not use a regular old measuring device? This guy did a great job of it. It looks like someone even tried to mount an IR sensor on a wrist. All worthwhile efforts, of course. But my goal was twofold:
1. How do I make it as painless as possible for patients recovering from injury to measure and track their range of motion daily?
2. What cool thing can I make with the Leap Motion?
)

THE PROPOSAL
I would create a web app that would connect to the Leap Motion and allow the user to measure and track their daily range of motion from the wrist. This data would be collected and visualized so that the user can see her progress. Other factors that can affect progress can be tracked as well, such as daily exercises, activity, and mood.The web app will store your data, so that your doctor/ therapist/ parents can see how you are doing too.

The great thing about using a device like the Leap is that it is extremely simple to use — stand in front of your computer, face the camera, and bend/ rotate your wrist as much as you can.

The not-so-great thing about the Leap Motion is that it’s not very accurate, at least for my purposes. It wasn’t made for this. There are some solutions, such as: doing it a few times a day so that you get a better average, filtering out the extreme values, and smoothing. These will have to be tested.

THE PROCESS
1. Feasibility testing
2. Interface sketches
3. Prototype
4. Test
5. Revise Interface
6. Circle back to #3 a few times
7. Visual Design & Fabrication

I BUILT THE APP
… Using Processing, the Control P5 library, and the Leap for Processing Library. The code can be found here.

As I mentioned above, it was tricky getting accurate rotation information from the Leap. I had to make some compromises, and I’m hoping to get cleaner values after some software filtering. I did, however, get to a point where I was confident that I could get “good enough” readings for me to progress to making a prototype. I was also, to be honest, getting better, so if I wanted to test this on my own progress I knew I had to hurry.

WHAT TO MEASURE

I consulted my therapist and my doctor, and we agreed that the easiest, and most important to measure were Flexion, Extension, Pronation and Supination, as seen above.

PERCENTAGES, NOT DEGREES
I also decided to do away with measuring in actual recognizable units — degrees or radians. Instead, I thought I would measure in percentages against the range of motion of the OTHER wrist. So in when the program first sets up, the user measures the range of the good wrist in all angles, and that is what we use as a baseline for the healing wrist. It frees me from having to accurately read degrees or radians, and also is I think a more friendly and intuitive way to measure your progress.

FIRST PROTOTYPE
I created this first prototype so I could begin testing on myself. Below is a screenshot of the entire app, and then I’m going to go into the details of what I have so far.

rom_1

THE STATUS BAR
The tricky thing with interfaces where your hand floats in the air is that you don’t get any real feedback, at least not right away. The first thing you think when you raise your hand over sensing device is: “where am I?” So the first order of business was to visualize the hand.

33emw

There are important status updates that happen here as well, such as telling the user when no hand is found (then obviously no recording can be made) or when there are multiple hands detected.

THE CAMERA
I initially added the camera for the same reason as the hand visualizer: I needed to see where I was on the screen. But then I saw that it was valuable to be able to take a snapshot at the same time that you are measuring the range of motion — it places an image to the number.

Also, it makes a neat stop motion after even a few records (see below).

RECORDING
When you hit ENTER to start recording, a little red circle starts blinking.
This is the bit that I like: as you bend/ rotate your wrist, your current rotation is overlaid on your previous one. That way, you get to compare any difference you may have made since you last recorded.

rom_4

TRACKING BACK
If you want to see your progress, you can scroll backwards. This is a precursor to a much more robust visualization to come.

33m4h

WHAT I HAVE LEARNED / NEXT STEPS
1. Stay in the air. When you are using the Leap as an interface, stay on the Leap. Don’t use the keyboard. I would like to be able to use my good hand — the right — to control the app while the other one is being measured.
2. Measure against last reading, this morning’s reading, last week’s reading, etc. It is always nice to know how much progress you have made.
3. Visualize the data. More attractive charts and graphs to track one’s progress. Add goals?
4. Put it in the browser. Create log-ins for users.
5. Create a physical mount so that your wrist is laying on the same spot every time. It might be unnecessary as the Leap can get its measurements anyway, but it makes better-looking photos for the timelapse.
6. Create an intro/ setup segment where you can learn how to use the app as well as calibrate it by setting the range of motion of your good wrist as the baseline.

There are plenty of little fun things that can be tweaked in the future, but for now these are the broad strokes improvements I can think of. Comments welcome.

Lia

leap_leap

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">