Leap motion sensor

[brantlew]

Wow! These videos look really impressive to me. 0.01mm accuracy - that's insane! Hardly any lag either. I'm interested to know what it's limitations are, but I bet the VR community could mod the crap out of this thing for all kinds of head/hand/body tracking goodness. Anybody seen this at a show or demo? Available January 2013, but I think dev kits are available now.

http://www.engadget.com/2012/05/21/leap ... e-control/

[youtube]http://www.youtube.com/watch?v=_d6KuiuteIA[/youtube]

[youtube]http://www.youtube.com/watch?v=N6hCwjwzUHg[/youtube]

https://live.leapmotion.com/

[brantlew]

I guess the most obvious idea that sort of "jumps out" would be to mount this on the front of your HMD and use your hand as a sort of data glove.

I wonder if you could also mount it on your head (or reverse mount it on your back) and use it as head tracker?

[WiredEarp]

How does it work?
0.01mm accuracy seems too fine for it to just be a camera or 3D camera. The fact that they have not mentioned any actual info about the technology itself, makes me a bit suspicious.

[brantlew]

How does it work?
0.01mm accuracy seems too fine for it to just be a camera or 3D camera. The fact that they have not mentioned any actual info about the technology itself, makes me a bit suspicious.

Yeah, I don't know either. You would need two high resolution cameras in that little device running at a pretty high frame rate. Seems unlikely. Maybe infrared laser?

[FingerFlinger]

Yeah! I saw this on Tech News Today, and immediately wanted to head mount it for hand/arm tracking.

[Bishop51]

That's really impressive! Latency is such a gigantic yet subtle issue with VR and this kind of tech could really help things along. I sincerely hope that someone smarter than I can make this work as a head tracker.

[brantlew]

Head-tracking idea 1: Hang a pronged 3-axis shape a foot above your head. Then mount the Leap on the top of your head pointing upward towards the shape. That will give you 6 degrees of freedom head tracking within a limited area.

Head-tracking idea 2: Mount some type of rod protrusion on the side or back of your head and mount the Leap on your shoulder or on your back pointing upwards. It will give you the orientation of your head relative to your torso. Not good for 360, but could be useful in more limited situations like flight sim. (Edit: Mounted on the back of your chair might work even better for cockpit sim)

Idea 3: I wonder if you could use multiple devices in a grid pattern? Idea #1 using a 2x2 grid would extend your range.

I wonder if the technology is scalable? Could they offer a device with the same range as the Kinect - except with full body tracking at millimeter accuracy and 1/10 the latency? That would be amazing! I can't wait for the technical details to leak out.

[cybereality]

This sounds really good, almost too good to be true. 0.01mm from an optical method seems crazy. There is also a long discussion thread on the VR Geeks mailing list, and people are saying that the thing looks suspect (as well as the people behind it). Can't really verify anything myself, but I am a little suspicious. I mean, they probably have something but I feel like maybe they are over-hyping their technology.

[zalo]

They have confirmed that they are using reflected infrared light to get the data.

I bet that this system is much more simple than the Kinect (since Time of Flight won't work at such short distances). I'd wager that there are some simple IR leds and a standard grey-scale IR camera under that black plastic. With a relatively diffuse object like the hand, it would be easy(-ish) to extrapolate distance from the brightness of the reflected IR light.

The misleading point is the sub-millimeter accuracy. You don't have that accuracy for the whole hand, just the point of contact. They would use a system like the Wii-remote camera called Point Interpolation that upped the camera's real resolution of 128x96 to a point tracking resolution of 1024x768 (8x). It works by averaging the positions of all the pixels that represent "finger-tip" to come up with a sub-pixel answer.

This technique will probably also work with non-finger objects like pencils that are also diffuse.
It will have a really tough time tracking mirrored and clear object properly.

I'm also suspicious how in the video, at the point where they show the point-cloud, you can see points for the back-side of the hand and fingers as well. Near-Infrared is cool, but it's not X-Ray.

[brantlew]

@zalo: Agreed that the sub-millimeter statement is probably only theoretical and misleading. Maybe only at the very front of the view frustum under perfect conditions. Or it could just be the optical density of the camera sensor.

I wonder if they might be using stereoscopic cameras. Their up/down distance calculations are really good. Hard to believe you could deduce that with just a single 2D light intensity map. Maybe they are using IR camera triangulation which would be a great improvement over the Wiimote.

I also noticed the back of the hand thing. Since they are dealing with a known shape they could just extrapolate and close the polygons to build a solid I guess.

[zalo]

I saw a youtube video somewhere, where it was in the "point-cloud" mode, and he moved his hands forward so his fore-arms were in the shot, and those were captured realistically too. I don't know if I should be more suspicious, or impressed.

In other news, Stereo Light-Falloff is looking like a likely candidate:
http://www.vis.uky.edu/~liaomiao/LASLive.pdf

They've photoshopped all the images on the site so the cover looks perfectly black, except for in the one on the main page under "Own the future. $70" you can vaguely make out a camera on either end of the device. If I had my photoshop with me, I'd up the contrast for you.

If this is the case, it's still a great achievement, but it's not going to be as reliable for all objects as Kinect's more robust Time of Flight measurement.

[cybereality]

The Kinect is not a time of flight camera. It projects an IR point grid and then reconstructs depth from the difference.

[brantlew]

I saw a youtube video somewhere, where it was in the "point-cloud" mode, and he moved his hands forward so his fore-arms were in the shot, and those were captured realistically too. I don't know if I should be more suspicious, or impressed.

I think if the software just closes all surfaces with a smooth curve on top, then you will get pretty good approximations of fingers, hands, and arms. It would be interesting to see what would happen if they shoved a book into that point cloud. I'm betting the model would not be flat on top.

They've photoshopped all the images on the site so the cover looks perfectly black, except for in the one on the main page under "Own the future. $70" you can vaguely make out a camera on either end of the device. If I had my photoshop with me, I'd up the contrast for you.

You've got eagle eyes because I don't see it, but I haven't fooled with the contrast so I'll take your word on it. Good catch.

[WiredEarp]

Even a 1024x768 camera will not get 0.01mm accuracy when reading a 4' (lets say 1m for simplicity) area. The best accuracy I believe it could get would be around 1mm.

[zalo]

That was the accuracy for the Wii remote camera. I imagine that theirs are much higher quality, full grey scale (the Wii's were monochrome), and stereo. This would help the point interpolation immensely.

Oops, sorry. Cyber is absolutely right about the Kinect. That means that the leap could use some sort of structured light too.

[WiredEarp]

If 1000 res will get you a 1mm resolution, to get a 0.01 resolution you'd need a 100 times that resolution wouldn't you?

[brantlew]

Like zalo was saying, it's not all about the optical density of the pixels. It's also about the sensitivity of the photo-sensors and the bit depth of the pixels. The greater the depth, the more finely it can be interpolated and the more "virtual" pixels can be computed. If each pixel is strictly binary then as the light moves from the center of one pixel to the center of the adjacent pixel, there are 3 distinct states (10, 11, 01). You have increased your effective resolution by 50% with that simple interpolation. If your pixels are two bits deep, then you can represent 5 virtual pixels. And so on...

[seilgu]

Well, I'll post my speculations.
First, from the black cover of the device, it's emitting IR light. This is also confirmed from interviews.
4 ft^3 amounts to 50cm x 50cm x 50cm cubic area. 50cm / 0.01mm = 50000. It would seem that it would require 50000 pixel in the sensor.
But with sub-pixel interpolation they could be just using 5000 pixels. This number of pixels is common in linear CCD arrays.
With this kind of sensor it's possible to achieve very fast frame rates. My guess is that the device projects a structured 1-dimensional pattern with an slanted angle that is modulated by some mechanism, and it scans this pattern with an ultrafast linear CCD sensor to obtain the depth of the scene, 1 line at a time.

[budda]

Hi,

Here is a "hands-on" video with the Leap, from Wired.

http://www.wired.com/gadgetlab/2012/05/ ... er-tested/


Thanks.

[Chriky]

It must have two cameras to get a point cloud that accurate, then it must use some kind of structured IR light. Perhaps a kind of radar sweep line of light that scans very quickly. You could do that by surrounding an IR light with a cylinder with slits in it that then rotates, for example.

I've always thought that VR systems should have motion tracking cameras mounted over the user's eyes, purely so that you can look at your own hands and wiggle your fingers. This would achieve that amazingly, but not much else from a VR point of view given the limited range.

[zalo]

Well, I'll post my speculations.
First, from the black cover of the device, it's emitting IR light. This is also confirmed from interviews.
4 ft^3 amounts to 50cm x 50cm x 50cm cubic area. 50cm / 0.01mm = 50000. It would seem that it would require 50000 pixel in the sensor.
But with sub-pixel interpolation they could be just using 5000 pixels. This number of pixels is common in linear CCD arrays.
With this kind of sensor it's possible to achieve very fast frame rates. My guess is that the device projects a structured 1-dimensional pattern with an slanted angle that is modulated by some mechanism, and it scans this pattern with an ultrafast linear CCD sensor to obtain the depth of the scene, 1 line at a time.

How would they fit two Linear CCD arrays with an animated structured light projector into that dinky device? Moving parts?
It would explain how they pump all that data through USB though.

I still suspect that it's still two "camera"s with an IR projector no more complicated than an IR laser diode w/ Diffraction grating, or the falloff thing I mentioned earlier.

[seilgu]

> How would they fit two Linear CCD arrays with an animated structured light projector into that dinky device? Moving parts?
> It would explain how they pump all that data through USB though.
This is what I originally thought :
I think they need only one linear CCD array and a one-dimensional structured IR projector, and a spinning/vibrating mirror.
You know, like the device they use at supermarkets to scan barcodes. In these devices, they use a point laser source, and scan the light dot across the barcode with a vibrating mirror.
Similarly by using a laser source that projects a line itself, you can scan a two-dimensional area with a vibrating mirror.
Of course the CCD array must be synced with the position of the mirror, I don't know how hard that is.

> I still suspect that it's still two "camera"s with an IR projector no more complicated than an IR laser diode w/ Diffraction grating, or the falloff thing I mentioned earlier.
But later I saw a report, which says they're using off-the-shelf webcams
http://www.cultofmac.com/168432/this-ip ... ing-uncle/
so, essentially they're using technology similar to Kinect's, but perhaps with better algorithm.

[cybereality]

Engadget did a piece on it, video and pics:
http://www.engadget.com/2012/05/25/leap ... -hands-on/

After seeing that video I think I'm convinced its legit. No way they could have faked that.

Might just put in a pre-order, what the hell.

[seilgu]

Ha! I bet my guess this time won't be wrong.

It's called photometric stereo!

[brantlew]

This all gives me an idea for a similar system. If you took two Wiimotes and placed them a meter apart and then put one of those strong IR security camera spotlights between them all pointing towards the player, you could create an extremely accurate and responsive body tracking system. Unfortunately with Wiimotes you are limited to 4 points so the best thing to do would be to place reflective dots on your body. You could tie the whole thing together with a GlovePIE script. It would be similar to a Kinect. Probably more accurate but not as versatile.

[brantlew]

I went ahead and submitted an application for a Leap SDK using the FreePIE and FriiSpace projects as development targets. If they grant me a kit, maybe I'll be able to give you guys some details earlier than December.

[brantlew]

Hey, I just assumed my SDK application had fallen into a black-hole, but I finally at least got a form letter back from Leap informing me that my application is under review. They talk about the selection criteria and use this language

Developers with a focus on empowering other developers via frameworks, language bindings, libraries and such have an especially high priority.

Hopefully FreePIE might fall under that criteria because I would love to get my hands on an early release of one of these.

[PalmerTech]

If FreePIE does not qualify for that, I don't know what possibly could!

[WiredEarp]

It does look good. I wonder if you could mount it to a HMD for tracking your hands/fingers...

[brantlew]

It does look good. I wonder if you could mount it to a HMD for tracking your hands/fingers...

Those are my thoughts as well. Stick it on your face and then swipe your hands and fingers in front of you to mouse and menu navigate. Who knows, you might even be able to simulate a sort of keyboard and then be able to type a few commands without taking off taking off your HMD. The range is pretty limited though so it will be interesting to see how much that limits its potential.

[WiredEarp]

Actually brantlew that is exactly what I believe we will be using in the future instead of notebooks. Virtual screens and keyboards all sensed/generated by the same device.

[brantlew]

There was a laser system I saw about 10 years ago that projected a little keyboard onto the desk in front of you and sensed your finger typing. I never understood why it never gained popularity.

[zalo]

I'm guessing it had no sensory feedback. Typing on an iphone without haptic feedback is easy, because you are always oriented to the small screen's dimensions. Also, the keys expand when you touch them to tell you which key you hit.

I bet that it was just a static image of the keyboard with no visual interactivity like this one: http://www.thinkgeek.com/gadgets/cellphone/e722/

I suspect it will be the same for the leap unless you have visual feedback of what you are doing every second.

This almost certainly is not what you were talking about, but it demonstrates the same functionality: http://lightblueoptics.com/

Also, good luck brantlew on getting that dev leap! Be our leap ambassador and tell us how it works and its limitations!