ControlVR - Full upper body and finger tracking

[BrandonJLa]

Hey guys, I'm Brandon Laatsch, I've been lurking for a long time but am posting for the first time today.

Awhile back I was approached by a few guys about using their inertial motion tracking tech for the purpose of VR. For the first time, they had gotten IMU sensors small enough to place on each finger and for the past few years they have been creating 20 - 50 thousand dollar solutions for mo-cap, robotics, physical therapy, etc... With the advent of VR and the price of sensors coming down, we feel that for the first time it is possible to get this tech into/onto everyone's hands. Thus, by combing the efforts of Synertial (for the motion tracking tech) and Inertial Labs (for mass IMU production) and myself plus a few others, we're nearing being able to bring the systems to the public!

Below is a video of the system working with an Oculus DK1. It features 19 IMU sensors working in a FK system to capture the upper body down to the fingers with 6DOF. One is on your chest, two on the upper arms, two on the forearms, two on the hands, four on the thumbs (2 on each), and eight on the remaining fingers. The sensors are accurate to a tenth of a degree and work great, we don't have to "smooth" the data at all resulting in a responsive and reliable track. Also, by rooting the system with the biped's pelvis and using the chest sensor to drive the upper body as well as using the Oculus' sensor to drive the head, we're able to give 6DOF to the DK1.

On the more fun side, having spent some serious time with my hands and fingers in VR, I can testify that the sensation of seeing your hands and fingers move accurately to reality is really special. The relationship and familiarity that we have with our hands is really powerful. I'm not a psychologist, but I'd theorize that our hands movement is more important to our sense of "self" then anything else due to the amount of time we spend seeing and using our hands.

The plan is to launch a Kickstarter within a month and deliver the dev kit by Christmas this year.

We'll be demoing the gloves at E3 on June 10th - June 12th, but I'm also thinking about holding an event at my office in Downtown LA in the next few weeks so that anyone local could come over and see it live, let me know if you would be interested / able to come over some evening.

I'd love to here your feedback as I continue to develop this tech.

https://www.youtube.com/watch?v=LPszKhewSec

[dokworm]

Amazing, so much better than my old powerglove

Any ballpark as to pricing on this? We are developing a VR and Dome product to help people with physical and cognitive disabilities, the extra immersion that having fully functional hands in the simulation would be ideal. I am a psychologist and you are spot-on with your comment about the importance of seeing your body, especially the hands, and having the virtual 'you' match the feedback from the proprioceptors in your body.

The lack of jitter on the positional feed is also really impressive, is there any potential interference problems with the tech?

Looking forward to more information!

Cheers

-Pete

[Davideus]

Wow ! Great to see someone finally making it !! thank you !

IGS Gloves is the best digital gloves around the world... I've seen a lot of them...
So you collaborating with Synertial in official basis?

I see that you are using synertials 12 sensor version of glove, there is also 16 sesor version. Maybe you think of expanding those for great immersive and precise experience of full hand tracking?

[amberbluefilms]

Hey Brandon,

I own a film studio in Hawaii called Amber Blue Films and I'll be at USC Cinema this summer. I am very intrigued by this device and your comment about how seeing your body gives you the greatest sense of "self." I am a big consciousness junkie and have studied the human mind on a philosophical level these past few months. I see this as a great step and consider this the discovery of a new "dimension," at which people are now able to experience an entirely new reality. Imagine what will happen when computer graphics will be even greater than human reality, being that we can overclock the amount of particle information from a simulation to give a person a greater omniscient sense of awareness. Anyways, I'll save that discussion for another time.

Speaking of which, I'd definitely enjoy stopping by one day while I'm in LA. I also see that there could be a huge documentary about this technology being made (you guys should look into that). Check out my personal site http://www.cagnation.com if you have the time,

Send me an email at christian@amberbluefilms.com

Cheers

[Mystfit]

This is brilliant to see! I'm super interested in what you've you're showing here since I've been attempting something similar for my Masters Thesis with teensy microcontroller powered bend sensors and gutted Razer Hydras.

What you've got going here though looks waaay in advance of anything that I've managed to get going. If you're planning on doing a kickstarter then I'd probably sign up first day.

[Kevin-Comerford]

Wow ! Great to see someone finally making it !! thank you !

IGS Gloves is the best digital gloves around the world... I've seen a lot of them...
So you collaborating with Synertial in official basis?

I see that you are using synertials 12 sensor version of glove, there is also 16 sesor version. Maybe you think of expanding those for great immersive and precise experience of full hand tracking?

Synertial is a large part of the development of this product. They have been developing motion capture suits for professional applications for years. The VR environment allows their work to expand to a larger market.

The version in the video is actually the 7 sensor version of the IGS glove. We are trying to keep cost down for the consumer version, so the fewer sensors the better. That being said, our interpolation is rather accurate and feels entirely natural. The 12 and 16 sensor gloves will still be available to professional market.

[Davideus]

There is clearly visible difference for example between 7 & 12 sensor versions. It's possible to correct this with software interpolation, forward kinematics? I'm a little skeptical for that..



Can you tell approx consumer price for 12 or 16 sensor if it can be made?

[Shelton]

Hey Brandon, long-time fan of your work (since around flower warfare days). I'm looking forward to the impact you'll be making on the gaming and video industries in your future.

I work in heavy industry robotics and wanted to tip you off to the phalange load cells NASA JSC has been using in their latest Robonaut rev. Though not necessary for motion tracking, you may want to keep them in mind for any future hopes of tactile input. NASA claims they are the smallest 6-axis load cells available, and they are open for commercial licensing. Check them out if you have a chance.

http://www.nasa.gov/centers/johnson/tec ... sensingCat - (Look for "Phalange Tactile Load Cells" under "Sensing")

I occasionally visit a couple augmented reality companies in your area and would love to stop by your studio to see this.

Andy

[cybereality]

This looks awesome. You have at least one buyer (me!).

[desther]

Good luck with kickstarter. You can count me in!

[simonion]

Looks awesome!
How's the latency?
When looking at the video it seems to be some latency to what appears on the screen but how perceptible is it in the rift and do you think you are going to make any headway with it before release?
Also wondering if you thought anything about a control scheme to use with gloves for movement while using these?
I think that is probably the thing that needs the most work right now since ODTs ain't really gonna make it for most people.
Do you think some discrete movements with your fingers could be used(depending on if the gloves are accurate enough to pick it up)?

Can't wait for the kickstarter!

[V8Griff]

Looks awesome!
How's the latency?
When looking at the video it seems to be some latency to what appears on the screen but how perceptible is it in the rift and do you think you are going to make any headway with it before release?
Also wondering if you thought anything about a control scheme to use with gloves for movement while using these?
I think that is probably the thing that needs the most work right now since ODTs ain't really gonna make it for most people.
Do you think some discrete movements with your fingers could be used(depending on if the gloves are accurate enough to pick it up)?

Can't wait for the kickstarter!

The video should look awesome those gloves are £16k each.

Hopefully the Kickstarter project will have the same accuracy and low latency for the low price being projected.

[Kevin-Comerford]

Looks awesome!
How's the latency?
When looking at the video it seems to be some latency to what appears on the screen but how perceptible is it in the rift and do you think you are going to make any headway with it before release?
Also wondering if you thought anything about a control scheme to use with gloves for movement while using these?
I think that is probably the thing that needs the most work right now since ODTs ain't really gonna make it for most people.
Do you think some discrete movements with your fingers could be used(depending on if the gloves are accurate enough to pick it up)?

Can't wait for the kickstarter!

The latency is very minimal and barely perceivable. There is also absolutely no optimizing being done in this video. The consumer version will be even better than what's shown.

We are already looking into the ideal control scheme. The demo shown already works with an xbox controller. You can actually see the controller sitting on my desk at the beginning, but the movement didn't make it into the video. The system is already good enough to pick up discrete movements such as a mouse click. You could potentially use small motions as inputs, it's all up to the developer. We already have a rock paper scissors game working.

[WhoIsRich]

This looks great!

Looks more practical than lots of flex sensors, like in Imogen Heap's recent glove while avoiding the occlusion problems that you get with camera tracked gloves.

Now it's just a question of pricing, I hope it's closer to the cost of a DK2 than 16k

[desther]

Now it's just a question of pricing, I hope it's closer to the cost of a DK2 than 16k

Yea, this has to be the lowest price possible, even by cost of lower tracking precision (lower amount of sensors etc) if the creator hope this to be supported by software developers and bought by mainstream users.

I think the sweet spot here is around 300$ for "lite" version (more options is a MUST here, just like with PrioVR). Of course the lower is price, the better.

Oculus Rift wouldnt be so heavily supported if it costs 2000$+ (and to be fair, even that would be multiple times lower price than competition back then).

[Zoide]

I agree, price is crucial. There's already a high-end market for multi-thousand dollar gloves (movies, military, medical, research, etc.), but the real breakthrough would be expanding into the mass market.

[zalo]

https://twitter.com/BrandonJLa/status/4 ... 2696361984

So under $1000... rather expensive.

Aiming for $350 is admirable. Just still not within the realm of fiscal feasibility.

[cybereality]

Honestly, if he solved the glove issue, I'd be willing to pay a decent price.

[Deop]

Looking forward to the kickstarter.
Price-wise I think anything beyond $150 will have a hard time reaching a large audience.

[rollerphreak]

I think this kickstarter project will skyrocket, since many are looking for this. Mainstream is the key here, since the more people get to "play" with it the more supporter it will receive.
Don't know but I feel we are leaping quite closer to the Matrix effect...

[Zoide]

Just one week until the E3 demo next Tuesday!

[V8Griff]

Kickstarter here:

https://www.kickstarter.com/projects/co ... on-and-mor

Impressive technology but surely something like this should work with other technologies like PrioVR for example , otherwise we're going to end up with so many peripherals but no cohesion?

[cybereality]

I backed it for $600. I hope they make it, gloves are a missing piece to the puzzle.

[mindless]

Would love to jump in on this, but unfortunatly i'm now totaly skint. gonna have to sit this one out, but it looks really promissing, hope they do well.

[GrantHnS]

Definitely the most impressive finger, hand, and arm tracking I've seen yet! Such and exciting time...

Definitely in to support you guys! Good luck on the KickStarter!

[desther]

This looks very nice. Good job!

[metronome]

I soooo want one of these they look amazing... $300 a bit steep for me atm... btw are there any plans for a more consumer / lower price version in the future? Just im wary that all these oculous vr type projects are still targetted for development studios....