Considering some possibilities of the Razer Hydra

[MSat]

This thread is sort of an offshoot of the open-source magnetic tracker thread, but is aimed at what might be possible with "non-standard" usage of the Razer Hydra.

I think many of us might agree second most important step (after HMD w/tracking) in VR immersion would be skeletal tracking - especially the hands. Imagine playing Doom3 with a flashlight in one hand, and a pistol in the other with 1:1 coordination. Imagine reaching out for a door, and opening it slowly with your visible virtual arm. Is this possible with the Hydra? I don't have one (but am seriously considering it), but here are some thoughts I've had about it (sorry if this has already been discussed)

The idea is to have accurate arm/hand positioning relative to the body using just a single hydra. Under standard usage, the base is placed in a fixed location. In order for accurate arm positioning to be possible, your body has to remain completely stationary. This is probably difficult even if you're seated. Since it's easier to track your arms relative to your body instead of a desk, what if you attach the base of the Hydra to your chest? That way, your hands can always be oriented correctly in the VR space even if you move your body. This may seem pretty obvious, but how well can you actually track and represent arm movements with just a single controller in each hand?

Well, you can model the physiological movements of a human arm, calibrate the software with the player's physical measurements, and/or pre-game hydra calibration by doing something like Vitruvian Man poses (maybe even a quick in-game calibration if it starts to drift). You should be able to extrapolate a pretty good representation of physical arm movements using just one Hydra controller in each hand.

I do wonder though how much the base can be moved around in an environment. Is there typically some distortion in the magnetic field? If so, would the changing of this field due to the base moving around significantly affect controller positioning accuracy. Does it even need to be calibrated like with joysticks?

So many questions because of so many possibilities

[brantlew]

Another thing I was thinking about the other day was modeling or augmenting skeletal tracking with inertial sensors. We all know that IMU's suck at maintaining position, but 9DOF sensors can be pretty good at orientation. So what if you used those stable angles and a skeletal model to constrain the position?

Take the example you just gave with the Hydra in your hands. You've got three points and are trying to derive the arm pose with just that information. There are probably a multitude of solutions that would satisfy those constraints. But if you augmented that system with an IMU mounted on each forearm, then you know the angle of the forearm in relation to the hands. Another sensor on the upper arm constrains the solution even more. So now you know the vector running through those IMU's has to intersect at the elbow so you can piece together the entire arm. In fact, given enough IMU's you might be able to track the entire skeleton in a stable way using just a few stable endpoints, driftless angular measurements on the bones, and a constrained skeletal model. (maybe)

[MSat]

I'm not necessarily stating this is the best method, but I think such a simple solution can still give good results. Try this experiment: Hold a fist out somewhere in front of you (not fully extended) and see how many ways you can move your arm without also moving your hand somehow. You might be able to do it a little bit, but it's uncomfortable and unnatural. Since the Hydra has 6DOF with absolute positioning, it should give you all the data you need to implement fairly robust hand/arm skeletal modeling.


Maybe it's time they make the Hydra Glove

[cybereality]

Yes, it can be done. Someone posted a video before with the Hydra base mounted on an HMD and it worked. Seems a bit unwieldy but I guess somewhat functional.

@brantlew: I am leaning toward something like you say. Basically a full body suit with IMUs at key points. I think with pressure sensors on the feet (to detect when a foot is planted and lock the bone) and some bend sensors on a glove, and you have the holy grail of VR tracking.

[brantlew]

Hmm. Well at the proper distance, I can swing my elbow through about an 80 degree arc without moving my hand in any significant way. So if you want to skeletal track the arm it seems like you would need a little more information. But if all that matters is the hand position, then I guess the arm model could just be estimated.

But that wasn't really my point. I was just trying to highlight that we might be able to augment a 3-point Hydra system with other readily available sensors to accurately track a significant portion of the skeleton.


@cyber: I'm starting to think the holy grail would be mesh of magnetic sensors and a single globally tracked origin point, but yes without a nice sensor package to develop with, we might still be able to approximate it using IMU's and other currently available devices.

[cybereality]

@brantlew: Well I was meaning something that could be made today with available parts (lets say for $1,000 or less). Obviously its not the ultimate solution (that would probably be markerless optical tracking) so maybe I used the term "holy grail" incorrectly.

[brantlew]

I'm agreeing with you. I think we could do something pretty cool for $1000 and current equipment.

Edit: Just for quick reference. You can get a batch of 10 Sparkfun sensor sticks for $900 and a Hydra for about $100. So all told, you would probably go over a thousand but I think we're in the ballpark at least. (maybe nrp's device would help?)

[MSat]

Ah yes, http://www.projectholodeck.com/oculus-rift-hands-on

I'm trying to figure if it's a bit odd to mount the base on the head, or totally brilliant. With the head tracker, and the hydra base attached to the head, you have a robust way to accurately detect head, and upper body rotations.

[MSat]

Now that I think about it, that whole setup is really solid. The combination of accelerometers and gyros for head tracking, the magnetic tracking base on the head, and the visual marker for a PSeye makes a very good motion tracking setup in the simplest, and most compact way (or at least potentially). You have all the major motion tracking that you need integrated in the HMD, the only external component necessary is the camera (probably need only one) pointing at the play space. I really think this makes sense to do for a model of the consumer Rift - it would make a tightly integrated and flexible VR package. I really can't see a better bang for your buck.

[WiredEarp]

Using gyros as brantlew mentions is a good idea. I do question how useful having full arm tracking is at this point however. Really, having a virtual hand adds a lot of immersion, but i dont know that adding a virtual arm is going to contribute anywhere near as much. It would be nice however. Even without proper tracking, you can just have a dummy arm that connects to the hand in basic ways... however, I can see that in the not too distant future, it would be nice to have fully tracked appendages - you could do stuff like elbow people etc in games!

[MSat]

@WiredEarp
Accurate arm tracking is definitely useful in the (near) future of wide-spread VR interactive media - maybe even mandatory for wide-spread adoption. It's almost like buying a gaming console that has no controller. It may not matter right this second, but going forward, there's a LOT of work to be done in VR/AR. It was never implemented in games, because there hasn't yet been the necessary hardware to do it. Aside from a good HMD, what well integrated VR peripheral would you want the most?


Limiting this conversation to arms/hands/upper body skeletal modeling, and absolute body positioning and location, I think the setup in the PorjectHolodeck link I posted earlier can't be beat for a few different reasons (at least until we get high-res, low latency Kinect sensors). Not only is it less expensive and cumbersome than having to place various gyros on your body, but more importantly, you can implement a very reliable, high accuracy sensor feedback loop that yields great relative and absolute position/orientation. Even if all things being equal except for using gyros instead of magnetic tracking, such an integrated feedback loop would be tough to accomplish. The only thing I think could be added to the PH rig would be accelerometers in the controllers. It's not necessary, but could significantly increase accuracy, it might even somewhat simplify computations. Well worth the slight increase in cost. As for the hands, finger tracking gloves would be great, but in reality, a controller in each hand is a lot more practical (has joysticks and buttons), and in some ways, can be more immersive -such as a previous example of holding a pistol in one hand and a flashlight in the other.

I wouldn't be surprised if Oculus is heading this route. If they are, I have to say it's absolutely brilliant! I would never have imagined being able to do so much with what turns out to be so little. I previously stated that a $600-$800 HMD wouldn't be worth the expense for most people. However, if you add a camera, and two controllers it just might be. In reality, I see no reason why it couldn't be much cheaper than that, even with a Hi-res display.