6 DOF Head Tracking Ideas

[Aabel]

I've played around with a number of different options for tracking motion, and I thought I'd quickly share some of my musings.

My intent is to set up a vr system where:

* you can rotate freely with yaw (you don't have to always be facing a camera or something)

* (relative to a fixed point) acquire body position/offset in 3d

* accurately track relative hand movements (rotation and 3d position)

* accurately track head rotation

This is very cool CMS. My wife and I recently started a project similar to this with a few differences. We are looking at using Wii motion plus controllers for tracking hand movement however I am starting to have doubts about the accuracy of the Wii motion plus, it is also a rather difficult piece of hardware to work with on windows as all the libraries people have written haven't been touched in years and have known issues. How did you find using the PsMove? what bluetooth stack would you recommend for working with Move on PC?

[cms]

This is very cool CMS. My wife and I recently started a project similar to this with a few differences. We are looking at using Wii motion plus controllers for tracking hand movement however I am starting to have doubts about the accuracy of the Wii motion plus, it is also a rather difficult piece of hardware to work with on windows as all the libraries people have written haven't been touched in years and have known issues. How did you find using the PsMove? what bluetooth stack would you recommend for working with Move on PC?

Awesome, tell us more about your project.

It was annoying to get the move working, more of a hassle than the wiimote + motion plus. I actually did it through linux, so I can't really comment on getting it to work on windows. (I tried with windows for a little bit but didn't have success with it.) I had to tweak the code for a reverse engineered sixaxis driver. It wasn't too hard, but I imagine it's a bit easier to get going now. I haven't looked into it for a while.

This got me going:
http://www.pabr.org/linmctool/linmctool.en.html

And I thought this was cool:
http://www.eissq.com/ps3_move/

When I get going again, I'll probably just get some sparkfun razors, rig em up and cut out all of the extra annoyances of using moves/wiimotes (though it will be a bit more costly).
http://www.sparkfun.com/products/10736
http://www.sparkfun.com/products/10724

(If only they hadn't retired the all-dof http://www.sparkfun.com/products/10623)

I want to get Sebastian Madgwick's AHRS code working with them (I was using his stuff in my past experiments). It looks like the FreeIMU library http://www.varesano.net/projects/hardware/FreeIMU should work, but others have mentioned that it has some major drifting issue out of the box with the razor. I think I'll just start with https://dev.qu.tu-berlin.de/projects/sf ... i/Tutorial and rewrite a bit of the code to use Madgwick's.

[Nick3DvB]

Bit on BBC R&D about Headtracking, also using markers on the ceiling:

http://www.bbc.co.uk/rd/projects/holohelmet.shtml

[Sadhu]

Have you seen new Leap 3D demo? It looks really responsive.
http://www.theverge.com/2012/6/26/31185 ... e-controls

[cybereality]

Have you seen new Leap 3D demo? It looks really responsive.
http://www.theverge.com/2012/6/26/31185 ... e-controls

Thanks for the link (even if it is a little off-topic). This Leap looks amazing! This is exactly what I need.

[brantlew]

The Leap does look fantastic. The range is a problem since it only senses maybe 20 or 30 centimeters. When it first came out, I thought that you might be able to mount one on your back pointing upwards and then stick some protrusions on the back of your head. That would hopefully satisfy the range and although it would not track absolute world coordinate position, you could get very accurate movement relative to your torso. For many purposes that might serve just as well.

I sent in an application to that company for a pre-release development kit about a month ago, but they never responded to my request.

[Krenzo]

This looks like a nice chip? I guess all we need is a small interfaceboard with this chip on it and we would have a very nice 'headtracker' ?
http://invensense.com/mems/gyro/mpu9150.html

The specs looks great.

That is a great chip, and I think I will implement it into my own system since it's so small and cheap.

After looking over this thread, it appears that optical tracking is the favorite here, which makes sense from a ease-of-implementation point of view, but it's not without its limitations. Has anyone experimented with magnetic or RF implementations? I have put all of my effort into RF tracking but don't have anything to show yet.

[cadcoke5]

Since all orientation systems have weaknesses, I imagine the main need is not a particular measuring technology, but rather software that can access a variety of sources and integrate them into one. I.e. it sees the inertia sensor thinks we have turned our head to the right by 25 degrees slowly over the last 10 seconds. The optical system last reported our position as being straight forward, but that was 20 seconds ago and it apparently has lost sight of its target. What is the real direction our head is facing?

The software would have to make a decision, and of course be able to read the output of those sensors. Not necessarily be image processing, but at a higher level be able to read the outcome from the video image processor target tracker, and the inertia sensors, and gyro, etc, then integrate the various sensors to make its best guess about where we are.

Joe Dunfee

[brantlew]

With unlimited resources you can't beat high-resolution multi-camera setups. For simpler setups, a lot of sensor fusion heuristics are required but they have a lot of trade-offs and limitations. The LEAP is interesting as an optical technology - assumedly similar to the Kinect. I am curious if it is also range limited in principle or if they simply chose a close range device for cost and practical manufacturing reasons.

RF or magnetic setups seem promising in principal. I don't have any practical experience with them, but they seem prone to a lot of noise issues and a difficult signal processing problem. The Hydra is a nice proof-of-concept but I wonder how far you range that technology safely and reliably.

[Chriky]

@ Joe

Sounds kind of like a Kalman filter - http://en.wikipedia.org/wiki/Kalman_filter
Read the example on that article - http://en.wikipedia.org/wiki/Kalman_fil ... pplication

[brantlew]

@Chriky: I haven't delved deeply into Kalman filters yet, but what I gather is that they are only as good as the motion model that you are able to put together. So for example in aeronautics they work great because flying projectiles tend to follow very restricted and predictable trajectories. But in general, I don't know how well they work for something like real-time human motion - it being potentially so sporadic. You can probably limit the signal noise a bit since humans operate under inertial and speed limits but maybe not well enough for 6DOF head tracking. Still they are definitely a piece of the giant sensor-fusion puzzle.

[Krenzo]

With unlimited resources you can't beat high-resolution multi-camera setups.

Yes, optical works great if you have the money to spend on fast, high resolution cameras. However, the one place where RF has an advantage (besides lower cost) is that it can penetrate objects that would occlude in an optical setup. It experiences some degradation when going through the human body, but it can still be received on the other side. Non-metallic objects are passed right through without any attenuation. You could fill your space with wooden/plastic objects to interact with and have no occlusion problems. For example, having a plastic rifle to wield as your gun with sensors attached won't occlude any sensors on your body.

I believe magnetic systems have the same advantage, but I haven't experimented with them.

[brantlew]

@Krenzo: So just out of curiosity, what are your experiences with RF? I guess at a small enough wavelength you could (in theory) differentiate some really small motions at a very long range. There must be a bunch of practical issues with it though. Would you use something like a phased array transmitters and a bunch of antennas placed on the body - something like a mini-GPS? Or do you try to bounce signals off of metallic markers or something? I guess it would be subject to all the same noise and multi-path problems that plague all radio. What are the practical limits of precision for it?

[Krenzo]

@Krenzo: So just out of curiosity, what are your experiences with RF? I guess at a small enough wavelength you could (in theory) differentiate some really small motions at a very long range. There must be a bunch of practical issues with it though. Would you use something like a phased array transmitters and a bunch of antennas placed on the body - something like a mini-GPS? Or do you try to bounce signals off of metallic markers or something? I guess it would be subject to all the same noise and multi-path problems that plague all radio. What are the practical limits of precision for it?

I've been developing an RF tracking system for about 2 years now, didn't really start making progress until a year ago. The resolution is very good, ~1 mm, limited by how precise and calibrated your electronics are and your frequency. Mine works at 3.1-6 GHz. The setup is a bunch of transmitters surrounding a space where you wear receiving antennas on your body. Bounced/multi-path signals are not a problem because you only sample the first signal to arrive which is always a straight path. The transmitter is not sending a continuous signal like radio. Imagine it like an ultrasonic pulse but much faster. Google "ultrawideband". Range of the system is limited by the FCC's limitations. Right now, I have about 20-30 ft of range with one transmitter, but with the cost not being as high as cameras, I plan on having many transmitters surrounding a large area to operate in. Some systems I've heard about operate in the 20 GHz band with sub-mm precision, but then they don't pass through objects as well. The complexity of the system also increases (as if it wasn't complex enough)!

[Chriky]

What sort of costs are you looking at? Is this a hobby or part of your job?

Sounds pretty good. Occlusion is not a major problem in a totally empty, large area like a basketball court but I think it would have a lot of problems trying to track people in a more real environment like a house. Perhaps you could build the environment out of clear materials, but it would be much easier to use RF or magnetic fields.

[brantlew]

That sounds fantastic!! I think 1mm precision would be sufficient for just about everything - head, body, and props. Well I could maybe see using a local gyroscopic head tracker just for the added precision but it wouldn't be strictly necessary. Sucks about the FCC limitation for commercial sales, but I guess a home-brew unit that ignored the FCC could go pretty far There is some cool stuff you can do with redirected walking once you get about 100 ft or so. Is this intended for commercialization or is this just a side project? I would be super interested in either.

PS. I've been blasting signals from my home distribution amp out my HDTV antenna for years now - so suck it FCC!

[Krenzo]

I'm working on this myself as a hobbyist. I hope to sell it, but the price will keep it out of range of your average consumer. Ultimately, I would like to have some sort of laser tag setup where normal people can pay by the hour for time in it.

[brantlew]

Well it sounds really interesting. How long have you been lurking here? You should consider posting a project thread in the DIY section. I would love to keep tabs on this. There are at least four of us around here working on natural motion VR systems and some cross-pollination might be beneficial.

[Krenzo]

I've been lurking for 4 years or so, just came back recently with all of the articles about John Carmack and PalmerTech's Rift. I'll definitely make a thread once I have something decent to show which should be whenever I get all of the bugs worked out of my latest board revision.

[brantlew]

Cool - looking forward to it. One last question. Assuming you could transmit with as much power as you wanted, what are the practical range limits of the technology. Could you detect motion < 1cm at 100m ? Is transmission at that distance practical and if not are the limits technological, cost, or just regulatory?

[Krenzo]

Cool - looking forward to it. One last question. Assuming you could transmit with as much power as you wanted, what are the practical range limits of the technology. Could you detect motion < 1cm at 100m ? Is transmission at that distance practical and if not are the limits technological, cost, or just regulatory?

Here's a paper where NASA did tests for a long range UWB tracking system. It says it's FCC compliant. The increase in distance is due to them using horn antennas with a narrower/more focused transmission beamwidth. It doesn't look like their accuracy is that great.

You could increase the range by decreasing the FOV of your antennas like in that paper, but then you need more antennas to cover the same area. If you were to increase the power, then you increase the complexity of your transmitter as you have to split the signal into multiple copies, amplify each copy a little bit, and then combine them all back together to get a single signal that is more powerful than the original. That's not easy to do with UWB signals, and then you'll be over the FCC limit.

If I could increase the power of my system, then I don't think I would have any decrease in accuracy as distance increased, but that's just a guess.

[brantlew]

Were you guys aware of this? Am I just late to the party with this info? I know it's just Source games but that's pretty cool having direct access to the camera via an SDK! I've been looking for an easy platform to test full tracking with for a while (beyond just rolling my own Unity or Ogre project) and this seems perfect. Kudos to Valve!

https://developer.valvesoftware.com/wiki/Head_Tracking

[PatimPatam]

Hello there, I’ve been redirected here from the Oculus Rift thread by Brantlew (thanks). I've read all the posts on this one and couldn't find a clear answer so I'll shoot!

I think full body motion tracking/positioning is a really interesting subject but as we all know it can get really complex, and even if all the technical issues could be sorted out it would still not be very practical in a home environment..

I'll try to simplify a bit and limit the scope of the problem to playing a FPS game, sitting down on a fixed spot and using a gamepad to move the character's body, run, jump, shoot, etc (like in Carmack's E3 demos).

So it seems like even in this relatively simple configuration the issue is still being able to track the HMD/character's eyes position in relation to the center of the body, so we can do small strafe movements with the head/torso (to dodge fireballs for instance) or tilt slightly forward to look around corners, etc. Small details you may say, but something that in my opinion could make the whole VR immersion work (or fail).. All this because apparently gyroscopic sensors like the Hillcrest are no good to track total positioning.

In this context could we not simply attach a few stickers/leds on the Rift and use an optical device like Kinect to get a pretty accurate x,y,z eye position, while letting the Hillcrest tracker get the pitch-yaw-roll of the head? As I understand Kinect has a pretty low latency if you ignore the whole skeletal tracking (not necessary) and just get the depth map and the rgb image. You could tell the user to sit still before starting the game to get the initial position, and then just map the new relative positions with the game engine..

Could this not be a fairly simple solution where we would still get a much more immersive response than with a single gyroscopic sensor? Or am I missing something? Has anyone actually tried to integrate something similar in a working engine?

And even if it is not as simple as i make it sound.. should we not focus on solving this before anything else?!

[brantlew]

In a nutshell - yes. Optical tracking is probably the most straight-forward solution to the 6DOF problem. Take your pick - Kinect, Move, webcam, or even Wiimote. Any of these could be retooled for fixed-position 360 degree tracking - assuming you are willing to setup and calibrate an area with multiple cameras or markers. The proof-of-concept is in cinema and game motion capture which all employ this technique. It's just that nobody has really successfully completed and packaged this solution yet for general consumption. Maybe because all the necessary and inexpensive equipment has only recent become available? The Holodeck project looks like a promising experiment into this and there are few small projects scattered within this forum and academia that try to address it. But nothing yet has really made it out of the "lab" into user's hands yet. But it feels like things are starting to accelerate in this direction.

That's what Carmack is talking about. We're at an inflection point and now with immersive HMD's finally within our grasp, people are going to start going wild with VR applications. And a few hobbyists can make a huge difference

[PatimPatam]

Thanks again brantlew, I understand, but my point is: in the simple configuration i described before (which for me should be the starting point for VR games) where no 360 degree tracking is needed, should a single Kinect camera not be enough to determine the eye's relative 3D position to the initial point? Which in turn would simplify a lot the coding/integration?

There could be markers on the sides of the Rift as well, so position could still be determined when not facing forward..

[brantlew]

Without the need for 360 tracking, there are already optical solutions like TrackIR and FreeTrack that can accomplish this. A few flight sims already support 6DOF cockpit views this way. But there are some hard limits on the yaw angle that it can track.

Placing markers around the head is an idea that has been bounced around a good bit, and I'm surprised nobody has implemented that yet. It does make the problem a bit more complex, but if you augmented the algorithm with a gyroscopic sensor then it seems like a very workable solution to me and would fix the yaw limit.

There is a small issue with the Z-axis on front facing cameras though. They have to use perspective transforms to calculate the Z coordinate which is less accurate than the X and Y axis which they can retrieve more-or-less directly. So your side-to-side head motion will be tracked more accurately when facing forward than when looking at say 60 degrees off center. I don't know how much of a problem in practice this would be but it's there in principle.

I think a 2-camera setup with a wide stereoscopic view of the player would be a good solution. Triangulation of the markers would provide far more accurate results in all dimensions. Combining that with a "ring" of markers on the head and you could build a pretty damn good tracker (says me).

And for completeness sake - you can also invert the solution. Mount either a mono or stereoscopic camera (Kinect) on your head and track the environment instead. There are advantages/disadvantages to either approach.

[PatimPatam]

Cool sorry for my ignorance, I didn't know about TrackIR and FreeTrack.. I can see that the TrackIR 5 supports up to 120 FPS and it's "only" 150$, not bad.. and FreeTrack is an open-source Library for Windows that supports TrackIR, perfect match!

The question that comes to mind is: why are Carmack and Palmer playing around with gyroscope sensors like the Hillcrest then, instead of with something like the TrackIR 5? (or in combination with it). That would solve many of the problems that the prototype E3 demo had, as Carmack himself admitted in some of the interviews.

I can see why a solution similar to the TrackIR has not been used in FPS games until now, but I think that in a VR environment it makes a lot of sense. In fact I believe it could work really well, and is something that we could enjoy right now (without too much trouble it seems) and would set a standard for future FPS games VR integration.

Anyway I don't think that Carmack has the time to read this forums anymore.. I guess I could try Twitter. Or if you go to the Quakecon VR panel maybe you could ask him yourself Can we expect 6 DOF support via optical sensors for Doom 3 BFG / Doom 4!?

[brantlew]

My guess is because it's just simpler to implement and integrate a gyroscopic tracker quickly for the demo. For 3DOF, gyroscopic trackers are really great solutions. They are accurate, responsive, flexible, and require practically no calibration. You just plug it in and go - perfect for a quick demo on the road.

Also from the get-go, the Rift was designed as a 360 experience. That's why he's shipping the kit with a wireless HDMI transmitter. So you can be unbound by cables and spin freely. With that in mind there is no optical technology that can support that right now. So it's simpler just to throw a tiny little gyro tracker in with the kit as a reference design than to try and whip up and test a multi-camera solution. It's up to us now to work that out. If by next year, someone hasn't implemented a multi-camera and/or multi-marker version of FreeTrack then I may just take a crack at it.

[PatimPatam]

Thanks for the reply Brantlew, it makes sense.

Still I'm quite surprised that I seem to be the only one around here that thinks 6 DOF optical tracker support for Doom 3 BFG would be the way to go, even without the 360 degrees freedom (in a controller-in-hand, sitting-on-the-sofa setup). Basically the controls would be exactly the same as when you play the console version, with the only difference that you would BE THERE.

After that is really working then we could move on to the next step, like integrating the Leap or the Hydra to replace that ugly xbox controller

[brantlew]

Well it would have been great if Carmack put "hooks" into the game so that we could plug into the camera and/or player model and control it via third party control devices (and he may still), but I can understand why he would not expose that type of functionality without a reference design to test with. So even if we do come up with good 6DOF trackers we have a serious lack of software support. There's no interface for general camera translation like there is for camera orientation (mouse). You can sort of "hack" it by tapping into the rendering pipeline and offsetting the camera (see cybereality's 3D "roll" driver) but it's not the best solution. So we're left with either custom and open source game engines or potentially Valve games (that's why I got so excited about 10 posts back when I found the Valve SDK!). Honestly that's probably why nobody has pursued this heavily. Even if you built a fantastic 6DOF tracker, what can you do with it? Research mostly. But for gaming you're stuck with flight and car sims.

[PatimPatam]

That's precisely why i think it would be great if Carmack (or someone from ID) tweaked the engine to support this type of optical tracking and camera translation input (available right now with a commercial product like the TrackIR 5). Then at least we would have one great example of how 6 DOF can work with FPS games and a HMD. If this was a success like I think it would, then other future games and engines would support it as well.

It may be a bit too much to ask, but I think now is the right time to do it!

[brantlew]

Well if the Valve SDK actually works the way they say it does, then we can implement perfect tracking examples in some pretty good games. Just need the Rift warping drivers...(hint hint Cyber or Emerson)

[cadcoke5]

While I have generally thought camera tracking was the way to go myself, I know that one of the main problems with that technology is latency. You have to get the image from the camera over a USB port, and then process the image.

Probably the ideal version is an integrated camera and processor, like what the Wii remote is, but with greater resolution and field of vision. Then there would have to be some higher level connection between multiple cameras to integrate the results which could probably be done on the main computer.

Joe Dunfee

[cybereality]

You can sort of get 6DOF with intercepting DirectX calls, but it comes with problems. Mainly I have not found a way to alter the view frustum yet, so objects and areas of the screen are clipped and/or culled prematurely. In theory you should be able to move the camera's position (which is what the stereo drivers are doing) but you can only move so much. For example, if you tried leaning around a corner, the game would probably think the area beyond the edge was not visible and thus would not be shown. But I am still investigating this scenario.

[android78]

You can sort of get 6DOF with intercepting DirectX calls, but it comes with problems. Mainly I have not found a way to alter the view frustum yet, so objects and areas of the screen are clipped and/or culled prematurely. In theory you should be able to move the camera's position (which is what the stereo drivers are doing) but you can only move so much. For example, if you tried leaning around a corner, the game would probably think the area beyond the edge was not visible and thus would not be shown. But I am still investigating this scenario.

I think the problem is back face and occlusion culling, which is handled by the application, not the graphics engine.
Oh how I dream of a world where all games have support for this implemented!

[PatimPatam]

While I have generally thought camera tracking was the way to go myself, I know that one of the main problems with that technology is latency. You have to get the image from the camera over a USB port, and then process the image.

Well I haven't been able to find out the exact latency of the TrackIR 5 after image processing but 120 Hz refresh frequency is promising. From all the reviews i have seen it seems to be very low.

The fact that none of the reviewers complained about latency is a good sign as well, especially if you take into account that people use it mainly for flight simulators and driving games.. a type of gamer who tends to be very picky with this sort of thing.

It would be interesting to know if it could get the Carmack low-latency seal of approval

[brantlew]

Chriky has some practical experience with camera latency with his little project. He claims his image processing can mostly keep up with the raw Move camera feed of 187 FPS. I don't know what the initial latency is but I suspect it's pretty low. Maybe not Carmack-low but he's aiming at perfection. I think you can still have some pretty profound experiences running even twice as slow as the Carmack-rate.

http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=15064

[PatimPatam]

Oh how I dream of a world where all games have support for this implemented!

OK I've just sent a PM to Mr Carmack mentioning this thread and asking for 6 DOF tracking support in Doom 3 BFG / Doom 4..

I think it's a valid request if he really wants to push VR forward, so, you never know

[Nick3DvB]

Just been catching up with this thread, a lot of great ideas in here. Optical still seems like the way to go but RF sounds really interesting to, the more options we have the better really, which got me thinking about ultrasonic range sensors, has anyone done any work with that? You can get dirt cheap "parking sensors" that are supposed to have ~10m range with 1mm accuracy, so in your typical sized (square-ish) domestic room couldn't we use a couple of them at 90 degrees and calibrate from a known starting position? A good yaw reference still seems to be a major problem, or are there any optical / electronic "compass" sensor packs that can do a decent job? I'm still getting up to speed on all this, and really don't have the math abilities to work out the integration etc, but I just wanted to throw ultrasonic sensors into the mix as they might be a good fit for certain environments, any thoughts guys?

[brantlew]

@Nick3DvB: I don't know anything about them, but I've always been curious about them myself. It seems like they would have a lot of the same good properties of RF, but without the government regulations. I'm not sure how much of an issue occlusion and multipath would be, but it seems it could be a fruitful area of research. If you find some good info, you should post it.