6 DOF Head Tracking Ideas

[brantlew]

I broke this discussion off from the Oculus Rift thread. The subject is: problems and solutions for retrieving accurate X,Y,and Z coordinates of the head.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 327#p74295

Here were some posts:

JohnCarmack: I've seen you mention in several videos that you haven't gotten translation tracking to work properly with the unit. One thing I wanted to try with it was to use a Playstation Move or similar device to optically track the 3d location of the unit and try to use that in game. Have you already tried that and found out that it doesn't work for some reason?

@JohnCarmack - It sounds like you have made significant progress in this area, but it leaves me wondering, from your experience, in what ways is motion tracking currently lacking? It sounds like front-to-back, side-to-side translation does not currently exist. How much of a detriment is this to the experience? What sort of movement tracking do you believe is important for the immersion factor?

I’m really glad you guys raised this, John has touched on it several times in interviews and I think that being that able to demonstrate parallax simulation effects would give a massive boost to the whole project; in fact it could be your "killer-app". I don’t underestimate the importance of the FOV improvements, that’s a massive achievement in itself, but combine that with the potential of the tracking implementation and I’m really starting to get excited, so forgive me if I go on a bit. Firstly I have to admit I’m a complete HMD noob, I tried one in a Uni lab about 15 years ago but the “floating TV” effect and tiny FOV meant I never really caught the bug. I’ve played around with some TrackIR type kit, but like many I was happy with my projector & shutter specs, and have been patiently waiting for those “holo-deck contact lenses” to arrive ever since…

Whilst the current gen console’s refresh cycle may arguably have robbed us of years of graphics innovation, (those expecting another UE3 / IDTech4 quantum-leap may be disappointed), they have gifted use one thing- mass market adoption of tracking sensors, and that could be SO much more important in the long run. Many have said that we are nearing a tipping-point, the convergence of technologies that will deliver a real game-changer, but we’re not there yet, something is still missing, it reminds me of that quote from the matrix:

You're here because you know something. What you know you can't explain, but you feel it. You've felt it your entire life, that there's something wrong with the world. You don't know what it is, but it's there, like a splinter in your mind…

There IS something fundamentally wrong with “3D”, everyone here knows it but many of the wider public still don’t understand why. Move laterally whilst viewing a stereoscopic image, something very strange happens, your brain is not getting any of the extra information it expects, from “behind objects”, it tries to compensate, it fails, and the illusion is broken instantly. I think people coming to HMDs for the first time really need to understand that head-tracking is NOT just another control system; the feedback it provides for optical effects, like parallax simulation, are a fundamental part of emersion in the 3D space. But they’ll have to see it to believe it.

I’m sure many of you played with Jonny Lee’s Wiimote VR demo a few years back, I have to admit I spent many hours pacing around in front of my projector screen wearing two old TV remotes taped to a baseball cap! I never quite got it working properly with stereoscopic 3D, I played with TriDef but the geometry was always off somehow? When the Kinect arrived there was a stream of similar projects, but still no sign of the killer-app we were hoping for. Autostereoscopic / holographic displays, and plenoptic / light-field imaging systems are all going to feed development in this area, sadly they are still a long way off, but surely WE can do something with this now! There’s no doubt retro-hacking parallax simulation it into current games is an immense challenge, if John is struggling with it, what hope us mortals? But if we could get a few basic demos together at launch to showcase these parallax effects it could have a huge impact on public awareness, when people actually see it working it will blow there mind...

You can probably tell that I’m way out of my depth here, so my terminology is probably all wrong, but if we can demo the strafe “peek around” type effects (dynamic occlusion?) or maybe just something done with lighting of static objects, have reflections distort dynamically or specula highlights move along edge surfaces realistically as the viewer moves their head position (whilst standing still, not just as their body moves through the 3D scene). I hope I’m making sense, hopefully those with a better grasp of basic physics and more knowledge of rendering process can articulate this better. I’m sure most of you have seen all this before but for the uninitiated here are some great videos of parallax simulation in action:

http://www.youtube.com/watch?v=Jd3-eiid-Uw
http://www.youtube.com/watch?v=BduSDvUU6MY
http://www.youtube.com/watch?v=8SDGG9HhbgQ
http://www.youtube.com/watch?v=1dnMsmajogA
http://www.youtube.com/watch?v=6tuizfOcdLQ
http://www.youtube.com/watch?feature=pl ... a5NQK563OI

Also, some basic DOF simulation in Quake III and other really cool stuff:

http://www.youtube.com/watch?v=HdW1v9TPNYw
http://www.3dfocus.co.uk/glasses-free-3 ... 3d-tv/8626
http://www.3dfocus.co.uk/3d-news-2/3d-t ... ature/6695

[/END RANT]

You guys keep up the good work, I'll keep spreading the word!

The technologies for tracking accurate head translation are more problematic than the ones for orientation. Gyroscopic orientation sensors operate within a local reference frame which makes them more generalized for a wide array of applications and immune to scaling problems. They are a near perfect solution for orientation tracking.

Local inertial sensors on the other hand have enormous error ranges when used to estimate translations - on the order of meters per second error!!! So to track translation you have to use an external reference. The Hydra uses magnetic fields and does a pretty good job, but most systems are based on optical technology - thus the Johnny Lee tracker, TrackIR, etc.. You can get really great results with these systems but they suffer from scaling problems - the sensor and reference points have to be within range, and in the case of optical there is a problem of occlusion. Not that these problems can't be solved, but it requires more complex setup and calibration (multiple cameras, light sources, etc) than the simple gyroscope used for orientation. So you don't see setups that include head translation as often because they don't fit in a neat little package, instead requiring a whole specially designed and calibrated play area.

I'm currently trying to make a headtracking system that uses the PS3Eye. I think it's a good piece of hardware to use because you can buy them for £11 delivered from eBay; much cheaper than most other pieces of kit. They can output 320x240 at 187FPS using the free driver from Code Laboratories, it supports up to 2 cameras (there's a paid version that supports more).

320x240 seems like a small resolution but bear in mind the Wiimote camera only has an actual resolution of 128x96, but by averaging several pixels' intensities it gets an effective resolution of 1024x768.

Anyway my basic idea is to put a camera on the head, looking up at the ceiling which will have a sort of poster with a known set of coloured dots on it. You can use the 2D position of the dots on the screen, and their known position in 3D space to work out the position and orientation of the camera (this is called the PnP problem, its not hard and there are loads of very fast (a few microseconds) algos out there to solve it). You only need 4 points to get the position and orientation. The hard and potentially slow bit is identifying the dots in the camera image.

The best thing about this system is that it can scale arbitarily; you can cover a whole ceiling in the dots, combined with some way of getting a global positon (say, QR codes every so often), and then you could walk around freely with a backtop system. What's more, several users could use it simultaneously with no extra work.


http://i.imgur.com/gqU9f.jpg

Still very early stages at the moment, but I've got it working out 4 dots' screen positions. As in, the coloured circles on the screen are drawn using actual (x, y) double coordinates.

@Chriky: This system has some angular limits but in general I think it's a good idea. It addresses the range and scaling problems nicely and putting the sensor equipment on the player and the markers in the environment is much cheaper than the other way around. One of the users on this forum, smaesen, has done some extensive academic work very similar to this. He uses cheap LED light ropes on the ceiling as markers.

"Scalable Optical Tracking - A Practical Low-Cost Solution for Large Virtual Environments"
http://research.edm.uhasselt.be/~smaese ... tions.html

[Chriky]

Ah yes thanks for the link I remember seeing that a while ago but couldn't find it again. Good idea breaking off the thread. Just on my phone at the moment I'll make a proper post when I get back home.

[Nick3DvB]

Thanks for your input guys, sorry to go off-topic, tracking is obviously a huge area of research in itself, I’m sure many here will be working of there own ingenious solutions, but in a way that is part of the problem! We are inclined to treat it as a completely separate problem and so produce fragmented solutions tailored to our own needs. Something like the Hydra or a simplified optical DOF6 system is certainly something to think about for the Rev.2 kit, but I’m more interested in the limitations of the bundled Hillcrest package that will be supplied for use with Doom 3 BFG (and hopefully Quake4 & Prey at some point!) because this is the sensor most people will be using, and it could define the experience in terms of the wider public.

I’m sure that if John could have implemented perfect parallax simulation he would have done it, but I’m not clear if this is purely a sensor limitation or other modelling issues specific to Doom3? With the exception of Doom 3 most games will be stuck with basic tethered “mouse look” control model, so I think it’s important to have a range of carefully designed tech demos to showcase the other possibilities, to try and de-couple “control” from the "optical effects" produced by subtle head-movement, and to do this using the bundled sensor package if at all possible. Without knowledge of it’s limitations, a couple of possible examples:

- Someone viewing a reflective object from a foot or so away (something like an intricate silver candlestick) would see large changes in the movement of specula highlights over its surface etc, maybe some could just mod that excellent RTHDRIBL demo!

http://web.archive.org/web/200812312008 ... /rthdribl/

- Someone standing on top of a very tall building, a densely packed urban area bellow, small changes in head position (only) are amplified to produce large changes at ground level (in terms of occlusion etc). Maybe something being viewed through a few layers of Half Life 2 style chain-link fencing?

If anyone can think of any better ways to demo these types of effects then I be really interested to hear it. If the Hillcrest sensor really isn’t up the job then we should still explore other options, I was hoping the Wiimote cam would do the trick, it seems sensitive enough to small strafing movements in the Jonny Lee demo, and I was intending to use a wii-zapper as a controller with Doom 3 anyway, so having a second one as a camera with a few LEDs / markers would be my preferred option, but the PS3 cam / gun combo looks really interesting to.

[Chriky]

I think using lights is a better idea that what I'm currently doing with a printed pattern. When running at high FPS, the v small aperture means the image is naturally quite dark, and you have to boost it a fair bit (introducing a lot of noise) in order to detect the points. I was going to just mount a bright LED torch. I'll probably try both ways

The Eyes are so cheap that it would be reasonable to put a few on an HMD to increase the effective FOV. One looking up and forward and one up and back would be about 140deg coverage which I think is enough. However, if you are making a holodeck-like room, I would just continue the pattern onto the walls as well. There's nothing special about the ceiling; as long as you can work out which set of points you are looking at, you can get their real world location and then get the camera pose. The ceiling is just the best choice because 99% of the time, the top of your head is pointing (roughly) up.

There are however very good SLAM algorithms out there that track interesting points on the fly and work out a pose from them. This top video - http://www.laurentkneip.de/research.html - is very impressive. It combines the video feed and inertial readings together and looks very robust. With a decent gyro to fall back on, it would be very good (it's current problem is tracking fast turns), but I don't know if it would be too CPU intensive. It also seems like making the problem harder than it needs to be, because we have a certain amount of control over the environment.

http://publications.asl.ethz.ch/files/kneip11novel.pdf
http://publications.asl.ethz.ch/files/kneip11robust.pdf

What I like about calculating it directly from known patterns, is that there is no 'tracking' element. It is basically a function that goes straight from an image to a global pose. There's no possibility for it to drift off track.

My ambitious design was basically going to be one camera on the head, one on the back, then Hydra base also on the back with one controller in each hand. This would be wireless and give global pose of head, torso and hands independently, which is probably the minimum needed for a completely immersive system. I'll probably first go for the slightly less ambitious system of a back mounted camera and using one of the Hydras (stripped open) for the head. Your torso really is almost always vertical so as long as the camera avoids your head I think it would work.

@ brantlew

Have you thought about using the PS3 Eye for FriiSpace? I recon running it at 640x480 and 60FPS would be really good for you. It has a wider FOV and higher resolution than a Wiimote. It is really not hard to convert it into an IR camera, I've done it before. The code for working out the position would be very, very easy (about 5 lines).

[brantlew]

@Nick3DvB:

I was hoping the Wiimote cam would do the trick, it seems sensitive enough to small strafing movements in the Jonny Lee demo, and I was intending to use a wii-zapper as a controller with Doom 3 anyway, so having a second one as a camera with a few LEDs / markers would be my preferred option

This sort of thing is very doable and is the way that TrackIR/FreeTrack work. But both of those implementations are limited to a forward position and limited head movement. Cockpit simulator guys use this to their advantage and there is existing software out there to demo cockpit simulation with 6 DOF head tracking. But FPS games generally need more freedom of movement - ideally with 360 degree turning. Setting up a Johnny Lee-like system for that requires either cameras or lights surrounding the player. So it's not that it can't be done or that it's particularly difficult, but it does require a dedicated person and a pretty specialized setup. And it's not as easy to bundle with a kit system. A person that is willing to setup and calibrate all that stuff probably does not need convincing of the importance of 6 DOF. Carmack did mention that he had tried to integrate the Hydra to provide this sort of thing, but there were some precision problems with it.

My ambitious design was basically going to be one camera on the head, one on the back, then Hydra base also on the back with one controller in each hand. This would be wireless and give global pose of head, torso and hands independently, which is probably the minimum needed for a completely immersive system. I'll probably first go for the slightly less ambitious system of a back mounted camera and using one of the Hydras (stripped open) for the head. Your torso really is almost always vertical so as long as the camera avoids your head I think it would work.

I'll be interested to see the results of your experimentation. Especially with the Hydra. In theory it sound like the perfect device to track body parts relative to the local reference frame, but Carmack mention that it was problematic and I tend to accept his judgements.

[Chriky]

Do you have a link for Carmack talking about the Hydra? I used it for headtracking before and I didn't notice any serious problems.

[brantlew]

I don't have the link, but I think it was in one of the 3 or 4 E3 videos that I watched of him.

[cybereality]

Yeah, it was in the GiantBomb video I think ( http://www.giantbomb.com/e3-2012-john-c ... w/17-6164/ ).

Basically he said that the absolute positioning/orientation was not accurate, and that after a while you would end up in a different spot than you were. To be honest, I have not noticed this myself but I haven't done the hardcore testing I'm sure JC has. In my mild tests it seemed pretty spot-on, definitely better than other stuff I've seen in the past. Still not perfect, I would get short range issues and interference. But when it worked the Hydra seems decent. I played through Portal 2 using only the Hydra and it worked well. Even if this stuff isn't perfect, its what we have today at a reasonable price so we should make use of it. Better stuff is always around the corner.

[nrp]

I've been planning out something for this problem as a next project for http://adjacentreality.org

The setup would be to have several wireless tracker devices attached to one's body. One on the chest/torso, one on the head, and one on each hand or peripherial device. Each of these trackers would have a gyroscope, accelerometer, magnetometer, microcontroller, and RF transciever on board. One device, probably the chest one for simplicity, would also have a coil on board generating a magnetic field of a known shape. This magnetic field would be modulated as a sin wave. Each tracker would then be able to determine orientation quickly using the accelerometer and gyro as well as a low pass reading of the magnetometer. Each tracker would also be able to determine approximate distance and orientation relative to the chest tracker by high passing the magnetometer output and determining the amplitude of the sin wave at each axis.

I believe this is similar to how the Razer Hydra works, but this would be wireless. Additionally, the orientations, which probably matter more than the positions where VR is concerned, would be more responsive and resiliant to noise, as they would be calculated primarily from the gyro and accelerometer, rather than the slower and potentially noisier magnetometer.

This allows for stuff like leaning and independent gun/head movement, but doesn't really solve the whole room movement issue. Perhaps something along the lines of this would allow for that though: http://acl.mit.edu/Prigge/

Rather than placing the coil on the person's chest, it would be mounted somewhere in the room. This would of course require a much stronger magnetic field, as the distances would be higher.

I'm probably not going to start working on this seriously until about 6 months from now, after I finish the regular orientation only Trackers.

[brantlew]

It's a great idea nrp! I haven't seen anybody trying a DIY Hydra-implementation. Keep us posted. Also, let us know what your research shows about range. I wonder how far you can actually go logistically, healthily, and legally with magnetic fields.

[MSat]

Thanks for splitting the thread, brantlew. Was worried about it going a bit off topic.


The million dollar question is what's the most sufficient minimum tracking requirements for an enjoyable experience? Angular head tracking is the most obvious, and I'd imagine translation is probably just as important as well. These seem necessary for for an first person games, not just shooters, but driving and flight simulators as well. This should be simple and robust enough with simple optical tracking, and gyro/G sensors, as long as your body remains relatively stationary.

For a more complete FPS experience, though, this can obviously become infinitely complicated. There is no well defined line unfortunately, and it'll probably never be complete until the day when we can stick a plug into the back of our heads and enter the matrix. Since that's no where in sight, what are the baby steps?

For FPSs, we definitely need more body movement. Wires don't help this. The obvious solution here is either integration, or going wireless. Both are doable. If ground-breaking to minimum cost ratio is key, then wireless is the obvious current solution. However, body movement is pointless unless we can track it. So, what movements need to be tracked? Certainly the hands/gun. There are probably hundreds of ways to do this that are currently being done. The exact method is irrelevant to my my discussion at this point. Next up is probably body placement/rotation, which needs to be separate from head rotation. I do want to get into a bit of the specifics here, as commercial gaming systems haven't focused on this as much, except for the MS Kinect and it's clones (which as far as I can tell are currently too slow for such duties, though a faster device is probably possible). Another solution is a foot tracking floor mat, with optical trackers at the periphery pointing up towards shoulder mounted optical markers. This combination not only provides overall body direction, but also twist and leaning, in addition to movements such as crouching and jumping. This method is probably less processing intensive than the spacial calculations needed for kinect-like systems. Another solution might be a body suit with optical markers tracked by a stereoscopic camera setup, but I'm not sure if this would be any more efficient than the kinect, it addition to all the obvious drawbacks.

The ability to freely walk around is the least likely unless of course you're willing to be strung like a puppet to an insanely complicated machine.

[Nick3DvB]

The ability to freely walk around is the least likely unless of course you're willing to be strung like a puppet...

Check out this floor:

http://www.youtube.com/watch?v=nQR49JGySTM

http://www.youtube.com/watch?feature=iv ... g8Bh5iI2WY

Shame the rest of their setup was so lame...

[WiredEarp]

Interesting idea NRP. I dont think the Hydra uses the 'gyroscope, accelerometer, magnetometer' bits however. Pretty sure its just a magnetic tracker only.

[brantlew]

The ability to freely walk around is the least likely unless of course you're willing to be strung like a puppet to an insanely complicated machine.

Funny you should mention that, because I've done some pretty extensive work in this particular area. It's closer than you think

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=14069

[youtube]http://www.youtube.com/watch?v=wpwN23wPeKg[/youtube]

[MSat]

Ok... That is SWEET. Watching that video, I was worried you were going to walk right into the the basketball pole

As far as being strung up like puppet, the way I imagined it is that you could stay relatively stationary, while the host of cables are attached at various points on the body and linked to motorize actuators in a crazy feedback loop controlling all your ligaments based on the virtual world. Perhaps not impossible, but certainly very complex. Even then, the result would probably be akin to walking around in a bulky space suit.

[WiredEarp]

I think I have a similar vision MSat For example, doing it for the feet could be as simple as 2 robotic arm/actuators with shoe attachments that could provide force feedback and a decent range of motion. Attach that to twist sensors etc and you could have walking, running and jumping. Kicking etc is more difficult but would still be doable.
Now, that would be a nasty system to be in if it went crazy...

[MSat]

@WiredEarp

I just seen something similar somewhere. It may have been linked here, or on youtube or something. Definitely more practical than being tied in a web of cables. If you wanted to take it to the next level, you can have a suit of sorts where all major points of articulation on the body are regulated by actuators controlled by the feedback loop. So for instance, if you went to touch a virtual wall, the shoulder/elbow/wrist actuators would prevent you from going beyond the point of collision.

It would be quite an experience indeed if it all went haywire. lol

[brantlew]

I've heard rumor that the military built a mechanical system like this for feet, but that testers were too nervous to try it because one little software glitch would snap your ankle. Could be urban legend though.

[cadcoke5]

I, too, recall reading about and seeing a video of a robotically controlled unit that a person strapped their feet into. It allowed people to walk up virtual steps, and other sorts of irregular terrain. I just spent a while to see if I could find it again, but failed.

My recollection of the test set-up was that it was too slow to respond to be very realistic. But, perhaps that was out of safety concerns, which might have required that spotters be given sufficient time to respond if something went awry.

Joe Dunfee

[WiredEarp]

@ Msat: the Novint guys have an exoskeleton they were supposedly working on. I've seen several other implementations of similar tech around, mainly for exoskeleton walking but also just for force feedback manipulators.

I think it would be possible to build a partially passive system just for the walking. You wouldn't actually need full forces with the right passive setup, although that would of course aid versatility etc. However, a sprung system could return most of the stepping forces, leaving the force feedback component to just provide sensation and adjustment of step height etc. That way, you could avoid using dangerously powerful actuators.

[Chriky]

Yeah, it was in the GiantBomb video I think ( http://www.giantbomb.com/e3-2012-john-c ... w/17-6164/ ).

Ah yeah thanks I think I know what he means; it's not drifting but more like 'warping'. I have noticed that a bit. If you keep the controller in a fixed orientation but move it large distances, it can get a slightly different orientation in different places. I think it is due to distortions of the magnetic field rather than accuracy errors because it still seems very stable and consistent still, just looks a bit wrong.

[AntonieB]

This looks like a nice chip? I guess all we need is a small interfaceboard with this chip on it and we would have a very nice 'headtracker' ?
http://invensense.com/mems/gyro/mpu9150.html

The specs looks great.

[brantlew]

That is nice. The compass is a nice addition and not something I have seen integrated on-board. However these are surface mount chips so a bit harder to work with for the casual hobbyist. We need the SDK board or a third party integrator to really make it usable. We talked about it a bit here

http://www.mtbs3d.com/phpBB/viewtopic.p ... =60#p72954

Of course, these sensors still suffer from the fundamental problem with all inertial sensors. They track acceleration very well so you know when motion is occurring. But you have to integrate the signal once to get velocity and then integrate a second time to get position. The double integration introduces so much error that it is all but unusable for a reliable position tracker.

[Chriky]

I've decided to switch to using coloured lights rather than a passive pattern, it makes much more sense.

I experimented last night with a set of Christmas lights and it definitely makes the dots easier to detect, and you can run it at a higher frame rate and still pick them out. However, I was using a set of micro LED lights and they are just a bit too small/weak to get a good position at 320x240. Another problem was that the different colours were noticeably different brightnesses. Tonight I'm going to try it with small bundles of 3 or 4 of the same colour, wrapped in something to help diffusion. By using more for the weaker colours I should be able to balance it out.

For a permanent solution, these lights look absolutely perfect- http://www.castlegatelights.co.uk/light ... set-p11263 - they are a perfect size, reasonably bright, very diffuse and are available in red, green, blue and white, which is what I need. However, the photos are clearly shopped so I'm going to email the manufacturer to get some technical specs for them.

--

EDIT; should we get this thread moved to the new DIY/Research board?

[brantlew]

Yes, this would be an interesting project thread Chriky.

[FingerFlinger]

@Chriky

I thought on this a bit last year, but in addition to marking the walls and ceiling, I wanted to mark the user's arms and wrists. There are occlusion problems associated with tracking limbs, but it's easier if each plane can be uniquely marked, via patterns or different colored lights.

Patterns introduce lag from image processing, as well as high resolution requirements. And colored lights are limited by the number of colors that are uniquely identifiable.

So anyways, my thought was, using the PS3 Eye at a high frame rate, synchronize multiple "sets" of the markers with the camera, so that a given set is only on for every 3 frame or something. You could multiply your number of unique identifiers by any number, given the effective frame rate remains sufficient.

Ex. 3 sets of Red/Yellow/Blue markers, set 1 fires on frames 1,4,7,10, set 2 fires on 2,5,8,11, set 3 fires on 3,6,9,12, effective frame rate is 60Hz.

Alternatively, using marking patterns, as long as you make sure that the cameras' FOV is sufficiently greater than the user's, you have some buffer area for the computer to positively identify the marker. This is because the cameras can see the object for some amount of time before the user. This helps for situations when the an object enters the user's view from off-screen.

[davidgutierrezpalma]

I have been talking with a friend about this topic and we had a crazy idea for a DIY 6DOF head tracker. We probably missed some important point, but I wanted to know if somebody has already tried a similar approach...

We know that a accelerometer is unable to calculate the "forward" vector when one of the axis (X, Y or Z) is pointing directly downwards, so most commercial 6DOF devices are built using a 3-axis accelerometer and a 3-axis gyroscope and the orientation of the sensor is calculated using the information of both sensors. The main problem of this approach is that gyroscopes have some drift and they can become pretty expensive for a DIY project. We have been talking about those issues and I've realized that if we put two 3-axis accelerometers together in a non-orthogonal angle, they would never have their axis pointing downwards at the same time, so we could use the information of both sensors to calculate the direction of the "forward" vector. In addition, accelerometers are much cheaper than gyroscopes so we could have a very cheap 6DOF tracker if this approach works.


Have anybody tried to build a similar device?
Does anybody know if we have missed some technical detail that could make this idea fail?

[Chriky]

My idea was something like this;


Basically a triangular grid of RGB points, with white points within each triangle used to identify them. I think the best bet it to try and identify them almost globally; that is there are enough unique identifiers to mean you don't need to repeat them near each other.

With three white points you get 8 IDs straight away - use them as binary bits, identifying them by which coloured corner they are nearest. However, if the coloured lights are placed about 20cm apart, you will almost certainly get several triangles within the frame. This means you can further identify points by the ID of their neighbours. If you have a 7, with a 3 on it's R-G edge, then you can distinguish it from another 7 that has, say a 4 on it's R-G edge.

This potentially gives you 8*8*8*8 = 4096 IDs. If you spread them out properly, you can easily make it so that it would be impossible to move between similar IDs in one frame (honestly, you really can barely move at all in 1/187th sec, even if you are violently shaking your head it's about 1 deg of movement).

So the basic global location algo would be something like;
- Get the two coloured points nearest the centre of the image
- Get the nearest point to these two of the third colour
- This is your centre triangle, get it's ID from whites inside
- If possible, get IDs of surrounding triangles
- Once you have all the info you can get, lookup possible positions on the surface map
- Pick the possible position that is nearest your last position

In terms of image processing this is pretty fast. These images demonstrate the stages...
This is my horrible hacky prototype light setup;

The 'points' are bundles of 3 micro LEDs covered with kitchen paper to try and make them diffuse. As you can see, orange and red are quite close colours so not ideal at all. Those berry LEDs should be much better.

Running at 187FPS that comes through on the PS3 Eye like this;


In HSV space that looks like this;


If you threshold based on the V amount, you get a binary image separating all the points in one go;


You can then get each of these blocks of white with a fast algorithm. It is not any slower if there are more points in the shot. Then for each of these spots, you can work out the average colour and the centre of mass based on the intensities (should give sub pixel accuracy);


You should be able to 'snap' the average colour to one of RGBW and get it right.

[cybereality]

@Chriky: Interesting work. Looks promising.

[brantlew]

@davidgutierrezpalma: Remind me again why you can't calculate the forward vector if an axis is point down. I know gravity is a contant 1g acceleration, but if you subtract this why is the downwards angle problematic? Also this may serve as a 3 DOF sensor but the X,Y,Z translations are still problematic because of signal noise. Adding 3 sensors may reduce the problem by averaging the error but it will still be present.

@Chriky: I'm interested to see where this goes. I would suggest moving this to project thread. My one comment is that although you have put a lot of thought into creating unique global markers, you may not need for many applications. Only if you need to track absolute position within a defined region would this be necessary. In many applications, you could probably just indicate a direction vector and that would be adequate.

[nrp]

I have been talking with a friend about this topic and we had a crazy idea for a DIY 6DOF head tracker. We probably missed some important point, but I wanted to know if somebody has already tried a similar approach...

We know that a accelerometer is unable to calculate the "forward" vector when one of the axis (X, Y or Z) is pointing directly downwards, so most commercial 6DOF devices are built using a 3-axis accelerometer and a 3-axis gyroscope and the orientation of the sensor is calculated using the information of both sensors. The main problem of this approach is that gyroscopes have some drift and they can become pretty expensive for a DIY project. We have been talking about those issues and I've realized that if we put two 3-axis accelerometers together in a non-orthogonal angle, they would never have their axis pointing downwards at the same time, so we could use the information of both sensors to calculate the direction of the "forward" vector. In addition, accelerometers are much cheaper than gyroscopes so we could have a very cheap 6DOF tracker if this approach works.


Have anybody tried to build a similar device?
Does anybody know if we have missed some technical detail that could make this idea fail?

Your reference vector is still gravity, regardless of how many accelerometers you have. You will still be unable to know your rotation around the gravity vector. Adding more accelerometers will only allow you to reduce noise or gain precision with oversampling. The reason a magnetometer is used is that you're adding another vector to magnetic north.

[nrp]

Basically a triangular grid of RGB points, with white points within each triangle used to identify them. I think the best bet it to try and identify them almost globally; that is there are enough unique identifiers to mean you don't need to repeat them near each other.

It would be interesting to try projecting the pattern onto the ceiling to avoid having to stick something powered to the ceiling. That is, either visible light using something like a SHOWWX, or an infrared laser through some kind of diffraction grating like the Kinect uses.

[davidgutierrezpalma]

@brantlew: I will try to explain better, but English isn't my main language and explaining this kind of stuff can be difficult...

• Let's suppose the Y-axis is pointing directly downwards, in that situation we have: X = 0g, Y = -1g, Z = 0g
• Y = -1 -----------> That means that the device is on a flat surface without any pitch or roll.
• X = 0 and Z = 0 ---> We don't know the rotation of the device around the Y-axis (in other words: we don't have any information about its orientation in the XZ plane) so we can't calculate the direction of the "forward vector".

@nrp: you are right, I would continue using only the gravity force, but I would avoid the situation described above because the accelerometers wouldn't be placed in an orthogonal angle, so they will never point directly downwards at the same time. However, I don't know if that would be good enough for building a reliable 6DOF tracker...

I haven't talked about including a magnetosphere because that would be a 9DOF tracker (or at least, that is the name that is being used in certain online stores) but I think it should be possible to use a magnetosphere (in addition to the accelerometers) if a greater accuracy is required. My post was about using "2 accelerometers vs 1 accelerometer + 1 giroscope", but we can also talk about "2 accelerometers + 1 magnetosphere vs 1 accelerometer + 1 giroscope + 1 magnetosphere" if you want, it could be an interesting discussion...

[MSat]

The various ideas here are interesting.

In regards to a foot mounted system, a somewhat passive system may actually be feasible. Say one where movements can be regulated, but not added. For instance, it might not be able to force you feet/legs in any position, but it can prevent you from making such motions. This can at least prevent the machine from ripping off your limbs..heh.

The motions might not feel completely natural, so it might feel a bit awkward. I do think that if the limited mechanics are incorporated as part of the game play, then it might make it feel more believable. Lets take the stepper for example - while it might feel unnatural, and mechanical, if the premise of the game is that you're in a body-controlled mech or super powerful robot, you integrate the physical sensation of the system as part of the game, so they end up feeling natural in the virtual world. IMO, any motion control hardware with it's limitations need to be integrated as part of the game play as much as possible for the most effective suspension of disbelief.


I've given my foot/body motion tracker a bit more thought and think it could be quite simple. Again, the premise is that you stand on a round pad maybe lightly larger than a hoola-hoop. The pad can track the placement of your feet either using an array of undetectable switches. I don't thing ultra high resolution is that important - maybe around 100 or so individual tracking points. The perimeter consists of 2 or 3 circles of monochrome light detectors, where the innermost circle points slightly inward, and all successive circles point more outwards. A single piece shoulder mount (think of the shoulder/collar bone protection american football players wear) would contain a light source sticking outward a little bit from each shoulder, which are detected by the floor pad's perimeter sensors. This detects not only the upper body's orientation, but also any forward/backward/side-to-side sway when used in conjunction with the foot position sensors. In fact, it should also be able to detect crouching and jumping. You can also add a third light source hanging off the back of the HMD to keep the head tracking sensors orientated (or completely eliminated except for tilt). A weapon would work in much of the same way.

[nrp]

@nrp: you are right, I would continue using only the gravity force, but I would avoid the situation described above because the accelerometers wouldn't be placed in an orthogonal angle, so they will never point directly downwards at the same time. However, I don't know if that would be good enough for building a reliable 6DOF tracker...

I haven't talked about including a magnetosphere because that would be a 9DOF tracker (or at least, that is the name that is being used in certain online stores) but I think it should be possible to use a magnetosphere (in addition to the accelerometers) if a greater accuracy is required. My post was about using "2 accelerometers vs 1 accelerometer + 1 giroscope", but we can also talk about "2 accelerometers + 1 magnetosphere vs 1 accelerometer + 1 giroscope + 1 magnetosphere" if you want, it could be an interesting discussion...

It doesn't actually matter which way the accelerometers are pointing. The only measurement they are making is of acceleration, and for the purposes of determining orientation, that is the constant acceleration of gravity. Regardless of how many accelerometers you have and which axes they are or aren't aligned to, the values you're reading out will only let you determine which way gravity is, which means you will not have a yaw measurement, a measurement of the rotation around the gravity vector. Adding a gyro means you do have a yaw measurement, but because it is only relative rotation that is being measured, the measurement isn't anchored to anything physical and it can drift.

Additionally, digital gyros generally sample at a much higher rate than accelerometers, and are also not thrown off by non-rotation movement. A MEMS based motion tracking device needs an accelerometer and gyro at minimum (unless you don't care about anchoring roll/pitch), and preferably also a magnetometer (to get rid of yaw drift as well).

[davidgutierrezpalma]

It doesn't actually matter which way the accelerometers are pointing. The only measurement they are making is of acceleration, and for the purposes of determining orientation, that is the constant acceleration of gravity. Regardless of how many accelerometers you have and which axes they are or aren't aligned to, the values you're reading out will only let you determine which way gravity is, which means you will not have a yaw measurement, a measurement of the rotation around the gravity vector. Adding a gyro means you do have a yaw measurement, but because it is only relative rotation that is being measured, the measurement isn't anchored to anything physical and it can drift.

Additionally, digital gyros generally sample at a much higher rate than accelerometers, and are also not thrown off by non-rotation movement. A MEMS based motion tracking device needs an accelerometer and gyro at minimum (unless you don't care about anchoring roll/pitch), and preferably also a magnetometer (to get rid of yaw drift as well).

A single accelerometer can't calculate the yaw vector when it's on a flat surface, but I have attached a fast sketch to show how can you use the information from a second sensor to decide if the "main sensor" Forward Vector is looking towards left or right on a 2D space. I think this example can be extrapolated to a 3D space for calculating the Yaw Vector, but a 3D image could have been more confusing than a simple 2D sketch for explaining my theory.

//-------------------------//

EDIT:
Nevermind, I've found a flaw in my theory, so I will remove my rough sketch.

[brantlew]

I ran across this thesis paper that is a near-exhaustive survey of all the issues and techniques for dealing with inertial sensor packages for dead-reckoning. It is way above my head, but I found it interesting all the same. The author was able to squeeze relatively accurate positional tracking out of consumer level inertial sensors using a barrage of mathematical and engineering techniques. Very impressive! Especially when put in the context of his findings on the error rate of the inertial sensors without correction. The accumulated positional error for an uncorrected sensor over a period of 2 minutes was over 1 kilometer ! In the end he knocked it down to less than 1 meter. Amazing. Anyway, if you're interested in anything related to inertial sensors, put on your thinking caps and take a look.

http://www.cl.cam.ac.uk/research/dtg/ww ... thesis.pdf

[LeeN]

I had an idea a long time ago to use potentiometers to create a 3d tracking device, and was able to create a prototype when I got my 3d printer earlier this year:

https://www.youtube.com/watch?v=xyEZV3l ... ata_player

But after playing with the hydra and looking at the cost, it would be more expensive to build this. And the potentiometers while being absolute were not ratiometric in their precision. It was still fun to do: )

[davidgutierrezpalma]

@brantlew: Dead Reckoning is also used in multiplayer games to create a smoother experience and reduce the required bandwidth. It tries to predict the position of the player between two networks packages based on historical information. The document seems very interesting, I will read it during the weekend.

[cms]

I've played around with a number of different options for tracking motion, and I thought I'd quickly share some of my musings.

My intent is to set up a vr system where:

* you can rotate freely with yaw (you dont have to always be facing a camera or something)

* (relative to a fixed point) acquire body position/offset in 3d

* accurately track relative hand movements (rotation and 3d position)

* accurately track head rotation

I've succeeded with most of this now. Though I haven't put it all together, and I am not yet satisfied with the latency.

The Kinect is nice in that it gives you fairly accurate skeletal pose information across the body (and the libraries make it really easy too). But, it does not do this fast. Of course fast is relative, but with the intent of getting things down to around 20ms or less, the Kinect is nowhere near practical/helpful.

I have used a high speed camera (PS3 eye) and custom OpenCV stuff. I've played with tracking fiducials. I toyed with wiimotes (and with the motion plus) and playstation move controllers. The software to get the moves talking and calibrated was a pain to get going, and unfortunately in its current state adds a bunch of latency. I'm convinced that ultimately you want an AHRS (with 9dom - 3 axis accelerometer, 3 axis gyro, 3 axis magnetometer). There is some work going on to directly reprogram the moves. I imagine that the moves might provide the cheapest avenue for others to explore AHRS.

I worry that overall, bluetooth might end up costing too much in latency.

I think, without necessitating a fixed camera or magnetic tracking, you would probably be able to integrate from a set of AHRSs to find displacement and trigger for things like jumping and crouching without losing much.

---

Using a bunch of ps moves, I set up a fun system. I made a number of straps that would hold a move each. The moves were calibrated. I'd strap one to my forearm, one to my upper arm, one to my chest, and one to my head (next to an hmd). I also held a ps navigation controller.

I did some further "baselining" in code. The user would go into a few different poses and I would calibrate. I constructed a skeleton, very similar to how I imagine Carmack is. It took into consideration the offset of the position of the strap/device from the actual point of rotation. Because I had the "body" sensor I could get an accurate 3d position of the head, upper arm, lower arm, and hand relative to the body basis through the composition of the rotations.

I downloaded the source sdk and started tweaking half life. I didn't get too far with it overall, but I did incorporate the pose tracking for the body and head. At some point, I'd like to return and incorporate the arm/hand tracking to be able to aim and shoot realistically.

It was really fun and immersive to put on an hmd, view things in 3d (thanks to tridef), walk around in game moving my head and body convincingly. I even would physically lay on my side and look up under things. For example, at the beginning of HL2 you arrive on a train. One of the first things I did was lean my body outside of the train door and move my head around. I also went on my side and looked up under the seats of the train. I got out of the train and looked up under the steps getting off the train. I'm sure I looked really stupid in meatspace, but it was awesome.

(I showed it to a few others, and they quickly pointed out problems with it. Why were the bullets shooting out from my face? Why could you lean through walls?)

---

How to make it better:

I don't think I'm doing the calibration quite right or something as I end up with a bit of drift.

I think if I added a fast camera mounted above the user looking directly down, and had some ir leds on top of the head, I could have a satisfying, cheap, convincing, and fast 3d position/offset. This would still allow me to rotate about freely in the ways that I care about. I could use this to get an absolute positioning of the head and use that as the reference basis instead of the body. This is likely best because the head tracking is most important and noticeable. Going the other way led to jittery head movement. This would also let me handle jumping and crouching convincingly.

Instead of each ps move sending out their signals via bluetooth, getting them in software, and processing the rotation calcs, I should reprogram them to do the calculations onboard and only output the final rotation quaternion, wire them together and probably just send the full output through usb.

---

I'm really excited about getting ahold of the rift and putting this all together!

@cms: That sounds like a very cool project. Congrats on getting it past just the initial movement tests and actually customizing to a game environment. There is a thread dedicated to just this sort of thing here.

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15040

You should post some videos of the system in action.

Brantlew pointed me toward this thread (from the Oculus "Rift" : An open-source HMD for Kickstarter thread), so I thought I'd just jump in and say hi. Unfortunately, I was working on this stuff about a year ago and it would be a bit of a pain to get it going again. But, once I get a rift, I'll be revisiting this and I'll keep everyone updated on my experiments (with pictures and code).