6 DOF Head Tracking Ideas

[android78]

...
I wasn't thinking the position calculation would wobble. Position error tends to just pick a random direction and shoot off towards it. But the finite resolution of the Kinect creates a sort of "snap-to" grid effect. So as the position starts shooting off, it gets snapped back to the Kinect grid. Then it goes off in another random direction and gets snapped back to grid. I'm postulating that this little random walk within constraining grid would cause a wobble effect.

I think the problem with that is the way the latency of the kinect works. It captures the image, then takes 300ms to process it and spit out results. So the results are for where you were 300ms ago. My guess is that by logging the increments taken from the accelerometer since the estimated time that the kinect image was taken, then you would have the current position = kinect position + acceleration data offset.
I would still think you would want to have it 'Trending towards' the position provided by the kinect too, since snapping to ever 300ms would create a horrible jerky effect.

[brantlew]

@android78: Yes a literal "snap-to" effect would cause an ugly sort of vibration. So the best you can do is try to gently drift towards the Kinect coordinate. I guess it depends on how far you are away from the Kinect and the size of the resolution grid. If the grid is < 1 mm width at your Z depth, then maybe you would not notice the corrections, but if the grid is 1 cm. Then it seems like this meandering correction behavior would look wobbly.

[Nick3DvB]

I'm not really mathematically gifted either, but I have read a lot about it. This is a good time to break out what I consider the end-all comprehensive survey of inertial tracking methods. This paper explains just about everything there is to know about inertial trackers and how to deal with them. I can't understand half of it but it was nonetheless extremely helpful for my understanding of them.

Thanks, I’ll have a good read. I actually went to Cambridge – the town not the collage! I live about 20 miles away. I just think it’s a real shame that people with the Hillcrest sensor are going to miss out on accurate parallax simulation, I was naively hoping that something could be done for them that wouldn’t involve any additional hardware widgets. I’m obviously not looking to develop the magic solution myself, never having written a line of code that needed compiling, and knowing just enough about the problem to realise I know nothing at all, but I have had some success over the years by bouncing lots of daft ideas around in the right company and hoping a bulb lights up somewhere, at the risk of inducing a few cringes along the way...

[divide]

android78: in my calculation velocity and pos data get forced by kinect result for everything older than 300ms. So there is no accumulation of error bigger than 300ms. I bet on this timeframe accelerometers are reliable enough to give a stationnary position or a 20cm move.
And if it wasn't stationnary enough, then every velocity slower than a treshold would be zeroed, so that only Kinect with its 300ms delay (okay since it's a slow move) would account for position.

Plus there is no snap effect since it's a continous calculation, in sync with Kinect refresh rate (30hz), as opposed to a refresh every 300ms which would indeed create a snap effect.

[Zaptruder]

android78: in my calculation velocity and pos data get forced by kinect result for everything older than 300ms. So there is no accumulation of error bigger than 300ms. I bet on this timeframe accelerometers are reliable enough to give a stationnary position or a 20cm move.
And if it wasn't stationnary enough, then every velocity slower than a treshold would be zeroed, so that only Kinect with its 300ms delay (okay since it's a slow move) would account for position.

Plus there is no snap effect since it's a continous calculation, in sync with Kinect refresh rate (30hz), as opposed to a refresh every 300ms which would indeed create a snap effect.

From what I've been made to understand... because this is exactly what I was suggesting a day or two earlier, is that the amount of time required to overcome the noise threshold of the accelerometers is quite high. Much higher than the 20ms needed to make it appear seamless... probably high enough for it to not be worthwhile without improving tech.

This is because we don't move our heads at full acceleration instantly - we have inertia to overcome.

Also, a follow up to my question on markerless camera tracking - it's simply too slow at this point in time as well. If the algorithms improve, and or the technology improves to bring the return latency down to around the 20ms-50ms range, it'll likely be adopted as the standard for absolute position and motion tracking.

It might be possible to include that sort of markerless camera tracking into a completely hardware solution - i.e. camera data is processed on the headmount via custom designed chips - but that's not an easy or trivial solution, and realistically would require some sort of market proof of VR/headtracking before such a risk could be taken.

I trust the occulus guys have probably already investigated this vector of approach - if it doesn't show up as a solution to translation tracking (with AR potential to boot), then you'll know that it's either not technically feasible or not market feasible.

[divide]

According to ACMeter app on Android, if left stationnary (smartphone lying on the ground) the velocity error integrated from accelerometers over 500ms is 0.04kmh=1cm per second. That's kinda good actually, the drift is still minimal over this short timeframe.

[Zaptruder]

According to ACMeter app on Android, if left stationnary (smartphone lying on the ground) the velocity error integrated from accelerometers over 500ms is 0.04kmh=1cm per second. That's kinda good actually, the drift is still minimal over this short timeframe.

That doesn't sound too bad. Seems like this sensor fusion technique should be at least tested before it's dismissed. Wonder how the accelerometers in the occulus stack up.

[Likay]

Just an idea, not based on what i see here. It should be possible to use ultrasonic sound with very good result and precision (i know... so very 70'ths).
The principle: The headsets sends a very short rf or ir (whatever suitable) signal which activates an ultrasonic transducer device placed a few metres away. The headset also has at least 4 (most probably more to detect rotation properly) microphones which detects when the short "squeek" from the transducer reaches them. Depending on when the signal reaches each microphone the absolute position relative to the transducer is easily calculated. Piezoelectric tweeters are usually only specified to a range to 20KHz while they actually works well way beyond that. They're dirt cheap, easy to get and could most probably be used both as transducer as well as transceiver.
Word of caution: Use frequencies above 45kHz, otherwise your pets could very well give you an ultimate vr-experience...
And as said: It's just a thought and not really my field.

[divide]

Likay: read 4 or 5 pages back

[Likay]

Ok. So i just added some extra info then.

[Nick3DvB]

Word of caution: Use frequencies above 45kHz, otherwise your pets could very well give you an ultimate vr-experience...

That's a very good point - a friend of mine used to have a very old TV with an ultrasonic remote, every time he used it his dog used to start barking!

The IR / RF trigger is a clever idea, sometimes good ideas come from people outside the field - the Rift itself is a great example of that.

[Nick3DvB]

Ok. So i just added some extra info then.

Some sort of ultrasonic tracking seemed like a good idea to me to, I was surprised to find out Nintendo have used it in the past and so does the new Gametrak “Freedom”:

http://en.wikipedia.org/wiki/Gametrak#Gametrak_Freedom

I was looking at the original GameTrak unit and that gave me another crazy idea, for a good old-fashioned mechanical solution to the tracking problem. I’ve been thinking about HMD cable management issues anyway (using slip-rings etc) and thought why not just combine the two?! If you mounted something like the GameTrack on the wall of a small room and ran very lightly sprung tracking cords to a free mounting point directly on top of an HMD cap / helmet, you might get away with using loops to run *very* thin DVI/USB cables back to the wall mount(s). You could still turn 360, you won’t trip over any cables, and you have an almost zero latency translational tracking solution. I suppose you could even use the gyro data to correct for changes in head pitch / roll, a rotation sensor on the HMD mount "pole" could even give you a yaw reading. Whether anyone would be crazy enough to actually build such a contraption is another matter, but desperate times…

[Likay]

Using the method i described with several transceivers on the headsets would probably give enough precision and on top of that a latency small enough to not matter (means gyros most probably not necessary but i could be wrong). In an average room it would take the ultrasonic pulses ~10ms to reach the sensors and as you say: the formula to calculate the exact position is very simple, especially compared to the algorithms of trackir, kinect etc. So the latency would be 10ms+calculation where i'm quite certain that the total latency would be very close to the travelling time of the pulses only.

[Chriky]

According to ACMeter app on Android, if left stationnary (smartphone lying on the ground) the velocity error integrated from accelerometers over 500ms is 0.04kmh=1cm per second. That's kinda good actually, the drift is still minimal over this short timeframe.

That's better than I would have thought but in my experience (not that much tbh) of working with accelerometers the main problem is not noise but easy maxing out of the sensor during fast movement. Shake the thing around for ten seconds hard and see if it's only 10cm off in it's estimate.

[brantlew]

@Likay: I've just been casually observing the acoustic discussion because I know little about it, so I could definitely be wrong here, but it seems to me that there is a concern about echoes. I think you have to wait long enough for the echoes to die down or you are going to have to deal with all these secondary waves. So doesn't that mean you have to wait some multiple of the 10ms travel time in order to let the sound bounce back and forth across the room before you can send you next pulse? So maybe the problem with an acoustic system is not so much the initial latency, but the maximum frequency (time between pulses).

Which brings up an interesting idea to me. If you didn't have to worry about echoes (say in a large open outdoor area) couldn't you just send pulses with a short delay between them and not have to worry about travel time at all. Could the receiver (player) just calc their position from a steady stream of pulses? I'm sure I missed something here. Help.

[cadcoke5]

I recall looking in to some ultrasonic 3D pointing devices many years ago. I seem to recall an accuracy around 6". Part of the reason for that is air doesn't always stay still for you. Especially, in a large area, you are going to have air currents adding or subtracting from your measurement. Also, temperature, air pressure, and humidity affect the speed of sound, and therefore your results. Though, you don't need to directly measure this. You just need to compare distance to a known target to calibrate. This calibration would be good for a while and would not vary from minute to minute like air currents would. One more thought, can temperature vary greatly from ceiling to ground in a large room.

Regarding video processing of an unmarked room. I wonder if a super fast method could be created that didn't really try to find anything, but functioned like an optical mouse. In fact, maybe a mouse sensor could be used. If it could stay focused enough on distant targets, then with more than one mouse, you would have enough information to get relative roll, pitch and yaw. Perhaps this would be better than accelerator based gyro info.

Of course, the camera target tracking equipment could do this as well. Perhaps it would do the optical mouse trick 90% of the time, and then hunt for targets the remaining 10%. (or some other ratio) But the mouse idea would be great it it could be implemented easily, because the hardware is so cheap. I suspect, however, that the act of focusing and positioning the lens would be a challenge.

Joe Dunfee

[Nick3DvB]

On the face of it an (ultra)sonic system should be viable, bats seem to manage ok with it, but then we are using very crude sensors in comparison, and they've been working on it for a few million years longer than us! As you say there are probably just too many environmental factors to work around, and if we want low latency the speed of sound doesn't help either.

I might have got the wrong end of the stick again but I seem to remember John saying that he'd talked to intel about some sort of "optical rate sensor" and didn't get very far, but that it "would make a good PhD project for someone". I immediately thought of an optical mouse to, but if that form of sensor would do it I'm sure we wouldn't be having this discussion. I think it was in the keynote somewhere, I'll try and find it.

The more I think about it a mechanical solution doesn't sound so crazy after all, I was planning on sticking a cable hook on the ceiling anyway so why not replace it with a $20 GameTrack on a projector ceiling mount? I can live without an accurate vertical reading, so could do away with the retracting tracking cord and just replace it with the combined signal cable (lightly tensioned / super-thin) going to a small slip-ring on top of the HMD. For zero latency I'm half tempted to try it...



http://janoc.rd-h.com/archives/129

[android78]

@joe (cadcoke5): I'm really wondering about the idea of using the optical trackers from mice, since they have really fast image sampling rates and already have processors to translate into an accumulated movement. Adding a lens to focus on walls shouldn't be hard at all. The thing that i'm trying to work out is how you could use 4 of these (front, back, left and right) and the relative observed movement from each to determine your movement in the room. if you had a textured roof and always held your head straight up and down, it could work well.
Maybe the best way would be to have textured roof, and combine the tilt of your head from the accelerometers (this is usually pretty accurate) to determine the position? You would need to make sure that you were not ducking down for this though.

[Nick3DvB]

Focusing is a real problem, maybe some sort of plenoptic light-field sensor will solve it one day, but laser range tracking might be an option? I wonder if the same mouse DSPs, or at least the same DDIT algorithums could be used with much larger camera CCDs? I assume these mouse DSP chips need a pin-sharp image to track on, but they are so cheap maybe you could use dozens of them together at different angles / focal lengths?

[cadcoke5]

Here is a link to a guy who hacked an optical mouse to use it to scan a page of text... well at least a few letters.
http://spritesmods.com/?art=mouseeye

I don't know if it really applies here, but it shows that they can be hacked to do things.

Regarding how two mice can give you roll,pitch, and yaw; Imagine you have one mouse looking up from the top of your head. That will give you roll and pitch, but not yaw. A 2nd mouse pointed horizontally can then give you yaw. But, and a big but, is if the sensor can be focused at objects at a distance.

I would think this has been discussed within the robotics community, and either used or abandoned as no workable. They hack old fashioned mice with roller balls, to get their encoders for the robot's wheels. Just aiming a mouse at the floor would be an even easier hack... if the lens part were easy.

I just found a great article about using these mice chips to aim at a few inches for robot odometry. He also cites a number of other resources.
http://home.roadrunner.com/~maccody/rob ... oms-1.html

Another vague memory I have is an article about getting an image directly off of a memory chip by just cutting the top off of a chip, and focusing an image on the chip itself. Now, please understand that this is a vague memory, so perhaps it is not fully accurate. I have not been able to find it again.


Joe Dunfee

[Nick3DvB]

Great links Joe, I'll have to try that mouse mod out at some point. This might actually be good for translation tracking to, I wonder if you could do the surface illumination with a brighter / larger area light source mounted on the ceiling / walls themselves, then you just have to worry about sensors autofocus mechanism, which could be the complex / expensive part.

[cadcoke5]

Ideally, you would want a system that doesn't need focusing. Fixed focus has been on cameras a long time, but they had to accept a limit on how close you could take a picture. I wonder if it might be possible to go for a range of 3' to infinity, and that would be acceptable fixed focus.

However, if the idea of a camera with on-board processing and looking for targets is used, this optical mouse type tracking is probably best done on the same camera. But, if the camera does not have on-board processing, then the mouse sensor may be a good supplement to that method because of the hardware response time being so good.

I know there is one other industry that uses linear optical sensors. The auto-focus in a camera needs to line up the two bits of image in the center of the lens. Years ago, when auto-focus was just being introduces, I recall reading about this. But, nowadays, i don't know if these are separate sensors, or just part of the main imaging sensor that is processed differently.

Joe Dunfee

[brantlew]

Regarding how two mice can give you roll,pitch, and yaw; Imagine you have one mouse looking up from the top of your head. That will give you roll and pitch, but not yaw. A 2nd mouse pointed horizontally can then give you yaw. But, and a big but, is if the sensor can be focused at objects at a distance.

I think it could be tricky deriving rotations from a mouse-like optical solution - hence the need for good resolution and focus. If however you left rotation to traditional gyros and just detected lateral and vertical motion using camera optics, it might simplify the problem. Then all you need to do is detect the direction of image pans which seems a lot easier to do even with low quality video and poor focus, and not nearly as ambitious as a PTAM solution that attempts to reconstruct a whole scene.

[FingerFlinger]

Found a paper on the use of optical mice for robot navigation. http://www.ri.cmu.edu/pub_files/2009/7/DilleFSR09.pdf

[GekkePrutser]

Hi Everyone!

Just another lurker here, been thinking about building my VR setup but the right hardware wasn't available.. Until now of course I ordered my Oculus the first day. I'd been following the forum for a while.

I was also thinking about making something with free room tracking... Because that would be the best way for immersion. What I was thinking about (I'm a HAM radio operator so I'm thinking along those lines), was 2 radio antennas transmitting a radio signal which is perfectly in sync (one transmitter with 2 cables of the same length to the antennas ought to do that).

The wave interference pattern would create a pattern of hotspots/low spots like so:

(Image not mine, I just googled it). This is similar to sound waves cancelling each other out depending on the phase. The radius of the circles will depend on the frequency used (for high-resolution you'd want a high frequency with not much spoiling interference so 2.4Ghz is totally out of the question), but maybe 5Ghz is possible as long as it's free of license (not sure if this is the case, haven't looked it up yet). But the user him/herself may block the signal if it's too high in frequency.

If you walk around then with a signal strength meter (tuned to the right frequency), you should get a varying strength als you walk around. With 4 receive antennas in a known square you could perhaps derive some kind of distance information because you could track the signal moving around the room. Unfortunately as you can see the distance between hotspots is not uniform, not predictable, and they don't occur in straight lines, and it's also dependent on the room: If the radio signal starts bouncing off items or walls nearby the pattern may be messed up so much that it becomes static (the wave pattern will be destroyed). But perhaps this is manageable by using as low power as possible.

But I was thinking that if this information is combined with gyro/accelerometer data, it could be used to augment the data.

Not sure if this would actually work but if I have some time I'll see if I can set something up and see if it's in any way useable. It's just something that came to me when I was thinking about this problem, and I just wanted to put this out there in case any of you were thinking along the same lines.

[brantlew]

@GekkePrutser: Keep working on it. There a few guys on the forum working on RF tracking solutions. RF is great because you avoid the occlusion problem, but it can get complex dealing with the signal timings and bounces.

[GekkePrutser]

@GekkePrutser: Keep working on it. There a few guys on the forum working on RF tracking solutions. RF is great because you avoid the occlusion problem, but it can get complex dealing with the signal timings and bounces.

Thanks, I wasn't aware of them.. I'll have a look around on the forum!

And yes you would avoid the occlusion problem, although depending on the frequency of the signal, the human body will block it a bit, or change the signal pattern a bit (the large bag of salt water which we are is basically a conductor and will couple itself to the antenna). Like when TV reception improves or gets worse when you stand close to the antenna.

I was thinking of putting the 4 receive antennas on top of the head (and hope nobody sees it ) and have the transmit antennas high up so this would not be so much of a problem, but I don't know how much it will be. And I don't think you could bend over very much.

The main thing I was hoping for was a bit higher resolution, as anything camera based is fairly low (even a full HD cam tracker would only have 1920 measuring points horizontally).

[brantlew]

The subject is touched on a few times within this thread. Krenzo seems to have made some great progress with an RF approach.

http://www.mtbs3d.com/phpBB/viewtopic.p ... =45#p75456

[GekkePrutser]

The subject is touched on a few times within this thread. Krenzo seems to have made some great progress with an RF approach.

http://www.mtbs3d.com/phpBB/viewtopic.p ... =45#p75456

Oops, sorry! I make a point of reading through a thread before I reply to it but I kind of skimmed this one because it was so many pages. I missed that. His sounds much better, he's clearly put a lot of time on it. I like the idea of modulated radios, I assume he's measuring the distance from several transmitters. I also like the way he can filter out reflections. It must be pulsed for that I assume.

Like I said, I don't have anything at the moment, just that idea, it's good to see people are working on it and are so far with it. I'm already very busy for the next month but after that I might have the time to start some experiments.

[bobv5]

http://www.youtube.com/watch?v=-KxjVlaLBmk
http://hackaday.com/2009/07/30/amazing-robot-dexterity/

Thought you guys might like these, shows how fast tracking can be.

Brantlew, I could have told you wiimote was 100Hz, in Freetrack anyway. I think that what you called inital latency is what I was worried about. I would say that 100Hz is the update rate, and the latency is still unknown. Perhaps I am using the wrong words, but what I mean is if the update rate is 100Hz, and the wiimote takes 10 cycles to process the signal before it gets to the PC, then that would mean the latency was actually 100ms, not 10ms.

For sonic tracking, it might be posbile to use 2 different frequencies, for example 50kHz and 60kHz. The different frequencies could be alternated, allowing the 50kHz echos to drop off while the next cycle uses 60kHz. By the time the next 50khz cycle was due, the echoes from the previous one shoud be low enough to ignore even if not completely dead.

I think the lateral postioning of the Hillcrest tracker has some value as a simple on/off type tracking, for example, ducking or standing, leaning or not leaning, just mapped to key presses.

Another idea is to use multiple acelerometer chips. If the errors produced are random, using 2 or more chips and averaging the results should give better accuracy. I have no idea how to program something like that, but I bet even a *pie script and a pair of wiimotes would be enough to check the general idea. If the sensors always drift the same direction, it might be possible to use two mounted the opposite way, and cancel the errors that way. This still won't be perfect, but it might be enough to make the Kinect+Inertial idea work.

[brantlew]

@bobv: Wow! Those videos are shockingly good. That robot throwing and catching the cell phone is amazing! So yes, I believe that proves the point that optical can be an extraordinarily fast and accurate motion sensor.

[divide]

I read that the PS3 Eye webcam, when used on PC, had extremely low latency (around 16ms).
If this is true, you could get something very nice out of a pattern based 3d tracker like ARToolkit (I've been using it a few years ago).
However I guess this is the same as using a Track IR kit, which has even smaller latency (10ms).
The only problem with theses solutions is that you are limited to a front view (no 360).

My other idea today was mounting a smartphone in front of the HMD, and do some PTAM tracking coupled with the sensors from the smartphone. You would have full freedom, but it gets a bit complicated (porting PTAM on smartphone, coupling with sensors and sending back by wifi...).

[divide]

...or you could mount a PS3 Eye on the HMD and do PTAM on the computer: fast, accurate, 6 DOF with full freedom of movement.
Plus the wild field of view (75 degree) is perfect for stable tracking, and latency can go under 10ms when used at 320x240@120fps. You could even use the 3d tracked points to issue a warning if you're getting too close to a real object while you're in the virtual world
The only downside would be the added weight (? cannot find any record about that). You could still dismount the camera to only keep the optic, sensor and electronic to drop some weight. The camera is pretty cheap anyway ($22).

[brantlew]

After you have run through all the candidate technologies and assess their strengths and weaknesses, it seems like everybody tends to eventually arrive at the same general conclusion. Head mounted camera technology like PTAM is probably going to be the best overall solution to the 6DOF problem. Unfortunately its probably also the most challenging technology for the hobbyist. You practically need a masters degree in EE and signal processing to start down that path, which is why everybody is looking for shortcuts that just require basic trigonometry. Maybe there are some decent image processing toolkits available that can help the layman get some basic things working relatively easy. But it would be difficult to extend the functionality without digging pretty deeply into the theory and code.

[divide]

I'm talking about PTAM because I know there's already a rock stable solution available with a free source code
http://ewokrampage.wordpress.com
So it's just a matter of software integration, no PTAM R&D required. Check out the videos:
http://www.robots.ox.ac.uk/~gk/PTAM/

A few years ago I've been implementing ARToolkit in a tracking software for film postproduction so I guess it's pretty similar here.

[brantlew]

@divide: It certainly does look promising - especially for the unidirectional/seated case. If calibrated against a cluttered background the results look really good. I would love to see someone try to implement this with PSMove camera or similar and feed the results into something like the Valve engine.

It's not clear to me that it works for 360 turning or the free-walking scenario, but I need to allocate some time to download it and figure out exactly what its limitations are.

I'm definitely interested.

[FingerFlinger]

Libviso2 is another option to look at. They claim high-performance stereo visual odometry, but I haven't done much except download the library yet.

[Nick3DvB]

I've been looking into SLAM / PTAM for a while, a lot of work has already been done around the OpenCV lib: http://www.cvpapers.com/rr.html

Camera "pose estimation" seems pretty fundamental for our needs:

http://cvlab.epfl.ch/software/EPnP/ (MatLab)
http://www.morethantechnical.com/2010/0 ... cv-w-code/
http://www.visual-experiments.com/tag/gpusurf/

It's not clear to me that it works for 360 turning or the free-walking scenario, but I need to allocate some time to download it and figure out exactly what it's limitations are.

Maybe of interest: http://www.visual-experiments.com/2010/ ... g-reality/

I was hoping there might be an "off the peg" Android PTAM package by now, to run on the RasPi / smartphone, but all I can find is some Korean research project.

PTAM with a cheap PSeye camera (or two) is starting to seem like a very good option, it looks like we are half way there already?

[brantlew]

PTAM with a cheap PSeye camera (or two) is starting to seem like a very good option, it looks like we are half way there already?

I bet Oculus is at least considering this technology for the consumer version. It fits well with the single self-contained device strategy, plus you get pass-through navigation cameras for free.

[cybereality]

I agree that markerless-tracking is ultimately the way to go. Seems like there are a few solutions out that that look promising.