6 DOF Head Tracking Ideas

[brantlew]

I keep going to back to Krenzo's comments about acoustics. 1ms per foot latency seems like a very big penalty for this type of technology when optical/RF/and magnetic all have essentially zero latency. Originally I had thought audio would be good for large outdoor spaces where the other technologies don't scale well or are heavily regulated. But the speed penalty would make it useless in those scenarios. Indoors it sounds like the big problem is the latency incurred waiting for the echoes to slowly die down. So it sounds to me like acoustic technology is a non-starter for low latency motion sensing or am I missing something here?

[Nick3DvB]

A speaker + mic system should be half the latency of a "sonar" type system right? In a typical room ~10ms latency sounds ok to me, I'm hoping the blanking time could be reduced by using multiple speakers / frequencies.

It's never going to work outside but in a small room with existing surround speakers it makes a lot of sense to me.

[divide]

I have been looking into the mic + surround speakers idea, it seems a lot of good work has been done in this area already:

http://ntrs.nasa.gov/archive/nasa/casi. ... 025096.pdf
http://www.wcl.ece.upatras.gr/audiogrou ... E_2011.pdf
http://www.picotech.com/applications/loudspeaker2.html

Sadly I don't have the physics knowledge or programming skills to do anything with it,

but these guys are aware of the Rift so maybe they can develop their idea further?

http://vimeo.com/8983229
http://www.divideconcept.net/papers/3DSENSORS-RL07.pdf
http://blogs.wefrag.com/divide/2012/08/ ... egrations/

Hi, I'm "these guys"
Indeed I've done some research on ultrasound sensors, and it could be a good match for 3d head tracking. I had a short talk about it with John Carmack on twitter 2 months ago. The only drawback is that air has some spring (delay) effect, but I bet you could go down to 30ms.

[Nick3DvB]

Hi, thanks for replying. I am sure it can be done, hopefully without having to suck all the air out of the room, that might be a problem!

I was thinking about volume analysis with several mikes, maybe that could be faster if the sensitivity was ok?

[brantlew]

@divide: Great to have someone with some practical experience. What accuracy and latency do you feel could be achieved in a 3x3 meter space? 5x5 meter? Do you think that using a gyroscopic tracker for rotations and acoustics just for translation would offer any benefits?

[Nick3DvB]

I'd say you'd still want gyros for sure, but having 4 or 5 fixed point sources must be helpful for yaw tracking.

I was hoping this could be done with standard speakers, but cheaper systems don't have tweeters in the satellites so getting into ultrasonic range might be a problem.

[brantlew]

So here is my modest proposal for a 6DOF solution. I want the full untethered 360, but seated-only would be a subset of this solution. First off, I think optical is the simplest approach. And one of this simplest ways to approach it optically is with Wiimotes. They are cheap, easy to interface, and have a lot of the necessary image processing implemented right on-board. When you interface the Wiimote, it just gives you the coordinates of up to 4 point sources within a 1024x768 coordinate system. No image processing necessary. I don't have any measurements and data to back me up, but my personal experience with them and other demos I have seen lead me to believe they have sufficient resolution and responsiveness to achieve good results.However, I do believe that for rotational tracking that a gyroscopic tracking device (ie. the integrated Rift tracker) would be better to ensure accuracy.

[youtube]http://www.youtube.com/watch?v=CmXWChzaMP8[/youtube]



Here is my crude schematic. Basically you have two IR light sources on your head and two Wiimotes mounted at the ceiling aimed at a central play region. In general you can triangulate the position of a visible light source using the two Wiimote cameras to get a fairly accurate (2 or 3 mm) reading. Additionally when both lights are visible you can use their angle to correct for any gyroscopic drift that may have occurred.

WiiRoom.jpg

[youtube]http://www.youtube.com/watch?v=c9eOq-cSdSM[/youtube]



The big problem with optical is occlusion. So your points need to stay visible as much as possible. Putting two points on the top of your head and mounting the cameras at a high angle will give you good coverage for the majority of motions that you will perform. When standing erect, both lights will be visible by both cameras. If you pitch your head then one light should remain visible - either the front or the back. Since we are using a gyroscopic tracker, we don't need a marker shape to determine the player orientation. The gyroscopic reading will help the camera know which light they are looking at - front or back. Omnidirectional IR lights will be very important here. This is a very solvable engineering detail and there are several possible solutions. Clipping the tips of LED's increases their spread. Using a ping-pong ball as a diffuser can help spread also. And of course you can always construct a physical "bulb" from multiple LED's.




Obviously there are some pathological scenarios. If you bend at the waist and tie your shoes you may lose tracking, but that's the tradeoff. The algorithm would need to be robust enough to handle lights blinking in and out. In particular, looking up or down might occlude lights behind the head and the algorithm would need to temporarily estimate the position based on a just a single light or no lights, plus the current gyroscopic readings, plus the last known values. So to really work, this needs a fairly sophisticated sensor fusion algorithm and a decent model of a person's head. There is another subtlety here. It's not enough just to know the position of the lights. What you are trying to model is the position of the eyes relative to the lights. So if you basing your calculations on the rear light, you have to extrapolate your viewpoint to the modeled position of the eyes and render the scene from there.

HeadModel.jpg

The remaining two lights could be used for a weapon. Occlusion by your body will be an even bigger problem for the weapon - but tracking accuracy on the hand is not as critical as the head. I would suggest also mounting a gyroscopic tracker on the weapon and using the same position/orientation sensor fusion algorithm for both entities.

Edit: I just realized that it's probably simpler to mount the two lights above your ears - like antennae sticking up from the side of your headphones, than front and back. But either way - the principle is the same.

[brantlew]

For whatever reason, I can't get this pic to post on my previous message. Here is a picture of the LED "bulb" that I was talking about.

[bobv5]

Some people have said that the ps camera would be better, as it connects directly so no bluetooth latency. The wiimote tracking seems pretty responsive. I wonder if the increased transmision delay is offset by the reduced processing overhead once it gets to the pc?

[brantlew]

I don't know, but that's a distinct possibility. There is certainly a trade-off between transmitting an entire video stream and then processing it on the CPU (Sony) versus a fixed function image processor and transmitting 4 little coordinates. I haven't done the analysis though.

Edit: Also the Wiimotes certainly win-out on price and installation over the Sony. Being wireless is a huge benefit for two widely separated, ceiling mounted cameras.

[Krenzo]

I keep going to back to Krenzo's comments about acoustics. 1ms per foot latency seems like a very big penalty for this type of technology when optical/RF/and magnetic all have essentially zero latency.

That penalty is if you put the transmitter on the target/HMD. If you're transmitting from speakers and the receiver/microphone is on your head, then there shouldn't be a penalty there. When the signal is received by your microphone, that is the actual position of your head at the exact moment the signal is received. When sending a signal from head to speakers, then you have the 1ms latency because the head transmitted the signal in the past and will have moved by the time the signal gets to the speakers (which are now microphones).

However, you incur a penalty to your update rate if you have the microphone on your head. Only one speaker can transmit at a time otherwise the signals will interfere with each other. For every extra speaker that has to transmit, your update rate is halved. Compare this to putting the transmitter on your head. There would only be one transmitter, and all receivers can receive at the same time allowing for the maximum update rate.

[divide]

I wonder if there is a very low latency webcam available on the market ? (like, 100ms)

BTW guys, 10-15ms latency is critical for rotation, but I guess 80-100ms is okay for translation.
A tiny rotation cause much more visual change in the picture than translating 10cm.
Fortunately, gyroscope are perfect for tracking rotation with zero latency; but we can go up as to 80-100ms for position tracking without feeling sick.

Krenzo: you can track several frequencies at the same time (each speaker can send a different frequency). Then with a FFT you can recover each seperate frequency, and do a phase calculation for sub-mm precision (though as I said earlier, air has some elasticity to account with, it's not a rock solid position while you're moving). Given a FFT of size 1024 with overlapping @48Khz for instance, you can check the distance to all speakers 93 times per second (and a somehow 30ms delay).

Now, there's something I did not extensively test about ultrasound, it's the room echo messing with the measured phase, and also how far you can go from the speaker while keeping a good enough signal (enough power, not to much messed up with echo).
Also I did not test what would happen if the mic was partially hidden by some object or not directly pointed to the audio source.
So I would not say it's a magic solution, but it definitely deserve a try, specially when you have a 5.1 kit and mic, no big hardware required. In fact when I did my tests some years ago, my idea was to use ultrasounds for head tracking...

[brantlew]

BTW guys, 10-15ms latency is critical for rotation, but I guess 80-100ms is okay for translation.
A tiny rotation cause much more visual change in the picture than translating 10cm.
Fortunately, gyroscope are perfect for tracking rotation with zero latency; but we can go up as to 80-100ms for position tracking without feeling sick.

That's always been my "gut" feeling as well that translation is a bit more forgiving than rotation, but I only have a small amount of experience to back up this claim. Have you personally experienced that or seen it documented somewhere?



@Everyone: I have been experiencing a lot of strange behavior on this forum related to embedded image links. For the last few week, when I post images sometimes they show up and sometimes they don't. That LED image above. When I refresh this page, sometimes it shows up in the post and sometimes it doesn't. Is anyone else experiencing this beside me?

[divide]

I did a bit of immersive experiments
( http://blogs.wefrag.com/divide/2009/01/ ... -low-cost/
http://blogs.wefrag.com/divide/2010/02/ ... cryengine/ )
and also I've been playing a lot with video stabilisers (both mecanic and digital)
so I know for a fact that rotation is MUCH more a factor of visual sickness than translation.
Even if you're doing some very crazy left-right-left-right translations, latency is always less critical here (unless you are very close to an object, like 10cm close, but most of the time you're not).
You'll feel sick if over a second you cannot feel your moves; but I would say a 80ms-100ms latency is just to small to notice in most cases.

If you wanna test for yourself, try with a video camera, pan around (without moving the cam), move around (without panning), go to a video editor and see for yourself what cause the most visual impact during 60ms.
Another proof of that: picture yourself into infinite space, with all the stars around; if you look in a different direction, everything in your vision will change. Now don't rotate, but use your jetpack to move 100m forward: nothing will have change. Exact same view.
So it depends on the distance you are from objects, I base my above assumptions on with objects ranging from 1m and beyond.

[brantlew]

@divide: Thanks, that's good info and bodes well for all these hair-brained ideas we are trying to come up with because the Carmack 20ms standard is pretty tough to meet. Now 100ms - that sounds very doable. Hell maybe even a Kinect can manage that.

[divide]

I would love Kinect to go that low, it would be perfect for 3D head and body tracking, unfortunately I'm afraid it has a 300ms latency... Which is too big :/

[Nick3DvB]

I think for responsive "peek around" parallax effects you would still want to keep translation latency pretty tight. I really like the surround speaker idea but it's going to take a lot of development work, and if we can not use a standard surround system for ultrasound maybe the wiimotes make more sense?

Brantlew have you ever experimented with the wii motion plus, can you get at the raw gyro output or is it all done internally? We could strip a third wiimote down just for it's gyros, maybe just mount the wii motion plus part on the actual HMD, and have everything go through the bluetooth stack.

btw, I have no issues with images at the moment (using Chrome) but I saw Neil post about some stereo image plugin changes recently, maybe they're still updating for it?

[divide]

What is the latency of the wii ? I think the wii camera has a 200ms+ latency, when you point at something in the wii menu there's an obvious delay...

[Nick3DvB]

I am not sure but I thought it was lower than 200ms, maybe the menu lag is the weak wii CPU or something? In your paper you used 3 speakers for tracking, is there any benefit to using all 5 or even 7 speakers, or would that cause too much interference? What about using the sub-woofer somehow, with a low frequency mic? or is the wavelength too long?

[brantlew]

@Nick3DvB: Actually I've never used the Wiimote gyros because I have others that I play with instead. I just use the Wiimote camera a lot. However the gyro values are available through the interface. You can get to them pretty easily through GlovePIE or wiimotelib.

Using the Wiimote for the orientation and for the cameras has a nice "ring" to it but hopefully, as far as the Rift is concerned, we can just pull the gyro values from the Hillcrest tracker either via Oculus SDK or libfreespace.

That's actually one thing that I want to bring up with Palmer is to provide a way via the SDK to "tap" into the integrated Rift tracker stream from a secondary app. I can think of 2 or 3 good uses for that including this one.

[brantlew]

What is the latency of the wii ? I think the wii camera has a 200ms+ latency, when you point at something in the wii menu there's an obvious delay...

I don't have the hard numbers but it is WAY less than 200ms. It is silky smooth when I have directly accessed the camera IR points. I'll try to get some sort of estimate this week from code I have laying around.

[brantlew]

Some people have said that the ps camera would be better, as it connects directly so no bluetooth latency. The wiimote tracking seems pretty responsive. I wonder if the increased transmision delay is offset by the reduced processing overhead once it gets to the pc?

The PS Move does also have the advantage of the Move controller which can be easily modded to create the head lights - and also any PS gun controllers which can automatically can be fitted with Move lights. So the assembly is potentially a bit easier with the Move.

However, I still hate the fact that you have to run cables up to two separate points on the ceilings. That would be really ugly compared to just two little wiimotes hung on the ceiling. Also, because of the Wiimote Whiteboard project, a lot of companies make easy ceiling mount solutions for Wiimotes.

http://www.amazon.com/Gator-Infrared-Re ... B002P6SITC

The Spot is cool because it has a standard camera mount screw hole, so you can use all kinds of camera mounting solutions to mount a Wiimote with. I set up a Wiimote Whiteboard in my company's conference room using it.
http://shop.irpensonline.com/product.sc ... roductId=3

[Krenzo]

Krenzo: you can track several frequencies at the same time (each speaker can send a different frequency).

How much bandwidth are you using? As bandwidth increases, accuracy increases. Is it worth it to not utilize your entire available bandwidth for your signal?

[Nick3DvB]

I don't have the hard numbers but it is WAY less than 200ms. It is silky smooth when I have directly accessed the camera IR points. I'll try to get some sort of estimate this week from code I have laying around.

Yes it must be much lower than 200ms, thinking about it Jonny Lee's Desktop VR demo would never have worked so well with 200ms latency. Glad the gyro data is accessible, but I suppose it would make more sense to use the hillcrest, I'm sure it's output will be available in the SDK, I really hope John can expose a flexable sensor input interface in Doom3 BFG, without upsetting the lawyers too much.

@Krenzo: http://www.divideconcept.net/papers/3DSENSORS-RL07.pdf

I was hoping we could find a solution that would allow standard speakers to be used for ultrasonic tracking and game audio output simultaneously, is that in any way feasible?

[bobv5]

18.5kHz is definatly audiable, at least for some people.

[EdZ]

Dedicated ultrasonic transducers aren't particularly expensive (sub-£ range), so it's probably a lot less work just to string a few of those up than try and persuade a surround amp to emit (without modifying the signal, and with minimal processing delay) frequency ranges way outside its design range, especially as any decent amp will be trying to actively suppress what it would perceive to be high-frequency aliasing artefacts.

[Nick3DvB]

Maybe you could just wire ultrasonic transducers is parallel with the tweeters, but even cheap surrounds kits quote a frequency response up to 20KHz, which is also where the AC3 encoder spec tops out, but you'd probably want to use analogue outs anyway, to reduce latency, and try kill any unwanted processing in the amp.

John considered an ultrasonic doppler system: http://fr.twitter.com/ID_AA_Carmack/sta ... 4254764032

But as I'm guessing the doppler effect is lot less reliable / accurate with longitudinal sound waves travelling through a compressible gas than it is with light (or RF)

[divide]

18.5kHz is definatly audiable, at least for some people.

Any accurate source for that ? Have you tried yourself ? The 20Hz-20Khz human range is more likely a rounding than actual values; I could not find anyone able to hear the 18.5Khz tone.
Anyway it's best to stay with usual 5.1 Kit and mic ranging up to 20Khz if you want everyone to be able to deploy the 3d tracking system easily...
Answering about the LFE: the problem with low frequency is that it reverb very easily in a room, I'm not sure you could do accurate tracking with it; plus most mic are pretty bad at capturing thoses low freq.

[Nick3DvB]

I know I can't, but my hearing is knackered anyway lol. I know children have a higher range and it decreases with age,

they even designed an ultrasonic alarm to keeps gangs of kids away! http://en.wikipedia.org/wiki/The_Mosquito

A lot of work has been done on practical ultra sonic doppler applications:

http://ecommons.library.cornell.edu/bit ... %20007.pdf
http://www.merl.com/papers/docs/TR2008-014.pdf
https://merl.com/reports/docs/TR2007-106.pdf

[divide]

they even designed an ultrasonic alarm to keeps gangs of kids away! http://en.wikipedia.org/wiki/The_Mosquito

proves my point: the Mosquito, designed against kids, is 17.4Khz

[Nick3DvB]

they even designed an ultrasonic alarm to keeps gangs of kids away! http://en.wikipedia.org/wiki/The_Mosquito

proves my point: the Mosquito, designed against kids, is 17.4Khz

Yes, and LFE probably not a good idea anyway - I also read about sub-sonic waves causing nausea in some people!

Do you know if sonic doppler is any more effective at higher frequencies: http://www.sensorsmag.com/sensors/acous ... rement-825

Many sensors seem to be in the the 40KHz range : http://www.hexamite.com/he240tr.htm

Another idea would be using some kind of ultrasonic "flow" meter: http://en.wikipedia.org/wiki/Anemometer ... nemometers

[druidsbane]

Nokia showed off a low-power bluetooth 4.0-based solution. Latency should be pretty low and the link claims high accuracy, however I don't know the exact resolution. It could be good enough for indoor tracking though.

http://grizzlyanalytics.blogspot.com/20 ... oning.html

[Nick3DvB]

Interesting, but I'm not sure the latency would be low enough for our needs.

@brantlew

Thinking about the wiimote again, I'm wondering if I could get away with just using one after all? If you scale that laptop demo up to room size most people won't actually need that range of movement, if they are tethered to a PC and using a handheld controller to move in the game. Personally I'm mainly interested in getting fast <5mm accurate horizontal head translation whilst standing stationary (or turning on the spot) so that it can be mapped to the game camera for proper parallax simulation. For this we should only need to cover an area of a a few feet, or a bit more to stop you from wondering out of the wiimotes viewing angle, which is about 45 degrees?

[brantlew]

@NickDvB: Well the 2 Wiimotes aren't designed to give access to a large space. I was thinking in terms of the high degree of accuracy you can achieve with stereoscopic triangulation. I'm basically "swinging for the fences" here with high precision in all three dimensions as well as an independent weapon, but there are smaller designs that could accomplish more modest goals.

Sure you could probably do it with one - directly overhead seems like the most optimal placement. But your vertical tracking is not going to be quite as accurate as the other 2 dimensions since it will rely on perspective detransformation. Also the amount of pitch that you can achieve might be slightly reduced (unless you can come up with a really good omnidirectional bulb design and placement). But overall I think it should work. And for the seated position it makes a lot of sense since they don't move vertically anyway.

Actually, now that I think about it - the main innovation that this design has over say, FreeTrack is that is fuses a gyroscopic tracker as the primary method for handling rotations. That makes the whole optical tracking solution a lot more robust because the camera can easily compensate for extreme angles. I think the addition of gyroscopic input to FreeTrack would be an incredibly robust solution for 6DOF seated play.


Edit: I guess you could look at it the other way around with FreeTrack - gyros augmenting optical instead of optical augmenting gyros. Using optical as your primary source of angular information and then switching to gyros at extreme angles. That way you keep all the nice low-latency/low-noise aspects of optical, but without the angular limits. It would be wonderful for cockpit sims.

[Nick3DvB]

Thanks, when I get time I'll have to dig out my wiimote and do some more tests, I won't be doing much vertical movement and can use the Hillcrest gyros for everything else.

These demos both use single wiimotes and seem pretty responsive to small horizontal movements at close range:

http://www.engadget.com/2010/04/15/wins ... absolutel/

http://www.creativeapplications.net/gam ... ows-games/

[brantlew]

I'll try to get some numbers this week, but my recollection of the responsiveness of the Wiimote IR sensor was pretty good.

[bobv5]

I pretty strongly favoured wiimotes a few posts back, but now I have doubts.
As some form of gyro tracker, preferably wireless, is goinig to be needed for weapon tracking, I planned to use a wiimote with motion+ and pierreyes glovepie script. This will allow me to use the rift tracker for head tracking, and the wiimote combined with custom gun/joystick thing instead of a mouse/keyboard. I think this would work great with most fps games, even if they don't really support VR stuff.
But if used in combination with a 6dof wiimote based head position tracker, that would be 3 wiimotes. From what I remember, this would give 300% latency compared to a single wiimote. Affraid I can't give links, I read this ages ago.

If wiimotes are used, would it be possible to cut down the latency by connecting each wiimote to a seperate bluetooth adapter? They cost very little now, avaiable in the local £1 shop.

Another doubt I have is bluetooth. This is suposed to be a family friendly forum so I can't accuratly describe how I feel about bluetooth compatibility. Having to use different bluetooth stacks depending on what device you use is really no good.

I find it slightly amusing that nobody round here (including myself) really thought about latency, then when doom guy shows up, everyone "arrrrgh latency, nooo!!!"

[brantlew]

I feel your pain about Blutooth. Beyond just compatibility, just the act of pairing multiple motes is pretty painful. Alas - that's the hardware we're stuck with on Wiimotes.

When I do some speed testing, I'll try to throw 3 wiimotes in the mix and see if that effects things. My gut tells me that the tiny amount of data in those little IR updates is just not enough to clog things up that much, but you never know.

[cadcoke5]

Somewhere, I don't know which thread, I mentioned the CMUcam. (www.cmucam.org) It is available in the U.S. for around $140.

Here is a very edited list of Features:
VGA resolution (640x480) RGB565/YUV655 color sensor
• Image processing rate of 30 frames per second (new firmware)
• Raw image dumps over serial or to flash card
Onboard Image Processing (QQVGA 160x120)
• Track user defined color blobs in the RGB/YUV color space
• Arbitrary image clipping (windowing)
I/O Interfaces ...• TTL UART (up to 115200 baud – 19200 baud by default)

It seems like the image processing may be able to frame its processing area out of the 640x480 full image. If so, then it gives an effective higher resolution over the larger area we would like.

The one thing I haven't been able to find is any reference to latency. The word does not exist in its Wiki, or web site forum.

Another possible, but perhaps unlikely to be useful, source for a hackable camera with onboard processing would be one of the video conferencing cameras with onboard processing. But, the code would have to be totally re-done, and I doubt that is an easy task... unless a vendor wanted to do it themselves.

Are there any other viable video systems with onboard processing for image tracking?

Joe Dunfee

[brantlew]

Well good news about Wiimotes. I dug up some old code that was using stereo Wiimotes so I could test the sample rate. This is not an entirely accurate reading however since I'm going through a couple layers of API that I have no control over. In other words I don't actually control the poll rate on the blutooth connection. I just receive period updates and process them. So the performance may in fact be better than this.

However...my quick little test shows that the signal frequency on Wiimotes is at least 100 Hz. That number holds true even with two simultaneous Wiimotes. Now I don't have the equipment to measure initial latency but the Wiimotes do seem responsive so I bet it is in line with the frequency - about 10 millisecond latency.