6 DOF Head Tracking Ideas

[Nick3DvB]

@Joe - The RazPi camera module will be out soon, and should be very low cost, you could probably run a full PTAM type system on there!

http://www.raspberrypi.org/archives/1254

@brantlew - That's great news on the latency, sounds about right from what I remember testing the Johnny Lee VR demo a few years back.

[cadcoke5]

Nick3DvB, be sure to mention to the RazPi camera developers the interest we have here. Perhaps they may be willing to do some software work that will help the cause here.

Object and blob tracking will certainly be of interest to robotic developers, as well as to us. I don't know how sensitive robotics developers are to latency, but it certainly has been discussed here alot. A camera direct to GPU solution like the RazPi camera might be ideal.

Joe Dunfee

[divide]

I think you guys are going towards a too complex setup, when ready to use 3d tracking solutioms are available...
If I learnt one thing from experiment, it is to reduce as much as possible the complexity of setup if you want a functionnal, accurate and easy to deploy solution.

Coupling 2 or 3 wiimote looks a bit complex to setup, calibrate and initialize, not even talking of added latencies.
5.1 Ultrasound, I've been playing with it and it could be a cool solution in the future but for now it requires heavy experimenting and coding.

Available right now working out of the box you have the PS Move: one device, one 3d tracking. Two devices, 2 3d tracking.

Integrating 3d tracking in games is a complex enough software problem that you don't want to complexify it even more by adding hardware problems on top of it...

[EdZ]

RasPi + onboard camera + IR led ring = Vicon on-the-cheap. Arbitrary tracking volumes (just add more cameras), arbitrary number of markers, great resolution, and the raspi is a darn sight cheaper than a Vicon! I was part of a group at university who tried to create a similar Vicon substitute with webcams but ran into processing and latency problems (budget of nil), having the RPi do the marker discrimination and just feed a stream of coords over Ethernet would make things vastly easier.

[AntonieB]

RasPi + onboard camera + IR led ring = Vicon on-the-cheap. Arbitrary tracking volumes (just add more cameras), arbitrary number of markers, great resolution, and the raspi is a darn sight cheaper than a Vicon! I was part of a group at university who tried to create a similar Vicon substitute with webcams but ran into processing and latency problems (budget of nil), having the RPi do the marker discrimination and just feed a stream of coords over Ethernet would make things vastly easier.

I think something like this must be worked out further... my guess it would be an awsome project and lots of ppl would use it if it is simple enough and the software is available to handle it.

[PatimPatam]

I think you guys are going towards a too complex setup, when ready to use 3d tracking solutioms are available...
If I learnt one thing from experiment, it is to reduce as much as possible the complexity of setup if you want a functionnal, accurate and easy to deploy solution.

Coupling 2 or 3 wiimote looks a bit complex to setup, calibrate and initialize, not even talking of added latencies.
5.1 Ultrasound, I've been playing with it and it could be a cool solution in the future but for now it requires heavy experimenting and coding.

Available right now working out of the box you have the PS Move: one device, one 3d tracking. Two devices, 2 3d tracking.

Integrating 3d tracking in games is a complex enough software problem that you don't want to complexify it even more by adding hardware problems on top of it...

I completely agree on the need to simplify as much as possible, and I believe having just one external tracking device is key if you want a consumer-friendly and robust home system. Having said that i think the PS Move is far from ideal, mainly because you cannot have proper 360 degree tracking (due to occlusion problems), and secondly because it looks stupid (even more with the wand attached to your head)

Personally I'm working on a system that uses a single PSeye camera and a fairly simple custom design for the HMD casing to integrate the markers, in theory should be more accurate than the move too. I'll show more as soon as i have something working! Brantlew’s idea could be better for larger play areas though, like a basketball court, so I'm looking forward to his progress as well.

[PatimPatam]

@Joe - The RazPi camera module will be out soon, and should be very low cost, you could probably run a full PTAM type system on there!

http://www.raspberrypi.org/archives/1254

Looks promising! Only down side seems to be the maximum 90fps, found this in the forums:

Frame rates: (sensor specs, not necessarily possible on Raspi)
QSVGA 15fps
1080p 30fps
720p 60fps
VGA 90fps

[Zaptruder]

I think you guys are going towards a too complex setup, when ready to use 3d tracking solutioms are available...
If I learnt one thing from experiment, it is to reduce as much as possible the complexity of setup if you want a functionnal, accurate and easy to deploy solution.

Coupling 2 or 3 wiimote looks a bit complex to setup, calibrate and initialize, not even talking of added latencies.
5.1 Ultrasound, I've been playing with it and it could be a cool solution in the future but for now it requires heavy experimenting and coding.

Available right now working out of the box you have the PS Move: one device, one 3d tracking. Two devices, 2 3d tracking.

Integrating 3d tracking in games is a complex enough software problem that you don't want to complexify it even more by adding hardware problems on top of it...

I completely agree on the need to simplify as much as possible, and I believe having just one external tracking device is key if you want a consumer-friendly and robust home system. Having said that i think the PS Move is far from ideal, mainly because you cannot have proper 360 degree tracking (due to occlusion problems), and secondly because it looks stupid (even more with the wand attached to your head)

Personally I'm working on a system that uses a single PSeye camera and a fairly simple custom design for the HMD casing to integrate the markers, in theory should be more accurate than the move too. I'll show more as soon as i have something working! Brantlew’s idea could be better for larger play areas though, like a basketball court, so I'm looking forward to his progress as well.

360 degree tracking is nice, but it has considerable functional drawbacks - including the inability to use a wired desktop PC to provide game visuals.

Ultimately, I think there's a great deal of efficacy to making VR tech as accessible to people as possible - we're still severely lacking on the content side of VR - and we're lacking because up until recently, there hasn't been any good VR hardware to develop for.

[PatimPatam]

360 degree tracking is nice, but it has considerable functional drawbacks - including the inability to use a wired desktop PC to provide game visuals.

Maybe I'm wrong on this one but i think the plan is to have a wireless video link in the consumer version of the Rift. I do hope!

[Zaptruder]

360 degree tracking is nice, but it has considerable functional drawbacks - including the inability to use a wired desktop PC to provide game visuals.

Maybe I'm wrong on this one but i think the plan is to have a wireless video link in the consumer version of the Rift. I do hope!

So long as they can resolve the tricky technical issue of sufficiently reducing wireless display latency for such a large display (at least for the target panel) at the kind of refresh rates that they want (120hz)

I'm crossing my fingers for this too; more options is always better. But if initial impressions are anything to go by - using the controller to turn isn't too bad at all!

[brantlew]

I think you guys are going towards a too complex setup, when ready to use 3d tracking solutioms are available...

I completely agree on the need to simplify as much as possible, and I believe having just one external tracking device is key if you want a consumer-friendly and robust home system

Talk is free so I'm just dreaming of a more-or-less "complete" solution for my own personal usage - knowing full well that the construction and calibration would be too demanding for the average consumer. And there are decreasing levels of complexity that you can aim for with the same type of optical/gyro fusion system. I think the simplest derivative that was previously discussed is just starting with the FreeTrack code and adding an inertial tracker (Wiimote+, Move wand, Rift Hillcrest, etc....) to compensate for extreme angles when the optical solution fails. For seated play, that solution is robust and already 3/4 done with the FreeTrack and IMU tracker code all being open source.

But for my own personal usage, I tend to "swing for the fences", so even this standing solution is really not satisfactory. I won't be happy until I have a 6DOF system with free walking. But that's a whole different thread...

[JMX]

Michael Abrash mentioned that the human 6 DOF tracking system basically relies on inertial and optical tracking, which seems to work ok. Maybe basing a simple sensor fusion system on something using just those would work well.

The Hillcrest tracker sounds like it can do accurate 6 DOF but I've heard it can get unsyncronized. Smartphones have gyros and compasses that can keep syncronized with the environment, the ones I've seen are slow but they work. There are also ways to account for the inertial drift to make it much more accurate, I remember reading that someone got a tracker down from kilometers of drift after a few hours of movement down to meters just by accounting for it in software.
The Kinect can do 360 positional tracking, and the torso tracking is probably accurate enough even though it messes up for limbs a lot when occluded or too close.
If I had access to the values for the Hillcrest and Kinect there should be a way to interpolate the inertia from the Hillcrest to the general head position from the Kinect every x number of updates. I don't know how it would look through an HMD but it's at least worth trying. Theoretically it sounds like it might work. Has someone tried this already?

[PatimPatam]

I think the simplest derivative that was previously discussed is just starting with the FreeTrack code and adding an inertial tracker (Wiimote+, Move wand, Rift Hillcrest, etc....) to compensate for extreme angles when the optical solution fails. For seated play, that solution is robust and already 3/4 done with the FreeTrack and IMU tracker code all being open source.

I hope you were right, but i have my doubts that with an inertial tracker you could compensate position after losing sight of 2 or 3 of the optical markers, which is really not hard do when using TrackIR, even while seated. It does not sound like a very robust solution to me.. but I'll be happy anyway if someone proves me wrong

@JMX
Problem with the Kinect is that it only has a 30fps camera, and terrible lag if you use the skeletal tracking libraries.

[Nick3DvB]

In the near term something like the Hydra (a greatly improved version of it) may provide a consumer friendly solution, but there is little doubt that eventually this is going to be solved optically with two "eye" cameras on the HMD. This just makes the most sense, you need visual pass-through for safety and AR applications anyway, but it's going to take a lot of development work on stereoscopic computer vision algorithms to make it happen. That is way beyond me, and most others here, so unless John wants to get stuck into it any time soon we will have to work out our own solutions, as long as we have some standard interface for our sensor data that should not be a problem.

[brantlew]

i have my doubts that with an inertial tracker you could compensate position after losing sight of 2 or 3 of the optical markers, which is really not hard do when using TrackIR

Sure...the fidelity and responsiveness will take a hit once the inertial tracking is engaged. If no lights are visible then you wouldn't be able to do any positioning - just rotations. Better than the the tracking just completely cutting out, but still a tradeoff. I guess you could go with the LED "halo" idea so that you were sure to always have lights visible. It would be a small change in the FreeTrack code to use the gyro yaw angle to preference one set of lights over the other during a transition. I think that would be simpler than trying to come up with distinct marker representations.


@Nick: I think you're right. A head-mounted/markerless camera system seems like it would offer the simplest solution if implemented well. But it does seem like it requires the greatest amount of processing power and R&D. Unfortunately I am incredibly ignorant about PTAM systems so I can't really gauge what would be involved with implementing this at the hobbyist level.

[Nick3DvB]

@Nick: I think you're right. A head-mounted/markerless camera system seems like it would offer the simplest solution if implemented well. But it does seem like it requires the greatest amount of processing power and R&D. Unfortunately I am incredibly ignorant about PTAM systems so I can't really gauge what would be involved with implementing this at the hobbyist level.

The RasPi camera gives me some hope, the robotics community might come up with something, and there are lots of people working on middle-ware solutions for AR phone apps, the RasPi can also run Android, or you could just use your 3D smartphone mounted on the HMD, if the output latency is low enough we might have a few more options soon. Until the technology matures I'm sure we could live with using a few markers of some sort and a certain amount of pre-calibration. As John said, eventually you are just going to slot your Tegra 5 powered, 8k AMOLED, wireless tablet into a dumb HMD mount, (and maybe stream in CGI quality visuals from a cloud based render farm!), if the Rift takes off those days might be closer than any of us dared dream just a few months ago.


ps - I'm sure John could get some version Doom 3 running on the RasPi, if he optimized the hell out of it.

[youtube]http://www.youtube.com/watch?v=e_mDuJuvZjI[/youtube]

[JMX]

@JMX
Problem with the Kinect is that it only has a 30fps camera, and terrible lag if you use the skeletal tracking libraries.

That's right, the Kinect needs to be combined with something fast like an inertial tracker for small/fast movements. The inertial tracker is bad for positioning ( thanks for the graph brantlew! http://www.mtbs3d.com/phpBB/viewtopic.p ... =15#p77553) but it gets adjusted by the good positioning from the Kinect.

The Kinect is more for full or partial body tracking though, it's not so good for seated head tracking like the TrackIR.

Maybe the Kinect and intertia trackers are just too noisy though, and IR can be occluded and it's hard to calibrate, so maybe multiple high speed optical trackers are the best way.

One thing I noticed people commented on after playing Doom with the Rift is they tried to move to the side to avoid fireballs, so maybe mapping the strafing to the hillcrest inertia will be good enough for rough moves like that.

Paralax is supposed to be pretty important so even just having the small and fast jitter from the inertia tracker mapped to micro movements might add a lot.

[divide]

BTW, do we know for sure what sensors we have within Oculus Rift ? Is there an accelerometer we could use there ?

[Nick3DvB]

One thing I noticed people commented on after playing Doom with the Rift is they tried to move to the side to avoid fireballs, so maybe mapping the strafing to the hillcrest inertia will be good enough for rough moves like that. Paralax is supposed to be pretty important so even just having the small and fast jitter from the inertia tracker mapped to micro movements might add a lot.

That's exactly what I've been thinking. It might go against the grain with some tracking purists, but even some fudged hack with the Rift's Hillcrest inertial sensors would be better than nothing. Even if it's only applied in very limited situations, say when the player is stationary (and gyros are at rest) having some form of lateral camera movement would be a powerful visual addition, even if its not mapped 100% and has to snap back to center itself, as long as it is initiated by the players real movement they should buy it. This would not allow simulation of subtle motions, like shifting your weight from foot to foot whilst standing still, but should be ok for quick dodging and some limited peak around effect. This is not a full simulation anyway, in most cases forward motion is still control by a game pad, think about the way walking is modeled in current FPS (adding a certain about of vertical bob / horizontal sway to the camera) these small things can give a compelling visual result, you can always have the option to turn it off.

[divide]

Wow, I just thought, if the Oculus emded an accelerometer, all you would need is a Kinect camera:
Realtime head pos=last Kinect head pos (with 300ms delay)+the last 300ms dataS from accelerometer double integrated and taking into account the last velocity recorded from the last 2 Kinect head pos (for proper integration of the accelerometer datas within the 300ms timeframe). Kinect data could even be smoothed out for more accurate positionning, you would just need a larger timeframe from the accelerometer. I guess as long as you don't go beyond a second of delay, the accelerometer is still accurate enough for double integration

With this solution no need to attach anything to the HMD (as long as it already embeds an accelerometer), and you would have the added value of full body tracking for gameplay and immersion.

[Nick3DvB]

I also thought about using a relatively high latency absolute reference in conjunction with fast inertial sensors, but maybe we are missing something, there must be a trade-off. If the inertial sensors are really noisy it may take too long to obtain a useful reading, and any abrupt corrections from the reference would be very distracting. If you can do some tests with this it would be interesting to hear the results.

[brantlew]

BTW, do we know for sure what sensors we have within Oculus Rift ? Is there an accelerometer we could use there ?

Yes, it will have at least a 6DOF sensor which in regards to IMUs means that it has 3 axis of accelerometers and 3 axis of gyroscopes.

[brantlew]

If the inertial sensors are really noisy it may take too long to obtain a useful reading

The reason that Carmack wants high frequency sampling is two-fold. First - in regards to actual micro-movements (as opposed to erroneous noise) the closer the frequency is to infinity the more accurate your integral calculation is going to be. Secondly - if you start at a high frequency (ie. 240Hz) then you have a little bit of room to do some smoothing to combat the true noise. So if your target rate is 30Hz then with 240Hz you can get a nice little 8-sample smoothing window to work with.

[divide]

Also Kinect could be used for continuous accelerometer and gyro calibration, so that acceleratiom and gyro datas always match the Kinect data, this way we get nice continuous moves with both data integrated.

[brantlew]

I'm a little nervous about using the accelerometers for positional data at all - even constrained by the Kinect. I keep thinking about the scenario where you are keeping your head very still. The accelerometers are going to keep drifting around within a confined region bounded by the Kinect. These correction will need to be done smoothly to avoid noticeable jitters in the position, so the end effect might look sort of like a little meandering warble in the viewpoint bounded by the resolution of the Kinect - exactly the kind of vision you might have when are drunk and can't keep your balance. Exactly the type of effect that might make you want to throw up! Just speculating...

[Nick3DvB]

The reason that Carmack wants high frequency sampling...

So inertial sensors are not fundamentally unusable for small distance tracking, given a high enough sampling rate this essentially becomes a filtering / pattern recognition problem, maybe the magic algorithm already exits in some NASA research lab, then its just a matter of enough compute power, which sounds like a perfect job for CUDA.

correction will need to be done smoothly to avoid noticeable jitters in the position, so the end effect might look sort of like a little meandering warble in the viewpoint bounded by the resolution of the Kinect - exactly the kind of vision you might have when are drunk and can't keep your balance.

...or exactly the kind of vision you get as you walk, so maybe only activate it when moving and try and intergrate it with the walking effect I talked about earlier, it's not a bug - it's a feature!

[Krenzo]

...which sounds like a perfect job for CUDA.

I cringe when I see people say things like this. Have you done any programming with CUDA or any kind of parallel programming? It's not some magical supercomputer.

[brantlew]

So inertial sensors are not fundamentally unusable for small distance tracking, given a high enough sampling rate this essentially becomes a filtering / pattern recognition problem, maybe the magic algorithm already exits in some NASA research lab, then its just a matter of enough compute power, which sounds like a perfect job for CUDA.

I think you misunderstand. It doesn't require anything computationally powerful - integral computation can be done on simple embedded chips. But if you can get like 1000Hz sampling rate and low noise (temperature control, low magnetic interference, etc..) then your integration will get more accurate and deviate slower from the truth. That's sort of how Navy submarines handle this sort of thing. They have these 100K devices in huge temperature controlled, shake retardant boxes that track the inertia of the submarine and dead-reckon their position. Even with all that they can get off by several kilometers per day. But that's a lot better than your typical MEMS unit that will deviate up to a kilometer per minute if unbounded! So even over short durations you are talking in the centimeter error range for position. Accelerometers just really suck for tracking position. Motion detection - yes. Velocity - sort of. Position - yuck. It's all very frustrating

[divide]

Brantlew: dont forget the noise you see from the raw accelerometer data is acceleration. By double integration it autosmooth a lot, so that position gently drift over time, it doesnt wobble. If it's stable enough over a 500ms timeframe, which I hope so, then it's a win.
Plus stationnary noise could also be filtered out, but i dont think you would ever need/want to do that.

[Nick3DvB]

...which sounds like a perfect job for CUDA.

I cringe when I see people say things like this. Have you done any programming with CUDA or any kind of parallel programming? It's not some magical supercomputer.

Nope, mercifully I have not. But I do help beta test a program called CudaBiss, which is used for brute-forcing CSA encryption keys, if you are lucky this still takes about a fortnight on a modern card- so I’m fully aware GPGPU is not a magic bullet. I also realise signal processing is an entirely different kind of problem, but given that we’d want an answer in ~5ms - my money is still on the GPU.

So inertial sensors are not fundamentally unusable for small distance tracking, given a high enough sampling rate this essentially becomes a filtering / pattern recognition problem, maybe the magic algorithm already exits in some NASA research lab, then its just a matter of enough compute power, which sounds like a perfect job for CUDA.

I think you misunderstand. It doesn't require anything computationally powerful - integral computation can be done on simple embedded chips. But if you can get like 1000Hz sampling rate and low noise (temperature control, low magnetic interference, etc..) then your integration will get more accurate and deviate slower from the truth. That's sort of how Navy submarines handle this sort of thing. They have these 100K devices in huge temperature controlled, shake retardant boxes that track the inertia of the submarine and dead-reckon their position. Even with all that they can get off by several kilometers per day. But that's a lot better than your typical MEMS unit that will deviate up to a kilometer per minute if unbounded! So even over short durations you are talking in the centimeter error range for position. Accelerometers just really suck for tracking position. Motion detection - yes. Velocity - sort of. Position - yuck. It's all very frustrating

Thanks for the explanation; I wasn’t thinking that the integration itself was complex (although beyond me as you can probably tell by now), I was hoping we could somehow attack the noise problem instead, if most of the noise is not truly random maybe you could model the hell out of your sensor (in our typical usage scenario) and rapidly compare your real world reading to this data set to make a very quick best guess about what’s actually going on, which seemed like a SIMD type problem to me, but what do I know - don't answer that. I’m just going to have to accept that accelerometers suck at position tracking and no amount of voodoo math is going to change that – understood.

[Krenzo]

Nope, mercifully I have not. But I do help beta test a program called CudaBiss, which is used for brute-forcing CSA encryption keys, if you are lucky this still takes about a fortnight on a modern card- so I’m fully aware GPGPU is not a magic bullet. I also realise signal processing is an entirely different kind of problem, but given that we’d want an answer in ~5ms - my money is still on the GPU.

If you were processing thousands of separate sensors like what you're doing with brute forcing keys/hashes, then a GPU might have some use. In the time it would take to move sensor data into a computer (ie via USB), across the memory bus, and into the GPU to even begin processing, an FPGA would have finished the job.

[Nick3DvB]

Last time I checked custom FPGAs were really quiet expensive, which is actually the main reason CudaBiss exists. I was thinking about rapidly comparing a single sensor's output against a large data set of known sensor outputs and trying to filter sudo-random noise. It was just an idea, I thought that's what this thread was for?

[Krenzo]

Last time I checked custom FPGAs were really quiet expensive, which is actually the main reason CudaBiss exists. I was thinking about rapidly comparing a single sensor's output against a large data set of known sensor outputs and trying to filter sudo-random noise. It was just an idea, I thought that's what this thread was for?

FPGAs start at around $10. You can get really expensive ones if you need Gigabit speeds. There are a lot of ideas being thrown around, and I'm trying to help add some actual implementation advice. If you're going to be dealing with any sort of sensor data, you need to gain some knowledge of FPGAs. Nothing I said was directed towards your idea. You suggested actually implementing it with CUDA, and that's where I felt justified in pointing out using CUDA would be bad.

[android78]

Brantlew: dont forget the noise you see from the raw accelerometer data is acceleration. By double integration it autosmooth a lot, so that position gently drift over time, it doesnt wobble. If it's stable enough over a 500ms timeframe, which I hope so, then it's a win.
Plus stationnary noise could also be filtered out, but i dont think you would ever need/want to do that.

I disagree.
The problem is that the error in speed is an accumulation of all the errors in the acceleration. So if you have a bias in one direction, over time, it will appear as if the speed is increasing (slowly as it may be). The error in final distance will be an accumulation of all the error in speed, which is an accumulation of all the error in acceleration. So the final distance error (hence position) is affected greatly by a small error in acceleration.
For example, if you were to move from one point to another in X direction and it took 1s, then you might accelerate at 8m/s/s for the first quarter, keep going at the same speed (0 acceleration) for the next half, then accelerate at -8m/s/s for the last quarter.
So the actual speed will be (try drawing it on a graph):
Quarter 1: Average Speed = 1/2*(8*.25) = 1 m/s
Quarter 2 and 3: Average speed = 8*.25 = 2 m/s
Quarter 4: Average Speed = 1/2*(8*.25) = 1 m/s

Given this, we can work out the actual distance:
Quarter 1: Distance = 1 m/s * 0.25s = 0.25m
Quarter 2 and 3: Distance = 2 m/s * 0.5s = 1m
Quarter 4: Distance = 1 m/s * 0.25s = 0.25m
Total = 1.5m

Now, compare this to if you had a 2m/s/s acceleration +ve bias (I know that it's not going to always be a bias, but works as example). Your reading would be 10m/s/s for the first quarter, 2m/s/s for the next half, then -10m/s/s for the last.
So the actual speed will be (try drawing it on a graph):
Quarter 1: Average Speed = 1/2*(10*.25) = 1.25 m/s
Quarter 2 and 3: Average speed = 10*.25 = 2.5 m/s
Quarter 4: Average Speed = 1/2*(10*.25) = 1.25 m/s

Given this, we can work out the actual distance:
Quarter 1: Distance = 1.25 m/s * 0.25s = 0.3125m
Quarter 2 and 3: Distance = 2.5 m/s * 0.5s = 1.25m
Quarter 4: Distance = 1.25 m/s * 0.25s = 0.3125m
Total = 1.875m

So for the one second, the amount that you are out is the same percentage as the error in the acceleration. But even if you have 0 error in acceleration for the next second, the distance will be out another .375m, and the second after that, and the second after that. So do you assume that the average acceleration will be 0, or the average speed will be 0 and filter towards that?

[brantlew]

I’m just going to have to accept that accelerometers suck at position tracking and no amount of voodoo math is going to change that – understood

I'm not really mathematically gifted either, but I have read a lot about it. This is a good time to break out what I consider the end-all comprehensive survey of inertial tracking methods. This paper explains just about everything there is to know about inertial trackers and how to deal with them. I can't understand half of it but it was nonetheless extremely helpful for my understanding of them. What the guy accomplishes by the end of the paper is actually pretty amazing. Going from 2 kilometers of error down to 1 meter is a herculean effort!

http://www.cl.cam.ac.uk/research/dtg/ww ... thesis.pdf

Brantlew: dont forget the noise you see from the raw accelerometer data is acceleration. By double integration it autosmooth a lot, so that position gently drift over time, it doesnt wobble. If it's stable enough over a 500ms timeframe, which I hope so, then it's a win.
Plus stationnary noise could also be filtered out, but i dont think you would ever need/want to do that.

I wasn't thinking the position calculation would wobble. Position error tends to just pick a random direction and shoot off towards it. But the finite resolution of the Kinect creates a sort of "snap-to" grid effect. So as the position starts shooting off, it gets snapped back to the Kinect grid. Then it goes off in another random direction and gets snapped back to grid. I'm postulating that this little random walk within constraining grid would cause a wobble effect.

[android78]

...
I wasn't thinking the position calculation would wobble. Position error tends to just pick a random direction and shoot off towards it. But the finite resolution of the Kinect creates a sort of "snap-to" grid effect. So as the position starts shooting off, it gets snapped back to the Kinect grid. Then it goes off in another random direction and gets snapped back to grid. I'm postulating that this little random walk within constraining grid would cause a wobble effect.

I think the problem with that is the way the latency of the kinect works. It captures the image, then takes 300ms to process it and spit out results. So the results are for where you were 300ms ago. My guess is that by logging the increments taken from the accelerometer since the estimated time that the kinect image was taken, then you would have the current position = kinect position + acceleration data offset.
I would still think you would want to have it 'Trending towards' the position provided by the kinect too, since snapping to ever 300ms would create a horrible jerky effect.

[brantlew]

@android78: Yes a literal "snap-to" effect would cause an ugly sort of vibration. So the best you can do is try to gently drift towards the Kinect coordinate. I guess it depends on how far you are away from the Kinect and the size of the resolution grid. If the grid is < 1 mm width at your Z depth, then maybe you would not notice the corrections, but if the grid is 1 cm. Then it seems like this meandering correction behavior would look wobbly.

[Nick3DvB]

I'm not really mathematically gifted either, but I have read a lot about it. This is a good time to break out what I consider the end-all comprehensive survey of inertial tracking methods. This paper explains just about everything there is to know about inertial trackers and how to deal with them. I can't understand half of it but it was nonetheless extremely helpful for my understanding of them.

Thanks, I’ll have a good read. I actually went to Cambridge – the town not the collage! I live about 20 miles away. I just think it’s a real shame that people with the Hillcrest sensor are going to miss out on accurate parallax simulation, I was naively hoping that something could be done for them that wouldn’t involve any additional hardware widgets. I’m obviously not looking to develop the magic solution myself, never having written a line of code that needed compiling, and knowing just enough about the problem to realise I know nothing at all, but I have had some success over the years by bouncing lots of daft ideas around in the right company and hoping a bulb lights up somewhere, at the risk of inducing a few cringes along the way...

[divide]

android78: in my calculation velocity and pos data get forced by kinect result for everything older than 300ms. So there is no accumulation of error bigger than 300ms. I bet on this timeframe accelerometers are reliable enough to give a stationnary position or a 20cm move.
And if it wasn't stationnary enough, then every velocity slower than a treshold would be zeroed, so that only Kinect with its 300ms delay (okay since it's a slow move) would account for position.

Plus there is no snap effect since it's a continous calculation, in sync with Kinect refresh rate (30hz), as opposed to a refresh every 300ms which would indeed create a snap effect.

[Zaptruder]

android78: in my calculation velocity and pos data get forced by kinect result for everything older than 300ms. So there is no accumulation of error bigger than 300ms. I bet on this timeframe accelerometers are reliable enough to give a stationnary position or a 20cm move.
And if it wasn't stationnary enough, then every velocity slower than a treshold would be zeroed, so that only Kinect with its 300ms delay (okay since it's a slow move) would account for position.

Plus there is no snap effect since it's a continous calculation, in sync with Kinect refresh rate (30hz), as opposed to a refresh every 300ms which would indeed create a snap effect.

From what I've been made to understand... because this is exactly what I was suggesting a day or two earlier, is that the amount of time required to overcome the noise threshold of the accelerometers is quite high. Much higher than the 20ms needed to make it appear seamless... probably high enough for it to not be worthwhile without improving tech.

This is because we don't move our heads at full acceleration instantly - we have inertia to overcome.

Also, a follow up to my question on markerless camera tracking - it's simply too slow at this point in time as well. If the algorithms improve, and or the technology improves to bring the return latency down to around the 20ms-50ms range, it'll likely be adopted as the standard for absolute position and motion tracking.

It might be possible to include that sort of markerless camera tracking into a completely hardware solution - i.e. camera data is processed on the headmount via custom designed chips - but that's not an easy or trivial solution, and realistically would require some sort of market proof of VR/headtracking before such a risk could be taken.

I trust the occulus guys have probably already investigated this vector of approach - if it doesn't show up as a solution to translation tracking (with AR potential to boot), then you'll know that it's either not technically feasible or not market feasible.