Auto-calibration of head tracker drift

[brantlew]

I sort of have this fuzzy idea in my head that you can accomplish automatic drift correction on a gyroscopic head tracker by using a statistical heuristic based on human head turning behavior. I have been googling around a bit but can't find anything related to this. It seems also that I remember Carmack mentioning something like this, but I haven't sat again through the 5 or 6 hours of recent video coverage to find it. Maybe I imagined it.

Anyway, the idea goes something like this. Assuming that 1) people have an angular limit to their head turns 2) people generally look left and right in equal proportions and at equivalent angles, and 3) people look in the forward direction more than to the sides - it seems like you could construct a heuristic algorithm that tracked all the right and left angular head rotations and computed a "forward" direction as the weighted average of these angles. (For now assume the person is seated so the torso direction is constant) So to correct for drift you would just have to nudge the sensor back to the computed forward direction.

For example. Lets say that at the start your sensor is calibrated so that forward is 0 degrees. As you turn your head left and right during gameplay it traces an arc of -80 to +80 degrees and averages around 0. Now if you start to notice that the arc starts shifting to -85 to +75 and averages at -5, then you would assume that is associated with yaw drift and you could automatically recalibrate your "true" angle with a +5 offset.

Does anyone have any experience doing something like this? I would love to read a paper on it. Are their some details that I am missing that makes this difficult or impossible? Also, I wonder what the ideal sampling and averaging behavior should be. Do you average over just a single minute or a longer period? Do you weight the average based on time spent looking in one direction - assuming that people look forward most of the time? Do you weight angles more when they are moving versus just standing - assuming that people mostly look forward when they are moving? Thoughts?

[FingerFlinger]

Seems like an interesting idea; the only thing that jumps to mind is that I think you would want to schedule any adjustment 'nudges' when the user is already moving their head. If you try it while the user's head is not moving, you will induce them to move their head and thereby introduce drift. Also, it might make them throw up or get dizzy.

[cybereality]

Interesting idea but I guess it assumes the user will tend to face forward. I don't know this is always the case. For example, when wearing an HMD I tend to look down a lot (facing the keyboard) and not realize it. I guess being tethered to the mouse/keyboard will keep you mostly in one direction yaw-wise (which I guess is the one you want to do this on since pitch and roll have very little drift). So I think its a decent idea, but I haven't enough experience to say whether it will work or not.

[Fredz]

I've skimmed over some papers that were able to somewhat mitigate the drifting with accelerometers but only with periodical or almost periodical movement (walking mainly).

I don't know if there have been research on the subject, but you could try to find yourself how the drifting is behaving by comparing it to a ground truth value acquired by an optical system.

You could record accelerometer and gyroscope values corresponding to the translation and orientation changes measured by the optical system when playing a game, and then try to find a correlation for several types of movements, to then be able to correct them in real-time when no ground truth is available.

Sounds a bit difficult but it could give interesting results.

For now unfortunately I think there is no other *good* solution than using gyroscopes for rotation and an optical system for translation. Head-mounted for your redirected walking or for big rooms, and on the desktop for home use (like TrackIR does, but something better needs to be done).

[Nick3DvB]

I think you might be onto something brantlew! After all, we already know a lot about how a human head should behave, mechanically, so we are not going at this blind. Even very limited sensor output applied to a detailed model could give you surprisingly accurate results. On a much broader level, we also have knowledge of context, so you could even account for the expected "human reaction" as you know something about what is about to happen, and everything about what is happening / has just happened in the game. In addition to your generalized assumptions you could also have some level of additional training on a per game / player basis and apply some form of calibration profile to the model.

You could record accelerometer and gyroscope values corresponding to the translation and orientation changes measured by the optical system when playing a game, and then try to find a correlation for several types of movements, to then be able to correct them in real-time when no ground truth is available.

I didn't articulate it very well but this is exactly what I meant the other day when I said "model the hell out of the sensor" and then make an informed guess. A lot of what looks like sensor "noise" isn't actually random at all, so I'm sure a lot can be done in the area of pattern-recognition and filtering. When constrained by the physical limits of the playing space, as well as in-game mechanics, human head movement certainly seems a lot less chaotic than many other predictive modeling problems, seismology / meteorology etc. Maybe some work has been done in a more analogous field that could be applied here? If I come across anything that seems useful I'll post it.

[brantlew]

I've been doing some preliminary work on this problem. I captured a highly synthetic sample of me just sitting at my desk looking around for a few minutes just to get some data to screw around with. Here are the results.

YawDrift.jpg

You can clearly see the drift occurring in the yaw angle. I have 3 different candidate methods for tracking the drift: a simple moving average, the moving median, and the moving range. All three use a 30 second window size which is ok for this synthetic sample but needs to be adjusted using real-world play data. Anyway, I still think it's a promising approach but I will need many minutes of real-world data before I can say for sure.

Oh and just fyi - for many scenarios yaw drift doesn't matter. If you are using a controller to turn or are just physically standing and turning 360 then yaw drift doesn't matter at all. But if you're in a cockpit simulator or are physically walking in a real space then you generally need to manage the yaw drift.

PS. Matlab has got to be the greatest piece of software ever written.

[cybereality]

Interesting. Wish you the best of luck.

[bobv5]

With the gyro in the Trimersion, it seemed to be more sensitive in one direction than the other. I think I read that the rotation of the planet is a factor with gyros, but I might have imagined it. What I did to compensate to some extent was a glovepie script that multiplied upwards movement by 1.1, and downwards movement by 0.9 (numbers approximate). This seemed that it would work quite well if I could get the numbers just right. It certainly took the gyro from being unusable to only being annoying.

[brantlew]

Well, pretty depressing news on this project...

I captured over 5 minutes of in-game head-tracker data from the Hillcrest 6DOF tracker against the "truth" data of a belt mounted digital compass. I also programmed a button to recenter my view every so often or when I noticed the view had drifted. I knew during the game that things were looking pretty bad when I had to recalibrate about once every minute. To be fair - this is during a free-walking game session so the tracker undergoes more violent motions than during a seated game and also has to endure real 360 turns. So today I've been doing a bit of analysis on the data and the results look pretty bad.

GameDrift.jpg

The graph shows seconds along the x-axis and cumulative angular movement on the y-axis. (ie. 720 degrees = 2 revolutions, etc...) The blue line is the digital compass. The green line is the Hillcrest on my head. I'm looking around and side-to-side so the green varies a lot from the blue. The red line is the head offset from the torso. You can see the steady accumulated drift over time of the red line reaching a maximum of over 200 degrees drift from the original position. I had hoped that this drift would accumulate slowly and that I could use some signal processing to automatically correct it. (A corrected drift would center the red line right around zero). Over time the drift seems to accumulate slowly, but if you look at the data closely it can move quickly over short periods of time.

The magenta dotted lines indicate the times when I clicked the button to recenter. Those are known events when my head and torso are aligned and the offset should be at zero. If you go ahead and correct the red line by using these recentering events then it produces the cyan line. And you can also tell at the time of the calibration - what was the magnitude of the drift?

DriftRecenter.jpg

Here is a closeup of one of those events. From one calibration to the next was only about 20 seconds. The ups and downs could be drift or they could be just normal head turns, but at the magenta lines the head is guaranteed to be centered. So starting at the first recenter, you can see the sensor drift over 40 degrees by the next recenter !! That is really, really fast. And in my opinion, way too fast to be able to use a statistical method effectively.

So I think these 6DOF trackers are just too noisy to auto-calibrate this way. I still postulate that a magnetometer included 9DOF sensor could be corrected this way however. At some point I might try to see if I can statistically correct a Sparkfun.

I hope Oculus decides to go with a 9DOF IMU sensor on the Rift. If not on the dev-kit, then at least on the production unit. It would be really great to minimize this type of problem for applications where absolute direction is important.

[cybereality]

Hmm.. not looking good. The only thing I am afraid of is if Doom3 is hard-coded to the Hillcrest tracker, I don't know how much time they will have to be switching around device support. Usually a game will go gold and be frozen a little while before it actually ships. Though I guess they could release a patch later on if necessary.

[brantlew]

There is another possibility here. I am using the Hillcrest "MotionEngine" sensor fusion firmware that comes preloaded on the device. We know that Carmack dismissed it as unacceptable, and we have mounting evidence on this forum that it is indeed crap - just reference software for a reference kit.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 282#p77905

So maybe with custom sensor fusion software these drift problems could be reduced? Palmer, you've had a good amount of experience with this sensor now. Does it drift this badly with custom sensor fusion software?

[CyberVillain]

Hmm.. not looking good. The only thing I am afraid of is if Doom3 is hard-coded to the Hillcrest tracker, I don't know how much time they will have to be switching around device support. Usually a game will go gold and be frozen a little while before it actually ships. Though I guess they could release a patch later on if necessary.

We could supply a spoofed freepsace dll (If thats what doom3 uses) and output to it from FreePIE using shared memory as pipeline, thats how our Spoofed TrackIR dll works. The difference is that TrackIR saves the path to the dll in the registry so you do not need to copy dll to each game, just change the path from real trackir dll to spoofed dll in registry.

But there is a big chance John does not use the freespace dll, time will tell

edit: I would love to get my hands on his fusion code

[brantlew]

My guess is that Doom3 will use a prototype version of the Oculus SDK designed for a 9DOF tracker. Who knows? Oculus may have purchased the fusion code from Carmack even.

[android78]

Regarding the drift, I wonder if the reason why the drift jumps sometimes may be because you have your head tilted when you turn in one direction or another. If you are just using the one gyro for tracking the yaw, then it will drift when you turn one way with your head vertical, tilt your head to the side, then turn your head back while still tilted. To get more accurate tracking, you would have to combine the accelerometer data (to determine the tilt), with the three gyro data.
I don't know how the tracking software works, but this could be an area to examine.
I think for better tracking, you will have to combine gyro and magnetometer. If you take the magnetometer and trend the calculated position from the gyro towards this, it should keep the drift from absolute reasonable.
To do this, you have something like this:
CalculatedDirection = (PreviousDirection+GyroAccumulation)*(1-TrendScale)+MagnetometerMeasurement*TrendScale
Where TrendScale between 0 and 1. Usually you would use something like .05.
So if your previous direction was at 10 deg. and the gyro was reading a +5 deg. rotation, but the Magnetometer read absolute direction of 20 degrees, then you end up with
Calculated = (10+5)*.95+20*.05
= 15.25
If the gyro was to not measure any more and the magnetometer kept the same value, then you would see it correcting itself:
calculated2 = (15.25+0)*.95+20*.05
= 15.4875

After 50 cycles, the calculated is 19.59502645, which is getting very close to the magnetometer value.

Since you have collected the data from magnetometer and head gyro, maybe you can try playing with this in a spreadsheet and see how the drift goes over time then?

[MSat]

I've been thinking about this a bit lately, and I just can't see a way to completely eliminate drift with just gyros and accelerometers. I do, however, think it can be lower than what you observed, brantlew, but I have to wonder how much the effort would be worthwhile. Even if you could make the drift 4X less, it still would probably be too much. As previously stated, yaw drift might not really be an issue if it doesn't just control the head, but also the body. Obviously, this is not a option for every game. I just don't think there's a one size fits all solution that is suitable for all the different types of gameplay, and play environments.

[brantlew]

@android78: Thanks for the input. I'll have to think more closely about your algorithm when I look into this problem again. Right now, it's been on the backburner since the Sparkfun has proven to be such a stellar tracker for my purposes. I have a worst-case scenario for inertial trackers since I experiment with real walking which creates a tremendous amount of jolting/bouncing behavior. So 6DOF trackers like the Hillcrest just completely fail, but several existing 9DOF tracker algorithms seem to keep up just fine.

@MSat: My play behavior is pathological for trackers and is why I see so much drift. When I'm just sitting stationary and moving my head with a 6DOF the drift is far, far less.