Anyone with experience from using Kinect as head tracker

[CyberVillain]

I've googled a bit and there are plenty of SDKs but no ready software (Didnt find one atleast)...
I guess inputlag iand CPU usage is a problem, but you could use a seperate PC and ethernet rotation and translation values...

I do not know if input lag is fixable, but all the youtube clips I've seen has inputlag (Even when a PC is used not Xbox)

[brantlew]

I have a Kinect still in a box that I've been meaning to start working with but haven't gotten around to it yet. My gut feeling tells me that the lag would be no worse than an inertial tracker but that precision would be the big problem. I'm doubtful that the current hardware would be able to resolve fine grain angles so your view would tend to jitter - especially with slow smooth head movements. Would love to hear from someone else's experience though. The Kinect 2 is supposed to have a much higher resolution (lip-reading and all that) so there's a good chance it would be able to do an adequate job.

[brantlew]

I'm also curious how an HMD would affect the ability of the Kinect to track head orientation. Undoubtedly they use the eyes as important landmarks for interpreting the head. Covering the eyes and distorting the shape of the head would surely reduce the accuracy (if not completely break) the skeletal reconstruction algorithm. Maybe you could tape pictures of eyes on the front of the HMD to help.

[CyberVillain]

The software should be able to be calibrated so that any form of object can be mapped and tracked... I dont own a kinect, but i would love to try out the open source SDK with it... Theres even a C# wrapper for it (Im a .NET Architect and programmer).. Maybe Ill have to shell out the dollars to get me a kinect just for testing

[brantlew]

Skeletal tracking is not programmed into the Kinect itself but is instead out-sourced to the host CPU, so yes you could pull the raw data feeds and do your own skeletal tracking. That's pretty much what's going on here with the OpenNI API.

http://www.engadget.com/2010/12/15/prim ... et-from-s/

But the amount of R&D to understand and write or modify one of these algorithms is pretty substantial. It is much simpler just to plug into the existing API to get the body coordinates. If you look at the Microsoft Kinect SDK the interface is near trivial. You skip all the messy image processing and go straight to polling high level objects like JointID.KneeRight and JointID.Head. So I think you could save a lot of time by visually tricking the current algorithm (ie. taping eyes to an HMD) versus adjusting the algorithm itself to handle a novel situation.

[Fredz]

If it's only for head tracking maybe it would be simpler to not use skeletal tracking but only the point cloud generated directly by the Kinect and find the model of your HMD inside it, using a ground truth acquired with the Kinect beforehand.

[WiredEarp]

From what I hear, Kinect isn't that good at tracking heads anyway. I think the lag will definitely be worse than an inertial tracker - all that has to do is read different sensors and run some calculations. The Kinect has to determine where the entire body is (or even just the head) track features of interest, then calculate the angle faced by the user. I suspect this is much more calculation intensive.

All the Kinect games i've seen all seem to have a little bit of lag... and for a HEAD tracker, lag is a major no no (causes simulator sickness if your head isn't tracked with sufficient speed).

[brantlew]

Granted the Kinect has orders of magnitude more data and calculations to perform but it has one distinct advantage - its optical which means the signal itself has no lag and less noise. Inertial and magnetic signals have inherent lag - plus artificial lag introduced by noise smoothing. So the question is - is the computation lag of the Kinect greater than the sensor lag of the inertial/magnetic? I don't really know the answer.

[WiredEarp]

Well, lets think about it. A magnetic sensor (hydra) has to process the magnetic field for 3 different directions, which takes time. A gyro solution has to process the data from multiple sensors into orientation vectors, also taking time. Now, if these sensors have inherent lag, then this can pose a latency problem. However, I don't know much about modern trackers, but I suspect they don't have much lag. My Razer Hydra has no noticable lag, and a Polhemus Fasttrak has only 4ms lag, which is pretty much negligible. I have no idea about gyro lag.

However, Kinect, despite being optical, has to firstly take a frame of the entire scene. Then, it has to determine where the head is, and then determine which way it is facing. Only after doing this, does it then have similar raw information to that provided by sensors etc. Therefore, unless the input sensors themselves (not the supporting hardware) has lag (IE, a gyro sensor that only returns its position X ms later) I don't see that it is very likely to be any faster (despite being optical) and in fact I suspect it will be slower, due to it requiring all this analysis of the scene first. Now, if this analysis is done by Kinect hardware, then it could well be as fast, but most Kinect stuff I've seen seems to be laggy so I suspect in the real world this scene analysis is the bottleneck. Now, this bodes well however, as one of the big Kinect complaints was the lagginess, so perhaps Microsoft will work on this aspect for the next version of Kinect...

[brantlew]

You are forgetting the delay introduced by smoothing of the inertial/magnetic sensors. Those raw measurements tend to be noisy so a temporal smoothing filter is used to tame the signal (multiple samples averaged over a given time window). However, its delay is probably also fairly small (on the order of a few milliseconds) so I agree with your assessment - the Kinect has more delay.

Rumor has it that the Kinect 2 will perform a lot more on-board processing so it will be interesting to see how that all shakes out. Dedicated image processing circuitry could really swing this the other way.

[CyberVillain]

The Kinect as none onboard procesing, its all done on the Xbox, thats why Xbox games have high input lag. The Xbox is a slow device. Its a different thing when connected to a PC.. But still I think the input lag will be higher than Freetrack or Sparkfun etc..


But it would be a cool project to try out

[brantlew]

The Sparkfun looks awesome. I've been meaning to cook up a driver for the Sparkfun for bridging its signals to GlovePie. It looks like one of the best inertial/magnetic trackers around (just needs software support), and it would give everybody another tool to play with. Just trying to find the time. That effort will probably converge with my other project at some point so hopefully in Feb or March I can get to it.

PS. Unless I can source this project to someone else... Anybody interested? I can supply existing C# code for bridging commands to GlovePie. If we open-source it we can even push the code over to Carl for future native GlovePie integration.

[Johnny-Mnemonic]

I used Kinect for my experiments, however with current SDK's you cannot use it to recognize head rotations, only head position, and with so-so precision.
Basically for this project I coupled positional (X,Y,Z) coordinates of my head and hands from Kinect (using FAAST http://projects.ict.usc.edu/mxr/faast/), and received rotation (Yaw, Pitch, Roll) from separate tracker mounted on HMD.

[youtube-hd]http://www.youtube.com/watch?v=ezahJ2Kr ... r_embedded[/youtube-hd]

[WiredEarp]

Thats really very cool.

[brantlew]

Very nice demo JM! The Kinect does appear to have low positional accuracy as evidenced by the instability of the augmented objects. Those errors would make for a very bad head tracking experience I think. Maybe the Kinect 2 will improve the situation.

[WiredEarp]

It does appear inaccurate, however, for the right sort of game or application it would be fine still. They should bring out a Skyrim/Kinetiq mod so you can throw fireballs using your hands

It doesn't look like its going to be anywhere near suitably lag free or accurate for headtracking however. Having said that, maybe it could be turned around, mount the Kinetiq on your head and make it track the surroundings... you'd miss the wireless aspect however.

[Johnny-Mnemonic]

Thank you guys, glad you liked the demo!
Partially instability in "virtual objects" is because of orientational head-tracker, how you can see, for tracking hands positions Kinect is very good.