6 DOF Head Tracking Ideas

[Nick3DvB]

Network latency should not be a major problem, but sadly I seems that current smartphones or RasPi would not have the horse power for low latency PTAM, but we already have more CPU cores than most games know what to do with, and this really is "the perfect job for CUDA" (cringe! ), a few people have already ported certain elements over to it with good results.

I just posted those links for background information, there are some great resources available and it's good to see a lot of work continues to be done in this area, but you would definitely want to start with the Oxford reference implementation. I'm not sure if it can make good use of a second camera yet but I also think it makes sense to use PTAM in conjunction with other sensors, even if there is currently no mechanism for them to inform the PTAM system itself.

[brantlew]

A lot of the example videos I have seen seem to be doing full scene reconstruction for AR purposes. I wonder if for just gross motion detection you don't need quite as much image fidelity. If so then maybe just for motion detection you could use low resolutions like 320x200 or even lower? Your precision will fall off - ie. maybe you go from detecting 1 millimeter camera motion to 3 or 4 millimeters, but that might be a reasonable trade-off if you could, for example run the process on a smart phone, or reduce the drag on the host computer by a factor of 4. We discussed earlier that latency and precision are not as crucial for translations as they are for rotations.

So (pulling numbers out of my a$$), maybe a reasonable tradeoff could be achieved by running low-res camera translation detection on a smart phone at 15fps and using a fast gyroscopic tracker for rotations. Sure it's not perfect, but it could still offer very good subjective results.

[divide]

I'm not sure this would reduce computation costs a lot, since PTAM use different level of details anyway (640x480 picture is automatically scaled down to 320x240, 160x120, etc).
Plus I think it's good to have absolute rotation from PTAM too, as gyros tends to drift over time.

[Nick3DvB]

If we can effectively integrate the gyro data, and make use of stereoscopic odometry, maybe we can get away with a faster "bare-bones" PTAM implementation, just cherry-picking the elements required for translation and to work as a reference to correct drift.

[brantlew]

I'm not sure this would reduce computation costs a lot, since PTAM use different level of details anyway (640x480 picture is automatically scaled down to 320x240, 160x120, etc).
Plus I think it's good to have absolute rotation from PTAM too, as gyros tends to drift over time.

This is one of those places where understanding the theory would really help. I guess I'm postulating that you could run the whole image processing chain at half the resolution to get half or quarter precision. So your step sizes would now be 160x120, 80x60, etc... At some point this would just break-down once the information level drops below some threshhold, but maybe you could still retrieve useful information at resolutions much lower than 640x480.

For angular precision you would need high resolutions, so I would be willing to go with gyros if it meant I could get decent translations at a reasonable cost. Besides, gyro drift is seldom a problem in practice - especially for 360 free rotation. If it is a problem there are other methods to control yaw drift.

So this is basically the "fancy" version of the optical mouse solution. Low-res image pan detection combined with gyro rotations. Now I just need to get my masters degree in image processing so I could actually modify the PTAM libraries for this purpose

Edit: @Nick: Yeah that's what I'm sort of saying. Just streamline the algorithm to the absolute bare essentials.

[Nick3DvB]

Now I just need to get my masters degree in image processing so I could actually modify the PTAM libraries for this purpose

I wouldn't know where to start, but you've got until November ok!

We really need to collar someone in the field and explain our requirements, there may be a ready made solution out there already. Maybe post something here:

http://cvisioncentral.com/forums/
http://tech.groups.yahoo.com/group/OpenCV/messages

[divide]

You don't seem to realize the amount of R&D iterations this would require until a rock stable and optimized solution is ready...
Stereo camera raise the problem of syncing the cameras (zero latency between the 2). Then you would have to develop a stable AND optimized tracking specifically for stereo cameras. AND it still has to be PTAM if you don't want your 3d tracking to drift over time.

Also, monocular feature projection only takes 3ms over the 20ms process in Klein's PTAM so you would not gain that much computation time with stereo cameras... You may even loose some time by doubling the frame processing.

I'm not saying you should not dream about such solution, but if you want a working solution right now with Oculus Rift you'd probably stick with the already working solutions such as Klein's PTAM and PS3 Eye, trust me there will already be enough work to do for a proper integration, not even talking about gyros/accel integration, properly mounting the hardware and injecting the 6DOF into video games...

[Nick3DvB]

I hear what you're saying divide; there is certainly no point re-inventing the wheel! But a lot of work has already been done by robotics guys on stereo VSLAM (some using lens splitters on a single camera), so I'm just saying it might be worth finding out if there are "any other wheels" that would do the job, PTAM certainly seems like the best option at the moment but I think it is still worth asking the question, if we can find someone to ask...

[brantlew]

Talk is free. We're exploring everything from cobbled together off-the-shelf parts to speculative software methods.

For a quick and dirty off-the-shelf method, it seems like sticking a Hydra on your head is currently the easiest option.

[divide]

For a quick and dirty off-the-shelf method, it seems like sticking a Hydra on your head is currently the easiest option.

Not sure about that, Carmack has been experimenting with the Hydra ( http://www.youtube.com/watch?v=wt-iVFxgFWk , starting 2h05) and he found out the 3d positioning is not linear; he tried building a calibration map for it but it ended up being too complex.

[Nick3DvB]

With Abrash / Valve involved maybe the "Hydra v2" will solve all our problems, but I still think cameras are the long term solution.

So I'm just going to dump these random Stereo VSLAM links right here:

http://www.ros.org/wiki/vslam
http://forums.trossenrobotics.com/showt ... tation-Eye
http://www.google.co.uk/url?sa=t&rct=j& ... 4cQlRFm2Sw
http://www.google.co.uk/url?sa=t&rct=j& ... Va-lXWX3VQ

[FingerFlinger]

I am most interested in PTAM as (initially) just a cheap effective solution for 2D translation. Even that level of functionality would allow free-motion out in a football field somewhere (Assuming that PTAM is robust enough for such a large map). It seems likely that as it becomes more sophisticated, it could be the ultimate solution, but this first goal seems (maybe) achievable for someone like me.

Does anybody have direct experience with the reference implementation? Would it be suitable for immediate application of 2D translation?

[brantlew]

@FingerFlinger: I've been thinking along those exact same lines. The Libviso stuff you posted looked even better for the application you are describing, but it's unclear how well it scales to large uncluttered areas. My biggest concerns are how CPU intensive it might be and more importantly, how well it works in an uncluttered environment. These algorithms works best when there are multiple objects at various distances and high contrast, so I don't know how well it would perform out in a largely empty homogenous area. If you could checkerboard paint the ground that would probably help a lot. Or in an indoor environment, you could checkerboard the terrain and obstacles. I've got a EE friend of mine sort of peeking into it it (Luckily where I work, I've got a whole room full of image processing PhD's to talk to). But at some point we're just going to have to download the thing and give it a whirl.

[FingerFlinger]

I'm hoping that objects just outside of the play field, such as buildings and trees, will provide enough parallax information for the algorithm.

[brantlew]

I'm hoping that objects near the horizon, such as buildings and trees, will provide enough parallax information for the algorithm.

Me too, but precision will drop proportionally with the distance of the objects in frame. If you're staring at your shoes, you will register millimeter level head motions, but if you're looking at the moon there is no parallax information. I just don't know how much motion precision you will have using objects that are 200 or 300 feet away. Of course, even if your precision is just a few centimeters that would be enough to register gross 2D character motion - maybe with some added latency though?? Or maybe just normal ground clutter will have enough texture for the cameras to grab onto, but I'm a bit skeptical. (Notice how in those AR videos they always film it against a patterned carpet or a cluttered background?)

[divide]

For now I would say the technology is safe for a properly registered room (let's say a large room if you really need some space) but I would not go larger than that, too many unknown factors would pop-up and ruin your tracking immersion.

So for now I guess we are stick using another device to account for large moves (xbox controller or something more sophisticated), or auto-move when you go beyond a certain radius.
Anyway I don't know any stadium large enough to play Skyrim...

[brantlew]

I just thought of another potential problem outdoors. Even if the ground texture is enough, what happens when you look up in the sky? All you can see are clouds and suddenly your in-game character loses their lateral head motions! You would need front and back facing cameras to to keep the ground in view.

So the ideal location would be indoors where you could paint the floor and ceiling (and maybe hang objects from the ceiling) to keep a consistent amount of scene complexity at a consistent distance from the camera. (I'm thinking in terms of a VR laser tag center)

[brantlew]

Anyway I don't know any stadium large enough to play Skyrim...

I do

http://www.mtbs3d.com/phpBB/viewtopic.p ... =75#p76401

[divide]

I just thought of another potential problem outdoors. Even if the ground texture is enough, what happens when you look up in the sky? All you can see are clouds and suddenly your in-game character loses their lateral head motions! You would need front and back facing cameras to to keep the ground in view.

So the ideal location would be indoors where you could paint the floor and ceiling (and maybe hang objects from the ceiling) to keep a consistent amount of scene complexity at a consistent distance from the camera. (I'm thinking in terms of a VR laser tag center of something)

Absolutely.
Anyway to me VR is supposed to happen in a controlled environnement, simulating walks inside a large sphere or a rolling ground, so that you really have infinite walks in the same controlled conditions.

[divide]

Anyway I don't know any stadium large enough to play Skyrim...

I do

http://www.mtbs3d.com/phpBB/viewtopic.p ... =75#p76401

My first reaction was.. WTF ?

What tracking system is that, is it able to track your walks on such large distance and accurately reproduce it ingame ?
Also, do you embed all your system in a backpack laptop ?

[brantlew]

Wiimote

[divide]

I still don't get it: you can't track a wiimote beyond a few meters, do you ?
Or you use very powerful IR LED, but you can't get centimeter accuracy ?

[brantlew]

No I can't get centimeter accuracy, but I don't need a high resolution for gross character movement and boundary enforcement.

[divide]

Oh, I get it (just saw the video on the first page).

Very clever idea to gently bias the walking direction to constrain the player within a playing space

Yeah, well.. I live in a small appartment in Paris, I don't have that much space to play with

[FingerFlinger]

I'm thinking of a specific field near my house, which is lined with trees, so maybe those will be close enough? I guess I'll find out when I get around to trying it. One of the various RF solutions many of us are researching should excel in a completely open area, however.

[divide]

Beware with grass and foliage texture, it doesn't seem to work that well with PTAM (see failure cases at the end of this video: http://www.youtube.com/watch?v=Y9HMn6bd-v8 ) because of repeated pattern, foliage complexity and wind in the grass/trees...

[FingerFlinger]

Doh, I didn't even think about wind!

[Nick3DvB]

Robot OS PTAM implementations: http://ros.org/wiki/ptam

The wiki states: "it is robust in outdoor and self-similar environments such as grass" & "runs on-board a computationally limited platform"

There is a very limited win32 port, but we could try running full-fat nix ROS in Qemu / VirtualBox, and it has been ported to the RasPi & Android.

Also, found some good PSeye info here: http://nuigroup.com/forums/viewforum/34/

[Fredz]

The main thing I was hoping for was a bit higher resolution, as anything camera based is fairly low (even a full HD cam tracker would only have 1920 measuring points horizontally).

If you know the size and shape of your markers you should be able to get subpixel precision.

[divide]

Robot OS PTAM implementations: http://ros.org/wiki/ptam

The wiki states: "it is robust in outdoor and self-similar environments such as grass" & "runs on-board a computationally limited platform"

There is a very limited win32 port, but we could try running full-fat nix ROS in Qemu / VirtualBox, and it has been ported to the RasPi & Android.

Hey, great finding !
Plus this one is based on Klein's PTAM, so at least they did start from a solid base
They even implemented an auto-initialization mode, so you don't have to manually define the first 2 keyframes.

You can even see the result of their PTAM here: http://www.youtube.com/watch?v=AdmTGAo1Jvg

[Nick3DvB]

It looks good, they have made some important improvements.

Found another video: https://sites.google.com/site/fuzzysingularity/Blogroll

Also some ROS virtual disk images: http://code.google.com/p/brown-ros-pkg/ ... alMachines

There might be some more ROS images on www.oszoo.org but I think it is down at the moment.

[divide]

I'm also reading this 2009 paper by Klein where he explain how he managed to run PTAM realtime on iPhone, despite the 15x-30x slower CPU.
Combining a few tricks both from this paper and from the ROS team, I bet you could easily reduce the PTAM computation cost from 20ms to 8-10ms, meaning a global latency of 20ms (PS3 Eye+PTAM) without using gyros

Also we can reduce the mapping thread priority without affecting the realtime tracking performance (as Klein did on the iPhone port), meaning the tracking process would only take one full core, leaving the other cores to video game computation.

edit: I estimated the performance gain from the ROS team:
ROS team benchmarked their implementation @20Hz on an ATOM 1.6Ghz (source: ROS website)
ATOM 1.6Ghz scored 303 on PassMark benchmark (source: cpubenchmark.net)
The original PTAM was benchmarked @50Hz on a Core2Duo 2.66Ghz (source: 2007 paper)
Core2 Duo 2.66Ghz scored 1675 on PassMark benchmark (source: cpubenchmark.net)
Thus, the performance gain is 20/50*1675/303=2,2.
So on a Core2 Duo 2.66Ghz their implementation should take 20ms/2.2=9ms
So just with the ROS PTAM implementation, we're already reaching the 20ms latency

Nick3DvB, you talked about a win32 implementation of ROS PTAM, have you seen it somewhere ?

[brantlew]

I chatted with a friend of mine last night who has a masters in EE and specialized in computer vision. In regards to the outside homogenous grass problem here was his view on it.

Something like grass or asphalt doesn't present a fundamental problem if it has adequate texture. (perfectly smooth surfaces are problematic) However, the requirements do go up as you as you reduce the amount of distinct objects in the scene. So for example, if you just have a single high contrast pole in your view, then it is easy to correlate that object from frame to frame. Even at low frame rates where the pole moves a great distance between frames the solution converges rapidly. But with something like grass it takes a lot longer to correlate random looking grass patterns from frame to frame - and the problem gets harder and longer to converge if the motion between frames is large. So for something like grass you either have to spend a lot more CPU time to correlate the frames, or you have to significantly increase the frame rate so that the difference between frames is small. So there may not be one solution that fits all problems equally. You might get decent performance in one setting only to see your performance drop significantly (latency and accuracy) in another. I think an important corollary message here is that with a carefully crafted environment, you can significantly reduce the requirements of your system. Something like a checker-board painted room with distinct objects hanging from the ceiling might allow you to achieve good performance even with meager hardware.

[divide]

About the grass problem, the ROS team partially solved it by using a different kind of visual feature detector (AGAST instead of FAST) so that 2d tracking could more often be repeated (as your friend suggested) and thus 2d tracking errors less likely to occur.
Plus the PS3 Eye can go up to 120 fps if needed...

[FingerFlinger]

Wow, good stuff guys! @divide: It's very encouraging that such a performance boost can be realized by giving mapping the backseat. Also, the PS3 Eye can actually run at 187fps, although it might require working around the DirectShow interface. Chriky seems to have some solid experience with the PS3 Eye; he should know.

[divide]

The performance boost is actually achieved with a faster feature detector (AGAST instead of FAST) and by reducing the map size.
Lowering the mapping thread priority does not boost PTAM performance, but it free some processing power for the video games

187fps, that's crazy ! But this would mean PTAM would have to run under 5ms to keep with the crazy framerate, so it would probably means some sacrifice quality-wise. Unless the CPU is fast enough

[Nick3DvB]

Nick3DvB, you talked about a win32 implementation of ROS PTAM, have you seen it somewhere ?

I meant there is a very limited Win32 port of ROS, I looked at it but sadly I don't think it has all the dependencies needed to run PTAM. But a ROS virtual machine on a fast windows host should not add too much latency, I have used them in the past with IO devices that require very fast responses (smartcard ATRs, USB FTDI devices etc). Because ROS has a distributed node like architecture maybe a windows ROS client could receive data from the main linux server through the virtual network interface? I'm still reading up on it.

Also found this on IMU / PTAM integration: http://students.asl.ethz.ch/upl_pdf/134 ... fbb8d751d1

[bobv5]

If one of you guys end up making something that runs in a virtual machine it would be great if it is also possible to run it on a seperate physical machine. I know in theory they use the same network stuff, just wanted to mention it.

[divide]

I'm a bit skeptical about running this in a VM; I use a virtual Mac inside VMWare everyday, and it's way slower than an actual Mac system. Plus there's the problem of usb-driver interfacing and probably the added latency of it.

[Nick3DvB]

I know it seems crazy if we need low latency but you might be surprised how well it performs, my FTDI device actually runs faster in VM with a linux driver than it does in windows! I would use VMware player, it has better USB2 support than Qemu or VirtualBox. I think it is worth trying, almost all modern CPUs have hardware hypervisor extensions, and the disk image could easily be converted to run on real hardware. RasPi could still be an option if it can be optimised enough, but we might have problems recompiling everything for ARM? It might also be worth compiling the original PTAM for windows to test performance, but the guide does say - "Windows will make you cry"