PS3 Eye - PTAM/Visual Odometry

Post by **FingerFlinger** » Thu Aug 16, 2012 8:58 am

This is a placeholder for discussion of PTAM and visual 6DOF techniques. It looks like a few people will be starting or are already working on projects, so I thought an area for focused discussion and sharing would be useful.

Post by **divide** » Thu Aug 16, 2012 10:23 am

A few references to start with:

PTAM is a robust realtime markerless 3d tracking library, source code can be downloaded from here:
http://www.robots.ox.ac.uk/~gk/PTAM/ (last revised 2010)

You can find instruction for the Windows deployment of PTAM/PTAMM here:
http://www.robots.ox.ac.uk/~bob/softwar ... manual.pdf
You may also want to read this post explaining more in detail how to deploy on Windows:
http://ewokrampage.wordpress.com/troubl ... mment-1479

ROS team has been improving on a previous version of the source code (2008) by replacing the FAST feature tracker with AGAST feature tracker (more reliable and faster). They also limited the map size to speed up the computation, and added an auto-initialization mode. However it does not have a win32 project.
Source code can be downloaded from here:
http://ros.org/wiki/ethzasl_ptam

Playstation Eye (PS3 Eye) is a very fast webcam (60fps, 120fps and more) with very low latency (1 frame), available for $20:
http://www.amazon.com/PlayStation-Eye-3 ... tation+eye
http://www.amazon.com/Sony-PlayStation- ... tation+eye
Code Laboratory has developped a fast Windows driver and SDK for it, available here:
http://codelaboratories.com/downloads/

Post by **divide** » Thu Aug 16, 2012 2:57 pm

I was wondering how the 3d tracking could be shared once it's working, looks like the VRPN protocol is the perfect match for that:
http://www.cs.unc.edu/Research/vrpn/
http://www.vrgeeks.org/vrpn/tutorial---use-vrpn

Post by **Nick3DvB** » Thu Aug 16, 2012 9:17 pm

Alternate USB camera source (old, before the official directshow source, but maybe useful?)
http://kougaku-navi.net/backyard/archiv ... webcam.zip
http://www.geocities.jp/in_subaru/ewclib/index.html

PTAM sources with some extensions for “gesture recognition” ?
http://ptam-gr.googlecode.com/svn/trunk/src/PTAM/

Linux PTAM on bootable USB stick (no image, bash script and instructions)
http://www.kameda-lab.org/_local/imagel ... v21/PTAMk/

...and a few random experiences compiling PTAM for windows, maybe a help to someone:
http://brianhoffmann.de/journal/thesis/ ... tam-win32/
http://translate.googleusercontent.com/ ... 6yGsuusoZw

ROS Virtual Machine images: http://code.google.com/p/brown-ros-pkg/ ... alMachines
ROS Sensor Fusion: http://www.ros.org/wiki/ethzasl_sensor_fusion
ROS VPRN Client: http://www.ros.org/wiki/ros_vrpn_client

ROS seems to publish everything through it's "tf" stack, if we want to output that as vrpn we really need a ros_vrpn_server right?

It makes a lot of sense to use vrpn, it seems to be the gold standard, but the sensor interface aspect of the Rift SDK is still a bit of an unknown...

There has been some discussion of vrpn in the FreePIE thread: http://www.mtbs3d.com/phpBB/viewtopic.p ... =75#p75580

It might be worth trying to use ethzasl_sensor_fusion to do the PTAM / IMU sensor fusion in ROS itself?

I'm not up to speed on the Rift's Hillcrest Labs sensor yet: http://www.hillcrestlabs-devstore.com/s ... h-2/Detail

So will have to take a good look at libfreespace to: http://libfreespace.hillcrestlabs.com/

When I get time I am going to try and get PTAM running a ROS VM, but I am moving house next week so will be really busy for a while.

Post by **FingerFlinger** » Fri Aug 17, 2012 10:11 pm

Okay, so I am currently trying (and failing) to compile PTAM on Windows; it occurred to me that if we get something worth sharing, we'll need a better installation method. What is the legality of packaging all of these libraries into something reasonable and distributing it? PTAM is free for non-commercial use, but it is owned by Isis Innovation

Post by **divide** » Sat Aug 18, 2012 3:55 am

I successfully compiled the latest version of PTAM on Visual C++ 2008, takes some time !
I also integrated the USB webcam support using the library linked on PTAM's blog. Looks like perfs on Windows are not optimal, but if you want to play with it here it is !

PTAM Project and Binaries (VC2008-USB)

PTAM Licence wrote:Isis’s permission is not required if the said reproduction, modification, transmission or transference is done without financial return

After camera calibration is done (grab 5 pictures of a checkerboard, click optimize, save if RMS <0.3) you may have to rename camera.cfg to settings.cfg before running PTAM.exe

Post by **Nick3DvB** » Sat Aug 18, 2012 2:22 pm

Thanks for posting the binary Divide, that makes life a lot easier!

It will be interesting to compare the performance, we might even find a linux VM is faster that the win32 port after all, I have the same problem with Dreambox emulators, that's why I started using VMs. I will have a good look at ROS when I get time but we might be able to get PTAM running on one of the tiny linux distros (<64MB image!). I am still waiting for my PSeye, when it arrives I might try doing an IR filter / lens mods, increasing the viewing angle is supposed to help PTAM, but maybe that is only for flying drones etc, and I'm not sure if it can correct a very large fisheye distortion?

http://nuigroup.com/forums/viewforum/34/

Post by **FingerFlinger** » Sat Aug 18, 2012 5:26 pm

Hey divide, thanks for uploading the binaries! I tried all morning, but I wasn't able to compile it myself...

EDIT: What kind of tracking quality are you getting? Mine is not very impressive; maybe my house is particularly bad for this...

EDIT2:Peau Productions has some wide-angle lenses and mounts available for the PS3 Eye. Up to 113 for the FOV.

Projects Backed · Post by **LeeN** » Sat Sep 01, 2012 7:28 pm

I tried integrating PTAM with my VR X project and discovered something disappointing about it. The accuracy is actually very poor.

The only reason it looks good in PTAM itself is because it pins an object to something visible in the video. Sometimes you might see a small amount of wobble to that object which would normally be okay because it is in still the same relative position in the video but if you try to extract a camera matrix and use it for head tracking, it is extremely noisy.

If I wanted to do something useful with PTAM for VR X, it would not be for head tracking, but for pinning an x server display to a checkerboard pattern.

If anyone is curious, the way I integrated it was rather a cheat. Because VR X already creates a window and has it's own libraries for doing vector math and I didn't want to deal with this, I used IPC to create shared memory and then had PTAM write it's matrix to the shared memory, and had VR X read the shared memory to integrate the resulting matrix. If anyone working in linux would like to try the same thing, let me know and I will create a patch that can be used against PTAM, and then a small c++ class for reading the matrix from shared memory.

Post by **FingerFlinger** » Sat Sep 01, 2012 8:18 pm

Hmm, that's too bad. I still want to try LibViso2, on the merit of the video they released, but I've been spending time on another project for the past couple of weeks. I'll do a write up once I have something to share.

Post by **divide** » Sun Sep 02, 2012 10:41 am

I also gave up with PTAM, now working on a stereo camera system. Hopefully it will be more stable thanks to known distances.

Post by **Aasimon** » Fri Sep 07, 2012 9:54 am

I think this is were DTAM comes into play.
http://www.google.co.uk/url?sa=t&rct=j& ... YN7Ea0fswA

Post by **brantlew** » Fri Sep 07, 2012 10:04 am

@Aasimon: Nice link.

The hardware requirements are pretty stiff. You would probably need separate dedicated computer to handle tracking. Maybe a lower resolution solution would work for gross position tracking instead of AR.

This demo was run on a commodity NVIDIA GTX 480 GPU hosted by an i7 quad-core CPU. Video: 640x480 at 30Hz 24-bit color

Post by **FingerFlinger** » Fri Sep 07, 2012 12:02 pm

Very impressive; looks like I have another paper to read and not understand...

Post by **cybereality** » Fri Sep 07, 2012 6:40 pm

Here's an embed for lazy people:

[youtube]http://www.youtube.com/watch?v=Df9WhgibCQA[/youtube]

Good stuff.

Post by **Aasimon** » Sun Sep 09, 2012 3:22 am

The hardware requirements are pretty stiff

If you code it up nicley with something like OpenCL a second gfx card or an AMD APU would do the trick.

Post by **Nick3DvB** » Fri Nov 09, 2012 7:50 pm

Hey all, long time no post, been away for a few months (long, depressing story...)

Just been catching-up on everything, so PTAM was a no go after all, but DTAM looks very promising.

As Palmer is going to be in London for Evolve maybe he should swing by Imperial Collage and give Richard Newcombe a heads-up.

http://www.doc.ic.ac.uk/~rnewcomb/
http://www2.imperial.ac.uk/robotvision/ ... people.php

He seems like a slam-dunk for either of these positions:

http://www.oculusvr.com/careers/senior- ... ical-flow/
http://www.oculusvr.com/careers/senior- ... -tracking/

Just a thought...

Post by **Aasimon** » Mon Nov 12, 2012 5:58 am

I like the look of the DTAM results.

For a 'sat at your desk' type option we can use a fixed position Kinect (Or Creative cam) that has a fast path for watching head movement and a slower kinematic solver for whole body (at last what every is above your desk) tracking.

This should allow for accurate VR headset position tracking and body interaction from one camera?

Post by **Chriky** » Mon Nov 12, 2012 9:48 am

Looks like a great algorithm but its pretty resource hungry. Unfortunately game devs and gamers generally assume they can use the whole GPU for rendering. This pretty much uses the GPU up itself, for tracking. It could be used in a large scale simulation that offloads all calculations to a server farm, but I'm not sure its quite ready this generation for home gamers.

It's also solving a much harder problem than we need. You can get drift free orientation quickly from an IMU, that could be fed into the solver to make it much faster. You'd only be solving for 3 position values not 6 pose values.

Do you really even need a tracker this good for home VR? Based on the redirected walking experiments it seems like as long as you react quickly and get it roughly right, your brain buys it. I think you only need something this good for AR or VR in a touchable environment.

Post by **FingerFlinger** » Mon Nov 12, 2012 1:18 pm

That's my thinking as well. For an odometric VR solution, I don't even really care about the mapping portion of these algorithms. I would be happy with a cheaper algorithm that purely tracks the motion without context; this would at least allow for parallax effects and whatnot.

Post by **Nick3DvB** » Mon Nov 12, 2012 4:01 pm

It makes a ton of sense to use as much IMU data as possible to help lighten the load, but even as is, many would consider the performance penalty worth it, after all (some) people are happy enough to dedicate a whole GPU to Nvidia's dubious PhysX system. But granted it is probably over-kill for tethered VR, I'm sure we are all looking forward to the day a camera based system can be used for VR "laser-tag" in any building, without the need to install complex marker arrays or pre-scan the environment with LIDAR / Kinetic etc, just mapping then generating the game world to match the physical geometry of the playing space, in real-time, we live in hope...

Post by **zalo** » Thu Nov 15, 2012 4:44 pm

BoofCV just added real time Stereo Camera Visual Odometry to their library.
[youtube]http://www.youtube.com/watch?v=D4I4NHSGaOc[/youtube]

It's an opensource java computer vision library. Maybe someone can get the code from this for a more CUDA implementation.

It can also do dense stereo disparity (getting depth maps from stereo cameras), so that should be useful as well should the rift have stereo cameras.

Post by **brantlew** » Thu Nov 15, 2012 5:02 pm

Cool, thanks for the heads up.

Post by **mahler** » Thu Nov 15, 2012 5:43 pm

Thanks!
This is great news... driftless positional & orientation tracking seems to be in reach.
I'm curious to try this out see if I can use other sensors or certain type of visual input to be used for automatic-recalibration.

Post by **Nick3DvB** » Fri Nov 16, 2012 10:03 pm

This looks very cool, but it would probably need porting - last time I checked Java wasn't known for it's efficiency...

I've always thought stereo cams are the way to go, I'd bet they'll be on the final retail Rift.

Post by **PatimPatam** » Sat Nov 17, 2012 6:31 am

That's very good indeed.

I think the real challenge (apart from the latency issue) is going to be making it work under harsh (but real) conditions.. like standing in a living room looking up to an empty ceiling, with almost 0 reference points..

Post by **brantlew** » Sat Nov 17, 2012 9:47 am

PatimPatam wrote:That's very good indeed.

I think the real challenge (apart from the latency issue) is going to be making it work under harsh (but real) conditions.. like standing in a living room looking up to an empty ceiling, with almost 0 reference points..

I have a feeling you will always have these type of visual boundary conditions and you can't predict all the variety of spaces that it would be used in. I think the only way to combat it will be to create a tight sensor fusion between IMU and camera. So when looking at a flat featureless wall, the coordinates would almost exclusively be derived from IMU. You might even have to switch back to 3DOF for brief periods under certain circumstances if there just wasn't enough information to compute a solution.

Post by **zalo** » Sat Nov 17, 2012 9:50 am

That's where inertial measurement units come in. They just have to get the feed to realize that all of its data is garbage. Maybe a certainty measurement would help it blend between the gyro and the odometer.

Dang it, Brantlew ninja'd me.

Post by **Nick3DvB** » Sat Nov 17, 2012 3:22 pm

I wonder if two cheap laser-pens mounted next to the cameras could help with the white wall problem, not parallel but at a known angle, say 30 deg. Given a known player height and measuring the distance between the dots should allow you do some basic floor / wall range detection, occlusion would be an issue but you could always fall back to IMU. You might blind a few innocent bystanders if you tired it outside, but you would look bad-ass, a bit like this guy!

Post by **brantlew** » Sat Nov 17, 2012 11:09 pm

Another possibility advantage of an IMU / optical-flow sensor fusion algorithm would be to counter-act the latency of the optical flow computations. Optical flow does take a lot of resources. I got the viso2 demo up and running on my Intel i7 and was able to process about 9 stereo frames per second on a single thread.

A couple of thoughts...first the input data for this demo uses two 1344x372 stereo images which is quite a bit larger than typical webcam resolutions. If you could move to two 640x480 or even (gulp!) 320x200 stereo images and achieve acceptable results that would increase your frame rates a great deal. Additionally, processing stereo images naturally lends itself to parallelization so you could probably easily double your frame rate by processing on dual threads. So moving to at least 30 Hz seems very doable in my opinion.

If a 60 Hz IMU signal could be sensor fused with a 30 Hz optical flow signal you might be able to achieve acceptable tracking results. Position tracking over short time durations on an IMU is not that bad so you might get the best of both worlds - the lower latency of inertial and the higher stability of optical.

The other problem with optical flow is the drag on the host CPU. I did a quick test of running a game simultaneously with the viso2 demo to see the impact. The results were promising. The game frame-rate drop was minimal. (moving from about 55 FPS to 52 FPS), while the viso2 demo dropped from about 9Hz to 8Hz. So there is some impact but not enough to scare me away from this possibility. The benefits of simple 6DOF tracking would outweigh a 5% decrease in game performance in my opinion.

Post by **FingerFlinger** » Sun Nov 18, 2012 1:36 am

My current project is to put together a bare bones optical flow tracker. I really want to see what performance is possible once the mapping portion is removed. My idea is that, for parallax effects, we largely don't care about drift. So the plan is FAST, create an egomotion matrix, and compare the results to an IMU.

brantlew, did you have an easy time getting LibViso2 set up? I was having trouble with it in VS2010 Express.

Post by **brantlew** » Sun Nov 18, 2012 9:18 am

@Finger: Yeah, I'm on the same track right now. I also want to see how far I can get with pre-existing optical flow implementations. Academics are not renowned for the ability to write efficient code - mainly for their tendencies to over-abstract everything. I do a lot of image processing optimization code at my job, so I'm hoping that experience can help speed up these implementations.

LibViso wasn't so bad to get going - not as hard as PTAMM was. I couldn't link the dependencies, so eventually I just included all the dependency code in the compile to avoid dll-hell. The easiest project to get going was BoofCV. I love Java. Just wish it was fast enough for stuff like this.

Post by **mahler** » Sun Nov 25, 2012 9:25 am

Is BoofCV performance interesting compared to LIBVISO2?

Post by **brantlew** » Sun Nov 25, 2012 10:00 am

Boof performance is pretty slow, but it is very easy to build and run the project since it is all in Java. The samples actually run in real-time as a Applet on the website. I think a C++ port of Boof, or a JNI-assisted version might be interesting.

Post by **Fredz** » Sun Nov 25, 2012 7:45 pm

I wonder if anyone has already seen this video :

[youtube]http://www.youtube.com/watch?v=xfJCUf1uD3M[/youtube]

It's not exactly the kind of visual odometry you're talking about, but it does uses a PS Eye (and a PS Move) and shows rotational + positional tracking. Could be a good candidate for head tracking when putting the PS Move on the head like in Project Holodeck.

I think the PS Move could also be positioned horizontally instead of vertically on the head in order to look a little bit less weird.

And it would make the ball completely visible when looking up too.

I've tested the PS Move API this video is based on and it does work without a problem on my Linux machine with a Bluetooth adapter. Compatibility seems to be less ideal on other platforms though (Mac OS X and Windows) by reading their mailing list and issues tracker.

I didn't find an executable showing the same thing as in the video, I guess it has not been commited to the Github repository yet. There seems to be initial support for sensor fusion though, and in the mailing list the author talks about his master thesis that is based on this and should be submitted at the end of november. So I guess it's only a matter of weeks before the source is available in Github.

Still, I've tested the other examples and the results are quite good, you can see live sensors values (accelerometer, gyroscope and magnetometer) and positional values from optical tracking (x, y and distance) but not both at the same time.

The frequency looks good but I really don't know which value to trust. The test_tracker app for optical tracking shows around 800Hz but the PS Eye is supposed to go around 120fps. Maybe it's because the camera only acquires a small region of interest. The test_read_performance app which measures sensor reading performance shows 85Hz, but it may be related to Bluetooth reading only.

The latency seems quite good, I don't notice lag when watching the PS Move and the image of the ball on the screen while moving it.

The accuracy for the x and y positions also looks good and stable, but the accuracy for the distance (ball diameter) seems a bit unstable. When putting the PS Move on a table there is some oscillation in the values, but the image of the ball on the screen doesn't seem to change. So no millimeter accuracy there, much closer to centimeter accuracy. But I guess that's only a problem with the recognition algorithm that is used and that it may be corrected by using a better one.

Post by **brantlew** » Sun Nov 25, 2012 8:27 pm

There are two categories of camera solutions that are being developed. External, inward-looking cameras like TrackIR and MemeBox's project. And outward-looking cameras like PTAM. This would fall into the "external" camera category. But the results look very good and external camera solutions benefit from low processing requirements and low latency so they may well end up being the best solution.

Post by **Fredz** » Tue Nov 27, 2012 8:06 pm

Outward-looking cameras look kinda overkill for small indoor environments, not very fast, stable nor very easy to implement. But for outdoors I guess that would be the most promising solution, especially for things like your redirected walking technique.

I tried the PS Move API again today just to discover that the author committed OpenGL examples for his sensor fusion algorithm some hours ago. Nice timing.

I'm now able to do the same thing than in his video, I still don't feel any latency (should try to measure that) and the FPS counter shows around 500 fps.

There is a problem with the accuracy for the orientation though, if I move randomly and come back to the initial pose the orientation is not always displayed correctly. The error can be as high as 90° with very fast and random movements and around 10° for slower and less random ones. I guess the sensor fusion implementation is not finished yet, I'll wait for some more updates before testing this for head tracking.

Post by **FingerFlinger** » Tue Nov 27, 2012 9:26 pm

But egomotion solutions are going to be more generalized, provided that the performance level is acceptable. I think this is probably pretty important for widespread adoption. Definitely not there yet, of course.

Post by **Chriky** » Wed Nov 28, 2012 1:52 am

What code is that you're looking at Fredz?

Post by **Fredz** » Wed Nov 28, 2012 2:28 pm

It's the PS Move API linked in my previous post.