PS3 Eye - PTAM/Visual Odometry

[lessthanoptimal]

Not 100% related to this thread, but I got good stereo vision results using a Rovio robot. A Rovio only has a single camera, which really sucks and is highly compressed. This makes me a bit more hopeful that using the PS3 eye might actually work for VO.

http://peterabeles.com/blog/?p=181

The camera blurs too much while moving so all images are taken when stationary. A PS3 eye has much less motion blur and works better in low light.

[mahler]

I found this study about ACTS: Automatic Camera Tracking System

http://www.zjucvg.net/acts/acts.html (updated with ACTS 2.0 in 4th of Dec 2012)

See this page for a video from 2009 compared to PTAM
http://www.cad.zju.edu.cn/home/gfzhang/ ... etracking/

[brantlew]

So I built my first stereo camera this week. Pretty much a copy of the one that was posted here a few weeks ago. Those custom mounting boards are wonderful! The main difference is that I put a 12cm separation between the cameras. That matches the spec of the Bumblee so I figured that's a good starting point. I haven't tried to wire the VSYNC yet (not ready to potentially break my new camera). I'll test first without sync and see how necessary it is. Also I haven't ordered new lenses yet but the plan is to eventually swap out for some wide angles. Here's some build pics.

CamAssembly1_small.JPG

Opening the PS Eye was pretty easy and those mounting boards are a life-saver.

CamAssembly2_small.JPG

I was too impatient to order tiny standoff spacers so I just separated the boards with extra nuts. Luckily I didn't short out my cameras.

StereoCam_small.JPG

The finished camera (sort of). I still need to put a mounting hole on the bottom so I can attach it to a camera tripod. Not bad for $50 bucks though.

[cybereality]

Nice stuff.

[Chriky]

Great work - can you run some tests to see how synced they are?

[mahler]

I'm not sure about the performance, but this seems to do the job of positional tracking quite well.

[youtube]http://www.youtube.com/watch?v=mqp3bWqxDsc[/youtube]

An extension to our Kintinuous: Spatially Extended KinectFusion system showing real-time fused colored mesh volume generation.

More information here: http://www.cs.nuim.ie/research/vision/data/rgbd2012/

Thomas Whelan, John McDonald from Department of Computer Science, NUI Maynooth
Michael Kaess, Maurice Fallon, Hordur Johannsson, John J. Leonard, CSAIL, MIT

And another more recent video (Sep 25, 2012) of Robust Tracking for Real-Time Dense RGB-D Mapping with Kintinuous

[youtube]http://www.youtube.com/watch?v=sxKgP2rpFzc[/youtube]

The location inside the generated 3D space seems to be updating really fast.

More info & paper: http://www.cs.nuim.ie/research/vision/data/icra2013/

Thomas Whelan, John McDonald from Department of Computer Science, NUI Maynooth, Hordur Johannsson, Michael Kaess, John J. Leonard, CSAIL, MIT

[brantlew]

Finally getting some results from my new camera...

http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=16117

[youtube]http://www.youtube.com/watch?v=FmRu0dAw46Y[/youtube]

[mahler]

Finally getting some results from my new camera...
http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=16117

Really happy seeing some actual results instead of us just talking about theory.
The bandwidth and processing power issue was to be expected, but this is a very good start.
I think you're right in your conclusion for now. We'll get there in a few years though.

It will still be fun to experiment a bit with the setup and do some troubleshooting.

Besides I still hope that with algorithm optimizations and perhaps tweaking the hardware a bit (lenses/vsync/filter) --- then all built into a dedicated processor, it could already speed up results to an acceptable level.

[lessthanoptimal]

brantlew: That looks like a good start. One thing that might help for debugging is if you draw the inlier and outlier feature set on each image. That way it's easy to see if certain kinds of motions break the tracker/VO.

FYI Viso2 is tuned for operation on a moving car and will drift faster than keyframe based approaches when stationary.

[brantlew]

brantlew: That looks like a good start. One thing that might help for debugging is if you draw the inlier and outlier feature set on each image. That way it's easy to see if certain kinds of motions break the tracker/VO.

FYI Viso2 is tuned for operation on a moving car and will drift faster than keyframe based approaches when stationary.

ok thanks for the tips. Yeah I understand the drift during motion, but I wasn't expecting so much stationary drift. I'll try Boof out at some point for comparison, but it makes it a bit more complicated since I have to bridge from C to Java.

[lessthanoptimal]

brantlew: That looks like a good start. One thing that might help for debugging is if you draw the inlier and outlier feature set on each image. That way it's easy to see if certain kinds of motions break the tracker/VO.

FYI Viso2 is tuned for operation on a moving car and will drift faster than keyframe based approaches when stationary.

ok thanks for the tips. Yeah I understand the drift during motion, but I wasn't expecting so much stationary drift. I'll try Boof out at some point for comparison, but it makes it a bit more complicated since I have to bridge from C to Java.

I'm working on improving the BoofCV code now actually. There is a dataset where viso2 is able to estimate motiion but boofcv fails. For bridging the code, my recommendation would be to use a TCP socket or something like that to output the 3D pose information. Even if you stick to pure C that approach has some advantages over running everything as the same app.

[FingerFlinger]

Is anybody else having problems with the PS3 Eye cameras disconnecting randomly? I'll be going along fine and then one of them will drop out and require me to plug-and-unplug it.

[brantlew]

Is anybody else having problems with the PS3 Eye cameras disconnecting randomly? I'll be going along fine and then one of them will drop out and require me to plug-and-unplug it.

Yep. It happens to me a lot in stereo mode when there is a large change in contrast. I have to be fairly careful with the cameras when I pick them up not to swing them too violently from light to dark or the driver flips out and disconnects.

[mahler]

From the other thread: http://www.mtbs3d.com/phpBB/viewtopic.p ... 117#p94936

Re painting walls, have you come across Dazzle Camouflage? (see attachment) It was used in both world wars to combat the U-boat threat and was dreamt up by a navy officer and artist who had studied cubism. I have often wondered if it could be used to produce VR 'fashion' if it turns out that it improves camera NUI interfaces. In the war it was used to prevent the U-boats calculating speed and direction but in VR the opposite might be true. The clothing idea was given a little credence when I found this: http://people.csail.mit.edu/rywang/uppe ... _final.mp4
You may find his webpage interesting http://people.csail.mit.edu/rywang/

I wonder if this will work

[mahler]

Realizing that my previous post was slightly off-topic, here's something we'll probably end up needing for the ideal orientation and positional tracking in VR.

Fusion of IMU and Vision for Absolute Scale Estimation (PDF)

[FingerFlinger]

@mahler

Cheers for the link to that paper. I just began working out some techniques for scale estimation over the weekend; this will help.

@brantlew

Dang, I was hoping I just had a bum camera.

[mahler]

Another interesting, more recent paper from the same lab (June 2012)

Real-Time 6D Stereo Visual Odometry with Non-Overlapping Fields of View

[youtube]http://www.youtube.com/watch?v=Yl3DbBUdnUw[/youtube]

[...] Our non-overlapping stereo system runs in real-time and employs the single camera VO framework from [15] on each camera individually, augmented by a windowed BA in order to increase robustness and accuracy of the estimated motion and structure.

[...] The computational time remains similar

[...] Note that, even though this VO framework employs information from an additional IMU in order to recover relative rotation priors in between successive frames, we still do not make use of the IMU for scale estimation here. Hence, the individual motion estimates could just as well be delivered by a vision-only based monocular odometry.

[...] The entire odometry system runs at a frame rate of 15 Hz (~70ms) on a 32 bit Intel Core i7 2.8 GHz machine with 4 GB RAM

I also found the PhD from one of the co-authors very interesting:

Real-Time Scalable Structure from Motion: From Fundamental Geometric Vision to Collaborative Mapping

It's a rather broad overview of the different studies done by their labs which explains concepts, (mathematical) models, challenges and different methods for single, multi-camera setups and fusion with IMU.

[Nick3DvB]

In case you guys missed this, CodeLabs / NUI group are putting together an "open" Leap alternative, Kickstarter going live soon: http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=16868