PS3 Eye - PTAM/Visual Odometry

[MemeBox]

Reading this thread has made me do an about face!

I had not seen that egomotion is more general than an external type system, it's made me switch my thinking.

I'm not sure that any SLAM solution I have come across would be robust enough. But I could see some rgbd type cameras like the kinect enabling this.
There was some promising results from:
http://www.youtube.com/watch?v=mqp3bWqxDsc

I believe this combines a kinect fusion type approach, with the more usual Visual type approach. I was looking forward to the source being shared with pointclouds.org, since this was developed in one of their sprints, but it seems now that it won't be released. Although there is a home grown version of kinect fusion called kinfu, also at pointclouds.org.

[brantlew]

Reading this thread has made me do an about face!

I had not seen that egomotion is more general than an external type system, it's made me switch my thinking.

I'm not sure that any SLAM solution I have come across would be robust enough. But I could see some rgbd type cameras like the kinect enabling this.
There was some promising results from:
http://www.youtube.com/watch?v=mqp3bWqxDsc

I believe this combines a kinect fusion type approach, with the more usual Visual type approach. I was looking forward to the source being shared with pointclouds.org, since this was developed in one of their sprints, but it seems now that it won't be released. Although there is a home grown version of kinect fusion called kinfu, also at pointclouds.org.

Yeah, a lot of us have come to the conclusion that optical flow techniques are probably the ultimate goal - but it sure consumes a lot of resources. There are a few isolated videos and demos that suggest it is workable, but nobody is sure whether it can be currently done in real-time (alongside a game engine) or not.

[mahler]

... but nobody is sure whether it can be currently done in real-time (alongside a game engine) or not.

Not sure either. Using an embedded system could solve that problem, that might reduce the processing power needed.

[brantlew]

... but nobody is sure whether it can be currently done in real-time (alongside a game engine) or not.

Not sure either. Using an embedded system could solve that problem, that might reduce the processing power needed.

The thought has certainly crossed my mind. For simplicity, it would be great if you could do all the important bits using a smart phone camera and CPU, and then just push the coordinates over to the host computer.

I've been playing around with the Viso2 code and a Minoru 3D webcam - just getting it all put together. But my frame rates are pretty low... < 10 FPS.

[FingerFlinger]

Turns out my problem was with CMake not liking VS Express for some reason. I just tried it on Pro, and it build with no issues at all... So hopefully I can start playing tonight! I'd like to compare the feature matching performance to OpenCV's.

[brantlew]

Turns out my problem was with CMake not liking VS Express for some reason. I just tried it on Pro, and it build with no issues at all... So hopefully I can start playing tonight! I'd like to compare the feature matching performance to OpenCV's.

Are you working with mono or stereo cams?

[FingerFlinger]

Mono right now. I've got 3 PS3 Eyes, but I haven't synched any of them yet.

EDIT: Actually, I have an idea for a "multi-mono" setup, but I am still trying to understand that math, so it may not have any performance benefit. I'll describe it after I get home from work today.

[brantlew]

My gut tells me that mono solutions will be less reliable and require more processing - so I'm starting with stereo. I had an old Minoru laying around so I'm starting with that just learning how to rectify images and what-not. Minoru frame rate is totally crap but hopefully I can still learn with it before moving to PSEye.

[FingerFlinger]

Essentially, my thinking is that, if you have multiple cameras oriented orthogonally to each other, one of them will always be able to capture "sideways" motion, which *should* be detectable with less robust (read:faster) feature matching. It should also lead to a more precise optical flow field, and with better performance. The question is whether that would out-pace a more sophisticated stereo algorithm (and whether any of my assumptions are true). I suppose that orthogonal stereo pairs would be even better, but it starts getting expensive in terms of cost and video throughput at that point... Actually, the best is probably stereo-pairs with fish-eye lenses, but I haven't got any.

[brantlew]

Yeah, I could see that. Essentially a "mouse" algorithm along all three axis.

[FingerFlinger]

Yeah basically.

[bobv5]

My sleep times are all messed up, that makes me have odd ideas. Sometimes good, sometimes very bad. What about using chips from optical mice? I have seen people use them as scanners, they are very fast compared to webcams, maybe useful with different lenses? I assume they are low res, but at fairly high update rate maybe some software voodoo can compensate?

Just an idea. Sorry if it sucks.

[brantlew]

An interesting idea. It would require some research to figure out how to pipe the imagery into those chips.

[bobv5]

Assuming the physical/optical parts are possible, would you be able to get the data from the chips?

[brantlew]

I would just leave the mouse I/O in there and try to intercept it via mouse driver.

[FingerFlinger]

That's an interesting thought. I don't really have any idea what kind of visual information a mouse would require in order to work poperly. Perhaps just some optics with a very large depth of field? I guess I should read up specifically how mouse vis-odo works.

[Nick3DvB]

I remember someone bringing this up in the old 6DOF thread, we would have to deal with the surface illumination / focus problem for it to be an option.

I wonder if you could feed an array of "mouse DSP" chips with small sections of pixels from a standard camera CCD? (~320x240@120Hz)

[lessthanoptimal]

Hey guys, I was searching the web to see if anyone has used the VO code in BoofCV that I posted and found this thread. Not sure about everyones background, but I would like to offer any general assistance/nudges in the right direction for getting VO working. As a warning, most cheap webcam type devices don't mix well with VO. I'm curious about LibViso and how well it works for you. It's been on my todo list for a bit to try out.

Also if anyone was still willing to give BoofCV a try, I can offer advice on how to get that to run fasert. One thing that did strike me as odd was that the applet ran in real time, but whatever code you were running ran really slow. That suggests to me that image input method might be the culprite, that you are trying to process large images, or something is configured incorrectly.

[Chriky]

Hey, great to have you lessthanoptimal!

From what I can tell most people here are pretty smart and have an amateur interest in optics, electronics, maths and programming with most people having a particular interest within that. However, as far as I know, there's nobody who has specialist knowledge of VO.

Perhaps you could give a brief recap of the state of the art if you are familiar with it
Basically, how capable are the different algorithms, what is the resource use, what are the main faults etc.

The most popular webcam people use is the Playstation Eye because it can do high frame rates (180FPS for 320x240 and 70FPS for 640x480)

[FingerFlinger]

Hey, great to have you lessthanoptimal!

From what I can tell most people here are pretty smart and have an amateur interest in optics, electronics, maths and programming with most people having a particular interest within that. However, as far as I know, there's nobody who has specialist knowledge of VO.

Perhaps you could give a brief recap of the state of the art if you are familiar with it
Basically, how capable are the different algorithms, what is the resource use, what are the main faults etc.

The most popular webcam people use is the Playstation Eye because it can do high frame rates (180FPS for 320x240 and 70FPS for 640x480)

Seconded! It would be great to have somebody with a deep knowledge of VO weigh in.

[lessthanoptimal]

A lot of the more recent work that I have seen has been using the Kinect/RGB-D sensors. I suspect that's because its much easier to work with than a regular stereo/monocular camera. So for the stereo case, the mathematics has essentially be solved for 15+ years. Robust, precise, and fast feature extraction is still being worked on. Monocular case is different and most papers in the literature that I have found are full of implicit assumptions, with the primary being that the camera has a wide field of view. The problem is that in the mono case the geometry for triangulation is very poor for standard cameras and general motion. So with a single camera, the hard case is if you are moving forward along the camera's optical axis, if the camera is moving perpendicular to the optical axis it's easily solved.

Here is an example of mine where perpendicular motion was used for 3D vision from a single camera:
http://boofcv.org/index.php?title=Examp ... gle_Camera

Here is someone who made additional assumptions and claims to have monocular VO working using a webcam:
http://www.hessmer.org/blog/2010/08/17/ ... -odometry/

I've been meaning to try that out myself. Now a comment about webcams. Rolling shutters will cause problems since different parts of the image are from different moments in time. Many stereo algorithms rely of the cameras being synchronized, so that images are captures at the same moment in time. I would say that the unsynchronized case where the baseline between the camera is unknown is an open problem.

[brantlew]

Hi lessthanoptimal, great to have you on the forum as a resource since several of us are just taking baby steps into this domain. I'm a software engineer at a company that does medical image processing so the concepts of feature detection and correlation are not totally foreign to me - but the specifics of the equipment and the real-time nature of the problem are challenging.

I think I was misunderstood when I was talking about the BoofCV Applet examples. They run equally well in Applet or application form - I was just pointing out how you could demo the code right there on the website (gotta love Java ) I saw your performance comparison graphs for BoofCV versus C implementations. It seems like for certain filtering operations, a JNI library could allow you to overcome some of those deficiencies.

Interesting to hear your take on the state of the field right now. I also got the impression that monocular solutions did not have great performance yet and I'm glad you feel the same. I didn't realize that about movement along the Z axis - but it makes sense. I also surveyed the Hessmer solution and it assumes a fixed height/fixed angle platform - great for robotics but not VR.

I'm starting down the stereo path and libViso just to get a basic understanding of the problem space. How bad is the camera synchronization problem? It would seem to be a function of frame latency and speed of motion. A slow moving camera (positionally) could tolerate a good deal of camera desynchronization, and similarly a high frame-rate camera could seemingly overcome synchronization problems - no? What stereo camera rig are you using?

[lessthanoptimal]

Yeah for low level array heavy operations there really isn't much you can do to get Java as fast as C/C++. Things start to even out when you move to higher level and more complex algorithms. That's why some feature detection/description code is actually competitive or faster in BoofCV than OpenCV, well that, and more efficient algorithm being used.

Another issue with the monocular case is that there is a scale ambiguity, which is impossible to resolve unless you know the size of an object in the camera's view. Have you guys looked at ARToolkit? It won't do everything that you want but could be used to provide camera pose for a single view and is well written.

I have only worked with synchronized cameras, mostly PtGrey Bumblebee2 (http://www.ptgrey.com/products/bumblebee2/) while not cheap, they do save you a lot of time. Those cameras have a synchronized shutter and gain, which makes things even easier. Plus they do a good job rigidly mounting the cameras. Image quality leave a bit to be desired.

You are right, errors caused by synchornization will be speed dependent. After you set up your stereo rig my suggestion would be to do this experiment. Put the camera on a small triipod, take a picture of a static scene, then move the camera two inches or so, and repeat 30 times. That way you will have a clean set of data that is synchornized from your stereo setup and can see how well VO is working. I'm not an expert with USB, but I believe it buffers data and that could throw off your synchronization scheme using time stamps.

When setting up your stereo system, make sure everything is rigidly mounted and won't budge. A rotation of 1 or 2 degrees could cause a lot of frustration. Saving your calibration grids for future use would be prudent.

[mahler]

Software engineer here too, but also just starting down the VO path.

I have only worked with synchronized cameras, mostly PtGrey Bumblebee2 (http://www.ptgrey.com/products/bumblebee2/) while not cheap, they do save you a lot of time. Those cameras have a synchronized shutter and gain, which makes things even easier. Plus they do a good job rigidly mounting the cameras. Image quality leave a bit to be desired.

This is quite a roadblock. These are expensive.
You said it would save a lot of time, but what is the alternative?
If we were to buy two separate (cheaper) camera's, what are the options to get them synchronized?
And which lower budget camera's could you recommend?

I read that the PS Eye has better specs than most webcams

640×480 pixels @ 60 Hz
320×240 pixels @ 120 Hz

I could buy two of these for ~$100 and mount them together.

Actually, I'm hoping the Leap Motion ($70) can be used for this.
Unfortunately these won't arrive until February 2013

[Chriky]

Two PS3 Eyes should be even cheaper than that, even. The problem is we can only really sync them by running them really fast, which makes them pretty dim or very grainy...

[mahler]

Two PS3 Eyes should be even cheaper than that, even. The problem is we can only really sync them by running them really fast, which makes them pretty dim or very grainy...

A quick search got me these:

• 
  _______________________________
• Minoru ($90USD - $30 at Amazon)
  
  
  _______________________________
• HYTEK 3D iVCam - Stereo 3D Webcam Suite ($60USD for complete setup - or $29USD for just the driver to use with any pair of webcams - USB or firewire - that support RGB24 and 320x240 or 640x480)
  
  
  _______________________________
• And something about a martian tripod from a long time ago ($???)

[lessthanoptimal]

I think a lot of the cheap stereo webcams don't synchronize the shutters either. Double check before buying. Professional cameras for computer vision often have a shutter pin that's easily accessible just for this purpose. Have also heard of people hacking the camera's circuitry so that they can provide an external trigger.

GoPro offers a stereo camera with synchronization:
http://gopro.com/hd-hero-accessories/3d-hero-system

Don't know anyone who has worked with a stereo GoPro and don't know if you can plug a go pro into a computer and use it like a webcam either.

A few months ago I was thinking of this problem. It should be possible to use unsynchronized stereo, but you end up with what amounts to a stable version of monocular VO.

[brantlew]

Here is another interesting option, although $550 is still a bit expensive. And I'm not certain, but I don't think these cameras are synchronized.



http://robotzeitgeist.com/2008/10/cheap ... amera.html
http://www.surveyor.com/

I bought two pre-owned PS3 Eyes for $16 a piece at my local Gamestop so I consider that a steal, but it will require some work to mount them properly.

In the meantime I have been using the Minoru. Finally got some preliminary motion test results last night with it. I'll post a video soon.


Edit: I've been reading a little more about the Surveyor. The interesting thing about it is that it's built for DIY projects so the firmware is open source and it has some built in image processing capabilities. Ideally, you could do some of the "heavy-lifting" right there on-board and then pass over either partially processed features to the main CPU or possibly even fully processed coordinates - wirelessly even. So that might make the $550 price tag a bit more appealing if it could also work as an embedded image processor. It would be great if you could just strap it to the top of your HMD without a wiring hassle and without taxing the host CPU much.

[FingerFlinger]

I seem to remember that the PS3 Eye cameras have a Vsync pin that you can connect from camera-to-camera that is supposed to synchronize the rolling shutter.

EDIT: I misremembered here's a thread from CL that discusses it a bit. http://codelaboratories.com/forums/viewthread/84/
I'm not entirely sure, but it seems like the FSIN pin (frame sync input) will block other cameras until the Master is ready. I haven't read the datasheet yet.

[Fredz]

For the synchronization problem another solution would be to use a single camera with a beam splitter, like the Loreo 3D Lens in a Cap.

[brantlew]

For the synchronization problem another solution would be to use a single camera with a beam splitter, like the Loreo 3D Lens in a Cap.

An interesting idea, although your FOV and resolution would be halved.

It's funny how those Loreo images almost look Rift compatible. If you stuck a custom warping lens on the front of each of the stereo lenses, I wonder if you could create a totally optical input source for the Rift?

[mahler]

For the synchronization problem another solution would be to use a single camera with a beam splitter, like the Loreo 3D Lens in a Cap.

An interesting idea, although your FOV and resolution would be halved.
It's funny how those Loreo images almost look Rift compatible. If you stuck a custom warping lens on the front of each of the stereo lenses, I wonder if you could create a totally optical input source for the Rift?

Basically you're solving the synchronization issue the same way the Rift does with HMD's
Here is a 3D video shot in with the lens

[youtube]http://www.youtube.com/watch?v=doUeqq9bq9A[/youtube]

Somebody could try it with a DIY rift

[Fredz]

An interesting idea, although your FOV and resolution would be halved.

Yes, but with a PS Eye it makes sense because it does have a 75° FOV. You would get 37.5° for each eye, as a comparison the Minoru 3D Webcam has a 42° FOV. The resolution would not be that great though.

[mahler]

The resolution would not be that great though.

Is there a known (linear or other) relationship between resolution and visual odometry perfomance?

[Chriky]

If you call CLEyeCameraStart on both PS3Eyes at exactly the same time, then call CLEyeCameraGetFrame at exactly the same time every frame, you'd think they would be synced to within 1ms right? I think you could do a good tracking algorithm similar to the absolute position calculating system I've been trying to make. You'd have to do it in a room covered in these LEDs but you wouldn't need to calibrate it at all so it would be feasible for a home holodeck.

The LEDs come out like this and can be extracted quickly...


The stereo matching should be really easy. Because the cameras are side-by-side, corresponding points must show up at the same vertical position in the images, and they will be very close in colour, and they will be in the same order left-to-right (I think).

Given the two screen positions of a point, you can work out the position in camera space (see where the extended vector lines cross) so for each frame-pair, you get a 3D point cloud. Then you just need to match up between point clouds and do a matrix inversion to get the frame-to-frame transformation.

[lessthanoptimal]

The resolution would not be that great though.

Is there a known (linear or other) relationship between resolution and visual odometry perfomance?

It's been my experience that the the baseline (distance between the cameras) and how much blur (motion or lens) there is has the biggest effect on accuracy. Take that BoofCV video, images are down sampled to 320x240 so that it will run fast even on older computers and you still get good results.

[lessthanoptimal]

For the synchronization problem another solution would be to use a single camera with a beam splitter, like the Loreo 3D Lens in a Cap.

An interesting idea, although your FOV and resolution would be halved.

It's funny how those Loreo images almost look Rift compatible. If you stuck a custom warping lens on the front of each of the stereo lenses, I wonder if you could create a totally optical input source for the Rift?

It's worth a shot. What worries me a bit is that the custom warping lens could introduce really weird distortions that camera calibration can't handle. All the calibration programs I have seen assume radial and maybe tangential only. You should be able to see based on the results from calibration if this is an issue or not.

[lessthanoptimal]

An interesting idea, although your FOV and resolution would be halved.

Yes, but with a PS Eye it makes sense because it does have a 75° FOV. You would get 37.5° for each eye, as a comparison the Minoru 3D Webcam has a 42° FOV. The resolution would not be that great though.

Not exactly what you were talking about, but FYI, having a wider FOV will produce better results than high resolution and narrow field of view.

[brantlew]

It's worth a shot. What worries me a bit is that the custom warping lens could introduce really weird distortions that camera calibration can't handle. All the calibration programs I have seen assume radial and maybe tangential only. You should be able to see based on the results from calibration if this is an issue or not.

Not exactly what you were talking about, but FYI, having a wider FOV will produce better results than high resolution and narrow field of view.

We're running two lines of discussion. The warping lens would be for viewing images in the Rift.

For visual odometry we would not want to distort the images. Visual odometry would be used for for positional head tracking - not for viewing. We could maybe fit a wide angle lens onto each stereo lens to increase FOV. It seems like a really interesting idea and could really simplify the synchronization problem.

[FingerFlinger]

In case it wasn't already common knowledge, Peau Productions has a selection of wide FOV lenses for the PS3 Eye.