PS3 Eye - PTAM/Visual Odometry

Post by **brantlew** » Thu Dec 13, 2012 1:59 am

That's basically what's going on with the first mod pictured above in the project box. I think he just removed the backing and screwed the front half to the project box. That is certainly an option. You can just order the modified lens assembly from Peau and put it right back into the original shell like that. The custom lens assembly includes a filter of your choice the lens mounting bracket, and a lens of your choice. And it is a much cheaper option if you get the cameras elsewhere and disassemble them. Only $109 for two lens assemblies plus the cost of the cameras - so about half the price.

I think I read that forum post as well. Someone later posted a remark that the camera was auto enhancing for low light, but if you turned off that feature then you wouldn't get those blooms.

Post by **MSat** » Thu Dec 13, 2012 2:53 am

Ok, I see the screws now. Initially I thought it was mounted by the spherical part. Looks easy enough.

$110 sounds like a reasonable amount for lenses+filters on a one off stereo camera setup. Probably the best way to go if you need IR-pass filters.

You're right about the autogain part. It was mentioned in the following post I think. Apparently you can now control most (if not all) the camera settings in linux (and presumably windows). Good to know.

Post by **mahler** » Thu Dec 13, 2012 5:10 am

brantlew wrote:The only thing left would be to wire up the VSYNC to synchronize the cameras and you should have a pretty nice stereo rig to play with.

Good looking setup, but the size seems a bit bulky.

The PS3 Eye Toy Stereo Vision System in the picture you posted seems more practical. Easy to adjust the camera distance, lightweight and probably be easier to wire up the VSYNC because they are closer together. Combining this with the filter and m12 lens mount I think it's more suitable for stereo vision experimentation. And saving you $100 on 2 casings and instead buy 1 or 2 scrap PCBs and a bar to attach them. This seems realiable and cheap enough to start buying parts and experiment. How much more reliable would this be compared to the minoru 3d webcam?

What does he mean with the "good version" and the "bad version" of the OEM lens? Are there different versions of the PS Eye out there? And it's not clear to me which filters and lenses would be best suited for Stereo VO experimentation - perhaps this is something that lessthanoptimal can answer.

____________________________________________________________
ps. On the flickr page from that guy with the stereo vision system I also found a pretty good looking casing

But no information on where he got the aluminum case.

Post by **brantlew** » Thu Dec 13, 2012 8:23 am

Yeah, that case looks nice. Plus it looks like he bought the CS lens mount kit from Peau. I wonder where he gets those scrap PCB's that seem to line up perfectly? I guess you could cut your own or even make one out of metal, but it would be nice to have those holes already machined out to avoid alignment problems.

Post by **mahler** » Thu Dec 13, 2012 8:38 am

brantlew wrote:Yeah, that case looks nice. Plus it looks like he bought the CS lens mount kit from Peau. I wonder where he gets those scrap PCB's that seem to line up perfectly? I guess you could cut your own or even make one out of metal, but it would be nice to have those holes already machined out to avoid alignment problems.

From his description below the pictures he says the following

zwdeal wrote: The blue PCB is custom made with mounting holes for different scenarios -- two M6 [holes] whose centers are aligned with center of lens, one 1/4 hole for tripod mounting, and four M2 holes for other type of mounting.

zwdeal wrote: M6 mounting hole if the PCB is not used in this aluminum case. Along with the ohter M6 hole, they allow PS3 Eye to be mounted on other fixtures such as 8020 aluminum

Finally found a perfect fit M12 0.5 lens mount.

Post by **brantlew** » Thu Dec 13, 2012 8:45 am

Ah crap. That's too bad. Was hoping for an easy purchasable solution instead of customizing a design and getting something manufactured. I'm not too familiar with that process.

Post by **brantlew** » Thu Dec 13, 2012 11:24 am

mahler wrote:This seems realiable and cheap enough to start buying parts and experiment. How much more reliable would this be compared to the minoru 3d webcam?

I don't know about reliability, but there is no way the quality could be worse than the Minoru. What a piece of crap that device is! I know now that the majority of the latency and variability I have been experiencing with CV experiments is due to that camera. Whatever is going on (autogain, data compression ?), it just cannot keep a steady frame rate, dropping below 5 FPS frequently in low light. It's hard to do just a simple checkerboard calibration with it because there is so much variability in the frame-rate and motion blur in the images. So after much frustration, I finally just came to the conclusion that I need to do it the right way and build my own rig. I'll post my progress with this effort.

Post by **mahler** » Thu Dec 13, 2012 11:30 am

brantlew wrote:I'll post my progress with this effort.

I am actually also interested in building the setup and test SVO software for positional tracking and perhaps help develop if necessary.
I noticed you got a reply from that guy about the custom PCB. If you could arrange to get 2 extra let me know

Post by **FingerFlinger** » Thu Dec 13, 2012 12:07 pm

What does he mean with the "good version" and the "bad version" of the OEM lens?

There is a version of the PS3 Eye for which it's easy to remove the IR filter, and a version for which it's very difficult. They are visually identifiable before cutting anything apart; Peau has a description somewhere on his site. I've done the mod on the "easy" one, and the only problem is that it's basically impossible to focus it perfectly again. You will definitely want a proper IR band-pass, by the way. I've done the floppy disk filter as well, and it works... okay, but it further degrades the clarity, and leaks a lot of visual spectrum through.

Post by **mahler** » Thu Dec 13, 2012 12:34 pm

FingerFlinger wrote: There is a version of the PS3 Eye for which it's easy to remove the IR filter, and a version for which it's very difficult. They are visually identifiable before cutting anything apart; Peau has a description somewhere on his site.

As long as they are identifiable before buying them online, I'm good

I've done the mod on the "easy" one, and the only problem is that it's basically impossible to focus it perfectly again.

I this problem only related to the "easy" one?

You will definitely want a proper IR band-pass, by the way. I've done the floppy disk filter as well, and it works... okay, but it further degrades the clarity, and leaks a lot of visual spectrum through.

Is this to avoid the focus issue you mentioned or just in general? What is proper and what is 'the floppy disk filter'?

Post by **FingerFlinger** » Thu Dec 13, 2012 12:58 pm

As long as they are identifiable before buying them online, I'm good

That's probably not possible, although Peau sells cameras that are vetted for this, or he did in April.

I this problem only related to the "easy" one?

The problem is how the lens assembly is constructed. On the "easy" one, it's possible to remove the IR filter without damaging the lens, but it's horrendously difficult to not damage anything on the "hard" one. With the IR filter removed, the optics have changed slightly, so the assembly is not set up properly anymore. On mine, it went from roughly infinite focus to about 6 inches, but still very usable for blob tracking.

Is this to avoid the focus issue you mentioned or just in general? What is proper and what is 'the floppy disk filter'?

It might help with the focus issue, but it would depend on the optical qualities of the filter you choose. Really though, it's just to block the visible spectrum more completely. The floppy disk filter is literally a piece of magnetic floppy disk sitting in the path of the lens; it will block some visual spectrum and pass through most IR, but as I said it doesn't do either particularly well, and has poor optical quality.

Post by **FingerFlinger** » Thu Dec 13, 2012 1:06 pm

By the way, according to this thread, it sounds like you can simply hook up an oscillator to the FSIN pins for easy synchronization; that would be nice if it is true!

Post by **brantlew** » Thu Dec 13, 2012 1:29 pm

All of the hard/easy modding of the stock Sony lens is a moot point if you get the custom lens assembly from Peau. The whole Sony lens contraption is simply removed from the CMOS and a new mount, lens, and filter are then added back - plus then you can then select from a custom filter and a wide range of lenses, so there are no residual focus problems either.

http://peauproductions.com/store/index. ... ucts_id=85

Post by **mahler** » Thu Dec 13, 2012 1:39 pm

brantlew wrote:All of the hard/easy modding of the stock Sony lens is a moot point if you get the custom lens assembly from Peau.

True, that's the plan anyway. The question remaining: Which band pass filter and lens would be best for VO?

Post by **brantlew** » Thu Dec 13, 2012 2:13 pm

Well for the stuff in the beginning of this thread like traditional PTAM and VO, you would want the visible spectrum. If you just want IR tracking like MemeBox's project, or Wiimote-like things, or IR-based VO then you would want one of the IR-pass filters. The frequency will depend on your light source. I would suggest trying to get a light source and filter as far out on the spectrum as you can - 940nm so you will have the least interference with visible lights.

Post by **FingerFlinger** » Thu Dec 13, 2012 2:21 pm

Specifically, as far away as possible from the environmental light, if for some reason you happen to know what wavelength your living room lights' peak output power is centered around.

Post by **mahler** » Thu Dec 13, 2012 2:47 pm

This well-known PTAM video was made in 2007 and clearly shows a map of the environment in realtime with what it seems just a normal camera. And the iPhone app from 2009 is from the same Active Vision Group. Based on this map, shouldn't it be easy to deduce the camera's 3D position and orientation from the code? I kinda skipped this PTAM thing at the beginning of the thread and it's slowly raising some questions for me.

[youtube]http://www.youtube.com/watch?v=Y9HMn6bd-v8[/youtube] [youtube]http://www.youtube.com/watch?v=pBI5HwitBX4[/youtube]

I forget for a minute, why are we still trying so hard to use special camera's for better performance and using different libraries? Updates of this project are now maintained @ http://ewokrampage.wordpress.com/ which was mentioned on the first page of this thread. What about the algorithms which were used. How does this relate to LIBVISO2 and/or BoofCV?

[/rant]

Post by **brantlew** » Thu Dec 13, 2012 3:08 pm

mahler wrote:I forget for a minute, why are we still trying so hard to use special camera's for better performance and using different libraries? Updates of this project are maintained here as you probably already knew, but I didn't. What about the algorithms which were used. Are these the same as in OpenCV, LIBVISO2 and/or BoofCV? How is everything related?

I think they are all related in the sense that they involve feature detection and frame to frame correlation. But I am not sure how different they are in the choice of the edge detectors, the way they search the space, etc.. Monoscopic computer vision would probably be best in terms of hardware simplicity, but the anecdotal evidence I have heard is that the problem is more easily solved stereoscopically. Even in that video, you can see some jitter in the tracking and the occasional large mistake. But then again, I have also seen some really good monoscopic videos on moderate hardware. So I don't know either. We're all just noobs here on this topic so the choice of direction seems pretty open ended.

Post by **EdZ** » Thu Dec 13, 2012 3:23 pm

I think I've found a cheaper source of lenses and M12 mounts for the PS3 eye than Peau's. I'm looking at 2.1mm lenses (for multi-camera IR point tracking), so things work out at $14.11 + filter per camera, rather than $54 from Peau, though you'd still have to source your own filters.
Any M12 board lens holder with a 18mm hole spacing should work with the PS3 eye. You can then use any standard M12 lens and filter combo.

Post by **FingerFlinger** » Thu Dec 13, 2012 3:33 pm

I've actually been wanting to contact JonOlick about the AR demo that he made earlier this year. It was very smooth, with little jitter and only slightly noticeable lag. But I don't think that he ever told us in-depth about his camera setup, or feature detection/correlation.

@ Edz - Nice find!

Post by **lessthanoptimal** » Thu Dec 13, 2012 3:43 pm

mahler wrote:
brantlew wrote:What does he mean with the "good version" and the "bad version" of the OEM lens? Are there different versions of the PS Eye out there? And it's not clear to me which filters and lenses would be best suited for Stereo VO experimentation - perhaps this is something that lessthanoptimal can answer.

In general, wider FOV result in a more stable solution. In practice I would just try a few options and see what works best.

I have a few PS3 Eye cameras now and planning on creating a stereo setup. Some other projects have a higher priority at the moment so it could be a few weeks. Lacking a proper metal shop, my plan is to first try mounting the cameras on wood and hope that I can make it rigid enough. Or wait until you guys come up with a better solution

Looking at the PTAM videos they use planar surfaces. If you have a planar surface you can resolve the camera's 6-DOF motion with a single camera fairly well. I haven't done much work with planar surfaces but their results do look good.

Post by **lessthanoptimal** » Thu Dec 13, 2012 3:57 pm

brantlew wrote:
mahler wrote:I forget for a minute, why are we still trying so hard to use special camera's for better performance and using different libraries? Updates of this project are maintained here as you probably already knew, but I didn't. What about the algorithms which were used. Are these the same as in OpenCV, LIBVISO2 and/or BoofCV? How is everything related?
I think they are all related in the sense that they involve feature detection and frame to frame correlation. But I am not sure how different they are in the choice of the edge detectors, the way they search the space, etc.. Monoscopic computer vision would probably be best in terms of hardware simplicity, but the anecdotal evidence I have heard is that the problem is more easily solved stereoscopically. Even in that video, you can see some jitter in the tracking and the occasional large mistake. But then again, I have also seen some really good monoscopic videos on moderate hardware. So I don't know either. We're all just noobs here on this topic so the choice of direction seems pretty open ended.

Does OpenCV have working VO code? The library has all the components you would need to do VO, but I don't recall seeing finished code that actually worked.

I've been looking a bit more at the LIBVISO2's source code since I'm processing their datasets. LIBVISO2's algorithm should take better advantage of using a stereo camera than the current approach in BoofCV, which is designed to be more generic and work on a Kinect. Preliminary results show both libraries have about the same performance for accuracy. Out of their 12 training datasets, there is one case where BoofCV does have trouble tracking features. I'm working on fixing that. BoofCV and LIBVISO use totally different tracking techniques and based on a crude comparison, seem to have similar runtime speed.

Post by **Fredz** » Thu Dec 13, 2012 4:05 pm

brantlew wrote:Monoscopic computer vision would probably be best in terms of hardware simplicity, but the anecdotal evidence I have heard is that the problem is more easily solved stereoscopically.

I think it's because you only need to find point correspondences on 1D lines instead of 2D plans, which should be a lot easier, faster and also more robust. It may have other benefits for the extraction of the cameras orientations and positions relative to the scene also.

Post by **brantlew** » Thu Dec 13, 2012 4:10 pm

lessthanoptimal wrote:Does OpenCV have working VO code? The library has all the components you would need to do VO, but I don't recall seeing finished code that actually worked.

I think it just has all the tools to put one together. I'm just using it for camera interface and image preparation before feeding it to libViso.

lessthanoptimal wrote:I've been looking a bit more at the LIBVISO2's source code since I'm processing their datasets

Cool. Glad to have your insights. One of the reasons I chose libViso right off the bat was because it obviously worked perfectly well with grayscale imagery. How does BoofCV operate in this regard? Is there a performance gain from using RGB? It would seem that using color information would help with feature correlation but I'm just guessing.

Post by **brantlew** » Thu Dec 13, 2012 4:17 pm

Fredz wrote:I think it's because you only need to find point correspondences on 1D lines instead of 2D plans, which should be a lot easier, faster and also more robust. It may have other benefits for the extraction of the cameras orientations and positions relative to the scene also.

Well from a high level standpoint - stereoscopic data would also seem to contain a lot "richer" information. From a single monoscopic image it's hard to figure out Z coordinates and so movement is required to do accurate scene reconstruction - and that's true even if you double your mono resolution. But for the same data quantity, stereoscopic images at half the resolution also contain an implicit camera "movement" so the data has a lot more contextual content than mono images.

Post by **FingerFlinger** » Thu Dec 13, 2012 4:35 pm

@ brantlew
What kind of detector are you using with LibViso?

I don't think OpenCV has VO code, but it's got plug-and-play Optical Flow in a couple of varieties.

Post by **mahler** » Thu Dec 13, 2012 4:51 pm

EdZ wrote:I think I've found a cheaper source of lenses and M12 mounts for the PS3 eye than Peau's. I'm looking at 2.1mm lenses (for multi-camera IR point tracking), so things work out at $14.11 + filter per camera, rather than $54 from Peau, though you'd still have to source your own filters.
Any M12 board lens holder with a 18mm hole spacing should work with the PS3 eye. You can then use any standard M12 lens and filter combo.

True, but the filter is $24 so that would be $40 per camera, rather than $54 from Peau ... and that's excluding double shipping you'll have to pay
List of filters: http://peauproductions.com/store/index. ... cPath=2_12

Still.... good find

Post by **EdZ** » Thu Dec 13, 2012 5:08 pm

Any 12mm diameter (M12) filter will fit, as well as any other filter that will fully cover the sensor.

Post by **brantlew** » Thu Dec 13, 2012 5:08 pm

FingerFlinger wrote:@ brantlew
What kind of detector are you using with LibViso?

I couldn't tell you. I skimmed through the paper a few weeks ago but didn't retain the details and I haven't buried into the source code either because I'm too preoccupied just trying to get a good and properly calibrated live feed.

Post by **brantlew** » Fri Dec 14, 2012 8:56 am

post deleted

Post by **brantlew** » Fri Dec 14, 2012 9:07 am

Good news guys. I was able to purchase a small lot of those custom PS Eye mounting PCB boards from the designer. They are being slow shipped from China so it might take a week or two to get them. I am only using 4, so I will have several extra pairs left over. If any of you guys are interested just PM me and we can make arrangements.

Post by **lessthanoptimal** » Fri Dec 14, 2012 1:35 pm

brantlew wrote:Cool. Glad to have your insights. One of the reasons I chose libViso right off the bat was because it obviously worked perfectly well with grayscale imagery. How does BoofCV operate in this regard? Is there a performance gain from using RGB? It would seem that using color information would help with feature correlation but I'm just guessing.

Hah yeah BoofCV works well with gray scale images. Almost every algorithm is grayscale only, color image processing is one of its weaknesses. The videos show RGB images simply because they are visually more appealing. Color feature trackers will typically be more stable than grayscale ones, but are a lot slower.

Post by **brantlew** » Fri Dec 14, 2012 2:14 pm

lessthanoptimal wrote:Hah yeah BoofCV works well with gray scale images. Almost every algorithm is grayscale only, color image processing is one of its weaknesses. The videos show RGB images simply because they are visually more appealing. Color feature trackers will typically be more stable than grayscale ones, but are a lot slower.

I only work with grayscale so I am not familiar with how multichannel image processing works. Something like derivative calculation for edge detection would be tricky since the transitions may occur from one hue to another at the same intensity level. I can certainly see how the added degrees of freedom would complicate the calculation. (I need to read up on that)

I'll have to check out Boof once I get some good data to work with. Good to know that all the algorithms use grayscale. Is that purely for performance reasons since single channel processing is a whole lot easier and faster?

Post by **Fredz** » Fri Dec 14, 2012 3:00 pm

brantlew wrote:Good to know that all the algorithms use grayscale. Is that purely for performance reasons since single channel processing is a whole lot easier and faster?

I know at least one feature detector that was implemented for color, found it in a paper from Cordelia Schmid. From what I remember it was better than other feature detectors in greyscale, and much better than standard Harris or KLT detectors (which seem to be used in BoofCV). You can access her papers here : http://lear.inrialpes.fr/people/vandeweijer/research

Post by **lessthanoptimal** » Mon Dec 17, 2012 9:07 pm

brantlew wrote:
lessthanoptimal wrote:Good to know that all the algorithms use grayscale. Is that purely for performance reasons since single channel processing is a whole lot easier and faster?

I've noticed that most CV algorithms are developed for gray scale. My suspecion that it's for performance reasons. CV is so computationally expensive that a 3x or more speed hit isn't worth it unless it improves its performance by a lot. The same general statement is true for BoofCV, gray scale works well enough that I don't feel the need to implement color algorithms.

The mounting boards you purchased. Do they require 80/20 to be mounted on?

Fredz: Thanks for the paper reference.

Post by **brantlew** » Tue Dec 18, 2012 12:21 am

lessthanoptimal wrote:The mounting boards you purchased. Do they require 80/20 to be mounted on?

It doesn't look like it. You could just use those holes in the mounting board to sink some screws through wood or metal if you wanted. But the channels in the 8020 makes it convenient to adjust the stereo separation it looks like to me. A clever and cheap solution. I think you could probably easily attach a nut to the bottom of the 8020 as well so the whole contraption could be mounted on a standard camera tripod.

Post by **brantlew** » Wed Dec 19, 2012 9:42 am

@lessthanoptimal: Do you have any suggestions for the amount of camera separation to configure? My initial thought was human interocular, but looking at the BumbleBee they have a 12 cm baseline. My guess is that it depends on the expected distance to objects in the environment. How much does stereo separation affect the performance of these VO algorithms? Do you have rough guidelines for good baseline distances in indoor and outdoor environments?

Post by **FingerFlinger** » Wed Dec 19, 2012 10:31 am

@lessthanoptimal

I've got a question, too! My understanding is that the math for VO is pretty much solved, so I was wondering what is the lower limit for the number of features required to resolve egomotion?

I don't know if you have read this entire thread, but we briefly talked about using multiple cameras in a minimal-overlap arrangement, so that the motion is always (roughly) orthogonal to one or more of the cameras. The idea is that you could use very aggressive feature detection, resulting in a quite sparse but reliable feature set. This should improve performance during the optical flow phase fairly drastically.

If someone knows of any research on this, I would be very interested. Haven't been able to turn up any papers.

Post by **Fredz** » Wed Dec 19, 2012 3:03 pm

FingerFlinger wrote:My understanding is that the math for VO is pretty much solved, so I was wondering what is the lower limit for the number of features required to resolve egomotion?

The minimum number is 7 correlated points for 2 cameras, but you get much better results with 8 points (hence the famous eight-point algorithm). See : http://en.wikipedia.org/wiki/Fundamenta ... er_vision)

But the results will be very sensitive to noise, so in reality you need much more points than that, and that's actually where the problems arise. It's very difficult to find with certainty which points in 2 sets are the same seen from a different perspective, you need for example to detect outliers which would otherwise make the estimation go completely wrong. For that you need non-linear techniques such as Ransac, LMedS, etc. which are very computationally intensive and not perfect.

The required number of points to have a good precision for the extraction of camera parameters is very dependent on the scene itself also.

Post by **lessthanoptimal** » Wed Dec 19, 2012 10:23 pm

Fredz wrote:
FingerFlinger wrote:My understanding is that the math for VO is pretty much solved, so I was wondering what is the lower limit for the number of features required to resolve egomotion?
The minimum number is 7 correlated points for 2 cameras, but you get much better results with 8 points (hence the famous eight-point algorithm). See : http://en.wikipedia.org/wiki/Fundamenta ... er_vision)

7 is the lower limit for the uncalibrated case. If you know the camera's calibration the minimum is 5 points. Every VO algorihm I know in practice assumes that the cameras are calibrated. If the cameras are not calibrated the camera's motion can't be solved for in Euclidean space, it will instead be distorted by some unknown transformation. Some papers claim to have a minimum less than 5 points. They typically do that by making additional assumptions, such as limiting motion to a plane.

I agree with everything else Fredz says. Well I guess I would make it more specific. The mathematics, once features have been extracted, is solved. Reliable feature extraction in less than ideal conditions is still open. There are plenty of cases where I look at sequence and can pick up features myself but even the best CV algorthm will fail. One of my complains about all the "standard" datasets is that they are too idealized and don't cover realistic conditions found even on a nice sunny day.