Oculus Rift and cameras for augmented reality

[amnipar]

Greetings, I am new to these forums, I just registered after having backed the Oculus Rift on Kickstarter. I did my best to search your forums for related discussion, but really couldn't take the time to read through the whole Rift topic (nearly 100 pages??). So please bear with me if some of this has been covered.

I am not a gamer; I am a Ph.D. student researching computer vision methods for augmented reality applications. I have been dreaming about a really good HMD for video see-through augmented reality that wouldn't break my bank. I have tried some Vuzix headsets but the FOV is really disappointing. Hence, it didn't take me too many seconds to decide whether to back Oculus Rift or not. But, immediately after receiving my Rift (perhaps after playing a few rounds of Doom BFG, I *did* play Doom 1&2 and Quake when I was a kid) I will start experimenting with hooking it up with a stereo camera. That's what video see-through augmented reality is - you have an HMD showing live video of the scene so you can move around and see some overlayed virtual information.

Now I decided to start this new topic here because this seems to be the "home" of Rift and hopefully I can get @PalmerTech and others familiar with the Rift to discuss some technical things related to this kind of modification project. I'm guessing that there are also others who are interested in this kind of application for Rift, so those people could join the conversation. I gathered from some comment made by Palmer in the Rift topic, that you have tried camera-based motion tracking, but haven't got the latency good enough yet. I can imagine that this would be the case. But initially, I don't want to use the cameras for tracking, but for creating the image displayed on Rift. To do this, I would need some specific information about what kinds of cameras and optics I should get and what kind of processing I would need to do to create a video signal suitable for displaying on Rift. If I have two cameras with, say, 90 or 100 degree FOV, would I need to do some additional pre-distortion or would it be enough to just crop and combine the images? What kind of overlap the cameras FOV should have to make it look 'right'? Should I point the cameras straight ahead or a bit inwards so the centerpoints would meet at some distance other than infinity?

I am aware that a major problem with this kind of system would be the latency between capturing the images and showing the combined image on Rift. But I am willing to start with a lower resolution and some latency and work on this to improve the experience. I have also some electronics knowledge and would be ready to expend some time and money to create a special camera board to avoid using usb cameras and to synchronize the views. I just wish to start gathering the knowledge and ideas well ahead of receiving the Rift. I have some small cameras with raw serial connection and M12 threads for mounting different lenses, so I could go and purchase some wide-angle lenses if someone can suggest some good specifications.

Also, in my research I am working on some algorithms for scene analysis that can be run on GPU and that might allow really fast tracking and object recognition; that is unpublished work so I won't go into more details yet. But I would certainly try using these on Rift as soon as I get the cameras working, and after publication the algorithms would be open source - I'm not in any funded research project so that's my decision to make.

Finally, I wish to make everyone dream a bit about future possibilities. While I'm no gamer but a scientist, I do have one gaming application in mind that would be perfect for this kind of technology, and that I would play personally. Imagine live action role playing games that are played in real cities and other locations, and players need to really move, talk and act. The catch would be, that all players have HMDs that show a completely re-created scene but with the same structure as the real scene. So where there is a wall there would be a wall, where there is an obstacle there would be an obstacle, where there is a person there would be a person. Same constraints, but with completely re-designed look. Also, if you read "Daemon" and "Freedom(tm)" by Daniel Suarez you will get more ideas what could be done (though much of his technology is a bit unfeasible). This would require some insane computing power and algorithms, not to speak about insane content design effort (and smaller and more inconspicuous HMDs to avoid scared people calling police and/or military to stop a Borg invasion). But given the current speed of progress, that might be within reach in 10-20 years, even earlier. But to get there, the work on the algorithms and technology must start now. And Rift could be a platform that allows people to get started.

[FingerFlinger]

A note on distortion latency: instead of doing it inside of software, consider using lenses on your cameras that pre-distort the image before sending it to the Rift; basically how the old LEEP system used to work. Also, try to get a hold of MTBS user JonOlick, he started a thread showing a pretty impressive AR demo. http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15002

[zalo]

If you do AR, you're going to have to undistort the image anyway for tracking and the like. Better to just use normal optics and do all your distortion in software. If you're already doing stuff on the GPU, it would probably be easier and faster to add the distortion as an extra step in your existing setup. The exact distortion code will be available in the Oculus SDK. If that's fast enough for games (being rendered AND distorted), then it's fast enough for real life (tracked and distorted).

Besides, where would he find the lenses for prestortion?

Also, Olick's demo looks awesome! If that tracking is as buttery smooth, real time, and accurate as it looks, we have a winner!
I was afraid we'd have to do magnetometer/accelerometer/gyroscope/opticalfiduciarytracking sensor fusion to get a fast drift free result like that.

[FingerFlinger]

Ha, good point; I was just thinking about piping the video straight through.

EDIT: Also, Olick's demo doesn't use any kind of markers; it tracks elements in the scene!

[WiredEarp]

@ amnipar

The catch would be, that all players have HMDs that show a completely re-created scene but with the same structure as the real scene. So where there is a wall there would be a wall, where there is an obstacle there would be an obstacle, where there is a person there would be a person.

Yep, this is part of the VR future IMHO. There are quite a few people on this forum who are interested in this. My own angle is of VR lasertag, where you can pick up virtual weapons and powerups etc, while running around like in lasertag.

AR is one possible angle to approach this from, since you could use markers to display the right graphics etc, however I think the best solutions will involve a system that can track multiple HMD's and guns at once. This would simplify stuff like the ability to use explosive effects to damage people you can't directly see (such as shooting a rocket into a doorway). Using AR without absolute tracking like this, would mean that would be a difficult scenario to take into account. Currently optical seems the best bet for the tracking of the HMD itself, but we might need some other breakthrough in wide area absolute tracking to make this work at a reasonable cost/accuracy.

[LeeN]

be careful what you wish for:
http://www.youtube.com/watch?v=PKTN93kdVK0

[amnipar]

If you do AR, you're going to have to undistort the image anyway for tracking and the like. Better to just use normal optics and do all your distortion in software. If you're already doing stuff on the GPU, it would probably be easier and faster to add the distortion as an extra step in your existing setup. The exact distortion code will be available in the Oculus SDK. If that's fast enough for games (being rendered AND distorted), then it's fast enough for real life (tracked and distorted).

But my impression was, that the distortion is at least partly because of the wide FOV. If I go for normal cameras, will I get wide enough FOV? Undoing the radial/barrel distortion due to the wide angle lens for tracking and object recognition purposes wouldn't be too hard to do. My question is, if I want to get a live video image for the Rift screen, would it be enough to use wide angle lens and them just crop and merge the two images into the DVI signal, or do I need something else?

[zalo]

Ha, good point; I was just thinking about potential latency gains and costs.

If the variable acuity resolution of the reverse distorted image doesn't throw your tracker off, by all means use a fish eye to get th video feed. I'd still like to know where you acquire your lenses though.

Just be aware that the rift is focused just under infinity, so to help your eye match up, you might want to set the cameras focus there too. As long as your optics directly match the rift's, there shouldn't be any extra steps. For this reason, you might want to wait until you have one to start building your lens assembly.

[EdZ]

It should be possible to use matching optics for the camera in order to avoid doing software undistort->modify->redistort for video pass-through. This is how the old LEEP system worked with the puppet/tele-head. You'd still need to correct the video to rectilinear for processing and overlay, unless you rewrite whatever image processing algorithms you're using to work on non-linear images, but I'm guessing it would be more acceptable to have lag free video and slightly laggy overlays than have overlay and video synced but both lagging (though this might depend on application).

[amnipar]

My first step would be to get pass-through video with as low latency as possible. Like I said, I have some small cameras that have a lens mount with M12 thread. There are many wide-angle lenses available for such mount, up to 113 degrees. I could purchase some lenses if I figure out what would be a good choice. I was thinking that perhaps it would be best to get a bit wider angle lens than required, so that I would have a bit of extra space to experiment with a good cropping for the Rift video feed. If the horizontal FOV of Rift is 90 degrees, what should the FOV of each camera be, and how much overlap they should have? If the horizontal FOV of Rift is 90 degrees and diagonal is 110, maybe something like 104 degree lens would be enough..?

@PalmerTech or someone else familiar with Rift, can you give some opinion on this?

Tracking smoothness requirement indeed depends on the application. For simple annotations it's not critical, but for overlaying virtual objects there is nearly zero tolerance. That certainly can be achieved with some work, but I will first concentrate on the video feed. And like someone mentioned above, visual tracking does not require markers, there are methods for tracking natural visual features in the scene. That requires quite much computing power, clever algorithm design, or both. But it can be done. Like I said earlier, a great technical platform that is widely available with little cost will certainly inspire research and development. There are already some AR research labs that have announced their interest in Rift.

For attracting the interest of AR researchers, there is the ISMAR conference in Atlanta this November. Might be a good idea to showcase Rift there.

[FingerFlinger]

Having a higher FOV for the camera than for the display has some advantages too. You can lock on to new objects as they enter the camera's FOV, but while they are still "in the black" for the display. Eliminate some of the lag that the user would experience when the cameras are acquiring new tracking targets

[mastasky]

I'd like to pick up this discussion, because I am also interested in using lenses in front of a webcam (or 2 to be specific) to pre-warp images for the OR. Has anyone been giving this topic any more thought? Is Oculus using standard lenses or are they custom produced? If standard, I am sure it wouldn't be too hard to find matching lenses for the distortion.

[geekmaster]

I'd like to pick up this discussion, because I am also interested in using lenses in front of a webcam (or 2 to be specific) to pre-warp images for the OR. Has anyone been giving this topic any more thought? Is Oculus using standard lenses or are they custom produced? If standard, I am sure it wouldn't be too hard to find matching lenses for the distortion.

The Rift lenses need barrel distortion to compensate for their pincushion distortion. Fisheye lenses are extreme examples of barrel distortion. A lesser degree of fisheye such as many wide-angle lenses (or just using the central portion of a fisheye lens) should compensate for the Rift pincushion just about right.

[zoost]

[youtube]http://www.youtube.com/watch?v=yeXWmKzbFwI[/youtube]

This is the work of John Olick. Very promising for use in the Rift

http://www.tentonhammer.com/firefall/ne ... ality-game

[mastasky]

Thanks geekmaster. I am aware of how the distortion works (barrel/pincushion), but I was wondering if Oculus uses standard, off the shelf lenses with standard specs. That way we could use equally off the shelf lenses on the camera side to distort the image perfectly and not just approximately. This way any need for software distortion (=lag) would be removed.

I might just take my Oculus lenses (once arrived) to an optics lab for measuring, but I was wondering if anyone actually knew the specs already.

[amnipar]

Hi all, it's a long time since my previous post! I recently got my Rift, and now I report my first progress with the cameras. I took apart a Minory 3D webcam I have lying around, taped it in front of my Rift (see picture) and replaced the original lenses. This is a convenient webcam (in addition to being a stereo cam) in that it has an m12 lens mount with the option of replacing the lenses. I inserted a pair of 2.1mm/104 degree lenses I had available (bought from Peau Productions web store, http://peauproductions.com/store/index. ... ducts_id=1), and wrote a simple app that just draws the images side by side.

rift-hack.jpg

After a few rounds of adjusting the image positioning (they have to overlap a little, at least with my lenses), magic happened: my brain accepted the visual input and fused the two images! I get quite convincing depth perception, though everything appears to be much closer than in reality. This is due to the cameras being several centimeters in front of the eyes, but also to the camera field of view, which appears to be a bit too narrow. There is also a noticeable pincushion effect in the periphery of the visual field, which suggests that the barrel distortion from the cameras is not sufficient to cancel out the pincushion distortion from the Rift. The latency from the cameras is noticeable, but surprisingly tolerable. I placed two 640x480 images side by side, and the image edge is visible only at the very top of the field of view, so even though the display is 800 pixels high, it's not necessary to fill all of that, which is good for the purpose of using cameras. With some scaling and cropping it would be possible to fill the FOV completely of course.

I have already ordered a pair of 1.7mm/113 degree lenses that will hopefully work better. Even wider angles would probably work if the image is cropped. I'm also planning to dismantle two PS3 Eye cameras (that are faster then Minoru) and place them a bit farther apart. If the speed is still not enough, I will start to consider using some board cameras with parallel data interfaces (that would also help in synchronizing the cameras better). But this is a very encouraging proof of concept, so I will definitely continue to work on it. The next step would be trying to build a mobile rig for walking around wearing the headset... I will try to do this using a Pandaboard or Odroid-X, which I have both available.

[Patrick Bauer]

I myself am a student, having a research topic about AR in HMDs (combined with a CAVE Environment).
What do you think about the new $20 camera for the Raspberry Pi? If my information is correct, on 640x480 it reaches 90! fps. I dont know how hard it would be to use this Camera (CSI connector) at a pc or what the lag would be from processing through the Pi. Has anyone some information on that camera?

[amnipar]

I have ordered the Pi camera, so I can comment when I see it. I guess it should be relatively fast, if you just pass the image through. But for stereo vision you would need two cameras, and Pi can only handle one. But if stereo imaging becomes popular I guess we will see developer boards similar to Pandaboard that can handle two cameras out of the box. My initial impression is, that even with web cameras it's possible to get tolerable results, but of course for any real applications the latency must be much smaller.

[Patrick Bauer]

Well, the board is so cheap now. I think its okay to get 2 Pi's with 2 cams. I don't know any other low-cost solution with 90fps + stereo under $100. Additionally I think with the computing power of the Pi we could be able to process some things (anti-fisheye distortion?) directly, without the need in our own program. How hard is it to replace the lens of a camera like the Pi's? And is it a good or a bad thing that it has a fixed focus?

[amnipar]

Fixed focus is a must, since the two cameras need to focus at the same distance. Of course, what would be really cool would be to track eye movements and focus where they eyes are looking. But that's for sure not feasible yet. Using two raspberry pi's would be problematic as you would need to communicate between the two devices to combine the images from those two cameras. Also I think it won't be easy to replace the Pi camera's optics, as the camera has some small mobile phone lens. But probably one could attach additional lenses in front of the original, there are wide-angle lens kits for smartphones. The most convenient option are webcams and board cameras that have the m12 mount, so you can fit different lenses and filters. But those usually aren't too fast. It seems the PS3 Eye camera is one of the best currently, it is quite fast at least.

[FingerFlinger]

What sensor is the Raspberry Pi camera using? VGA @ 90Hz is pretty exciting.

The PS3Eye and most others in the same general performance/cost range all use the same Omnivision chip, the OV7740, if I recall correctly.

[Patrick Bauer]

Its this one:
http://www.ovt.com/products/sensor.php?id=66

@amnipar
If can just take both video feeds and merge them in your pc software. With two USB Cams you would have to do the same.

Edit:
Just saw, at QVGA(320x240) 120fps are possible. But I think, 90fps is more then enough and we want to see at least some details

[FingerFlinger]

I wouldn't say more than enough, but it's certainly the right direction.

[Patrick Bauer]

In what way would >90fps or even >60fps provide benefits, while the oculus rift is stuck at 60fps? As long as the fps are stable, I dont think we need more fps right now. What do I forget?

[amnipar]

@Patrick Bauer, 60 fps is probably enough for Rift, but if the camera could provide more, and the computer could handle that speed, then higher frame rate would help in reducing latency. That's because in addition to capturing images they have to be transferred first to the computer and then to Rift. I believe getting 120 fps from the camera would reduce the total time from camera sensor to Rift display. But anyway the bottleneck will be data transfer from camera to computer, and all USB cameras will probably produce lots of latency. The Pi camera is connected to the board with a CSI connector, and that is a fast interface, especially if a kernel module is written that pushes the camera image straight to a display. But Pi has only one CSI connector. I have understood that in theory the Broadcom chip could support two CSI cameras, but in practice not, since the required data transfer speeds would be too high. Anyway the bottom line is, that the Pi camera can probably be used easily only with Pi (since special drivers are needed for each chip using the camera), and only one camera can be attached to one Pi. So for stereo you would need to communicate between two Pi's, which could be doable, but synchonizing the cameras is probably quite tricky. I hope one day soon we will have a beefier developer device that has full support for two CSI connectors and a fast enough memory bus. That should enable relatively low-latency see-through video for Rift, especially with 120 fps video streaming.

[Patrick Bauer]

@Amnipar
I didn't think of using the Pi directly on the rift. My idea was two Pi's with 90fps cameras which both will be connected with a pc. The pc calculates the merge and the rest of the AR content and streams it to the rift, just like a game. In this, the Pi's are passive and just provide a video feed (what would be best for that? USB? Ethernet? S-Video?). The additional idea was, that the pi itself could do some basic operations to the video feed, e.g. negate the barrel distortion of additional wide angle lenses etc. I don't think the Pi's need to be synchronized in that way as they are just fast camera feed provider

[amnipar]

So you want to create a kind of programmable camera peripheral. That is a nice idea, Pi has a decent GPU and could do some processing to the image. But it won't be very easy to transfer the image data to a PC fast enough. USB and Ethernet could be both usable, but you would need to create a kernel module that pushes the data forward; in user space it would probably be too slow. Also having a real time kernel could help. Both USB and Ethernet have their own sources of latency, so it would be difficult to get better speed compared to a dedicated USB webcam, I think. It might be possible to use the GPIO pins on Pi to create some kind of parallel data bus, but it would be a bit tricky to use that on pc.

[Patrick Bauer]

Yes, thats exactly what I meant. But to be honest I don't have much experience in this area. My initial idea is a "simple" streaming via ethernet. I think its gonna be pretty interesing for myself to experiment on these things

[mickman]

well its been a few moths since the last post here but hey thought I'd add this

There's a campaign on now... A faceplate mounted 3D Camera for the Oculus Rift

check here for more... http://www.indiegogo.com/projects/visio ... /x/4118329