Introducing Vireio Perception (3D Driver for Oculus Rift)

[MaterialDefender]

Any chance of pants shitting VR in system shock 2?

Yes, with VorpX and the freely available System Shock II DX9 patch.

SSII support for Vireo Perception might be added in the future, of course, if someone does it.

[Ardheim]

Great

I wonder how many cases of PTSD will be reported because of the rift and arma 2 though...

[cybereality]

Speaking of giving any date... when will you be releasing Vireio Perception? And when can we start helping with the code?

For sure, before the end of the month. Maybe sooner, stay tuned.

[Dilip]

@Cyber
You must keep "Donate" option, this is your hardwork and should be appreciated by all those who benifited from it. at least who wish to acknowledge it in a way.

Another Tought:
Why don't you charge minimal amount may be 10 or 15 $ or even less if you feel support is for less titles and you don wan't full time backing it still it covers some of best VR titles like MIRROR'S EDGE and its worth it.

Collectively it will sum up good if not make you super rich but i strongly feel hardwork should be paid.

[Ardheim]

I'd recommend a min donation of $5, since more people are inclined to donate that way.

[Mart]

I would absolutely donate Cyber; you deserve compensation. Don't undervalue your time!

Additionally it would be great to have a "vote with your money" facility to drive game compatibility. People would select (or suggest) a game and then make a donation/pledge. Once a developer has added support for a game, they receive the donations tied to that game.

[Fredz]

Additionally it would be great to have a "vote with your money" facility to drive game compatibility. People would select (or suggest) a game and then make a donation/pledge. Once a developer has added support for a game, they receive the donations tied to that game.

What about offering games to Cyber using the gift option on Steam ?

[Likay]

Donate to support certain games needs to have more levels i think. Usually when making a "basic" wrapper with a game there will be a certain amount of anomalies. I can imagine that different developers will make wrappers of different qualities. Timespan also matters since one person could release compability for a game with a lot of anomalies which later is fixed by another person. Some games today are often updated which could break compability and constantly needs maintenance. I think it's all doable though, there are just some extra things to keep in mind to be fair to the contributors.

@Cyber: For peoples ability to sleep and keep peace in their minds you should definitely add a way to donate. It doesn't necessarily have to be connected to the above mentioned donationsystem.

[yautjacetanu]

Can't remember if I said this already but I'd definitely still be happy to pay!

[Libertine]

Is there any word on whether or not you will add support for Metro 2033/Metro: Last Light? It would be the only reason I would buy the Oculus Rift or the driver support.

This is my kind of guy right here . Metro 2033 is amazing and would be unreal in 3D. I just wish they'd slow down the walking speed, its basically a run.

[voliale]

In particular, it will pre-warp the image to match the Oculus Rift optics, handle custom aspect-ratios (needed for the Rift's strange 8:10 screen), and utilize full 3DOF head-tracking.

Will we be able to turn off the 3D option and utilize just these features? I wanted to be able to use your driver with this upcoming device in the future:

http://www.fundable.com/3-dvision

Hi 3Tree, interesting project. I watched the videos and this guy seems like a true 3D enthusiast. However I've looked into (supervised) 2D to 3D conversion before and I'm skeptical.

The problem is, even though a 2D image contains a lot of depth cues (parallax, scale, shapes, focus, color saturation, brightness, etc), and even if the computer should get a perfect 3D solution (almost impossible), it still has to fill in a lot of information that is simply not present in the 2D image. And to do all that real time... (movies would be easier as they can be processed and buffered in advance)

Check out the two videos here, they explain the process well: http://www.yuvsoft.com/products/2d-to-3d-conversion/

My guess it that he uses these same techniques and just throws a lot of processing power at it, look at the size of that hardware. The results might be good, perhaps even surprisingly good. However he mentions that it doesn't have to be perfect because our brain corrects for errors. But with the Rift's massive field of view I'm not so sure our brains would be that forgiving.

And with his credentials and proven record, if he really has come up with a revolutionary device, why did he have to turn to a fundraiser webpage?

[3Tree]

In particular, it will pre-warp the image to match the Oculus Rift optics, handle custom aspect-ratios (needed for the Rift's strange 8:10 screen), and utilize full 3DOF head-tracking.

Will we be able to turn off the 3D option and utilize just these features? I wanted to be able to use your driver with this upcoming device in the future:

http://www.fundable.com/3-dvision

Hi 3Tree, interesting project. I watched the videos and this guy seems like a true 3D enthusiast. However I've looked into (supervised) 2D to 3D conversion before and I'm skeptical.

The problem is, even though a 2D image contains a lot of depth cues (parallax, scale, shapes, focus, color saturation, brightness, etc), and even if the computer should get a perfect 3D solution (almost impossible), it still has to fill in a lot of information that is simply not present in the 2D image. And to do all that real time... (movies would be easier as they can be processed and buffered in advance)

Check out the two videos here, they explain the process well: http://www.yuvsoft.com/products/2d-to-3d-conversion/

My guess is that he uses these same techniques and just throws a lot of processing power at it, look at the size of that hardware. The results might be good, perhaps even surprisingly good. However he mentions that it doesn't have to be perfect because our brain corrects for errors. But with the Rift's massive field of view I'm not so sure our brains would be that forgiving.

And with his credentials and proven record, if he really has come up with a revolutionary device, why did he have to turn to a fundraiser webpage?

Thanks for the link; I'm checking it out right now.

I would love to test that for myself if I could, but let's hope it will work!

He mentions that he uses some concepts from holography to help with the conversion. He also stated that he was using the fundraiser to start the initial funding and as part of a contest. Although he didn't choose a winner (I'm thinking the reason why has to do with Hurricane Sandy since he mentions that their office took a lot of damage from the storm) and claims they are working with investors to get the funding for the rest of the engineering on a smaller product as well as advertizing ..etc and that they will implement the technology into TVs soon.

These two videos will likely explain his methods better than I can:

1st http://youtu.be/tGKBihSoleg

2nd http://youtu.be/AlWckIQNncs

[Fredz]

The problem is, even though a 2D image contains a lot of depth cues (parallax, scale, shapes, focus, color saturation, brightness, etc), and even if the computer should get a perfect 3D solution (almost impossible), it still has to fill in a lot of information that is simply not present in the 2D image.

From a 2D image only it's most probably impossible, but from a succession of 2D images it's somewhat feasible, but not in realtime though.

Thanks to epipolar geometry there is a lot of 3D information that can be infered from 2 or more images of the same scene viewed at different angles. This can provide a mathematically correct 3D representation of the scene, much better than the luminance-infered depth or manual reconstructions used in most 3D conversions, which are mostly hacks.

Have a look at this video, it's the best example I know of automatic 3D reconstruction from a 2D video :

[youtube]http://www.youtube.com/watch?v=tyPtNlGnQqk[/youtube]

[MaterialDefender]

That looks quite impressive for a reconstruction based on nothing than color information at first. it's clearly the most simple case possible though. A lateral camera move around a fixed point of view in a scene without any foreground objects. That has basically a depth matte baked in over time. With some motion estimation and a little bit of advanced guesswork it shouldn't be too hard to extract it. Not many lucky cases like this exist unfortunately when it comes to real footage.

Looks still nice, of course. No doubt about that.

[voliale]

The problem is, even though a 2D image contains a lot of depth cues (parallax, scale, shapes, focus, color saturation, brightness, etc), and even if the computer should get a perfect 3D solution (almost impossible), it still has to fill in a lot of information that is simply not present in the 2D image.

From a 2D image only it's most probably impossible, but from a succession of 2D images it's somewhat feasible, but not in realtime though.

Thanks to epipolar geometry there is a lot of 3D information that can be infered from 2 or more images of the same scene viewed at different angles. This can provide a mathematically correct 3D representation of the scene, much better than the luminance-infered depth or manual reconstructions used in most 3D conversions, which are mostly hacks.

Have a look at this video, it's the best example I know of automatic 3D reconstruction from a 2D video :

[youtube]http://www.youtube.com/watch?v=tyPtNlGnQqk[/youtube]

Not too shabby. It helps though that the footage is a perfect candidate for conversion.

The yuvsoft software I linked actually uses this same technique, along with several others. Still takes a lot of work to clean it up. Also, background reconstruction.

[voliale]

He mentions that he uses some concepts from holography to help with the conversion.

I must have missed that part, do you remember in which video (and at what timecode, as the videos are very long) he said that?

[Fredz]

it's clearly the most simple case possible though. A lateral camera move around a fixed point of view in a scene without any foreground objects. That has basically a depth matte baked in over time.

Yeah, this scene is quite perfect for this kind of algorithm, but it's still a lot more complicated than what it looks like, even if it's not really evident from this video. They use several state of the art techniques in computer vision and they are able to compute the 3D parameters of the camera for each frame to enforce the epipolar constraint, so it's not a simple depthmap extraction in 2D from panning images.

Also it should work correctly with freely moving camera, even if it's limited to scenes with not too much moving parts, which is still a big limitation. But at its basis it's a feature correspondence problem, so it's completely independent of the scene structure and foreground objects shouldn't be a problem at all (they show an example with a tree in the middle of the screen in their paper).

They recently added this dense depth recovery technique to their automatic camera tracking project (ACTS), which was already able to compute 3D reconstructions, so it may be even better now. Unfortunately I didn't find videos showing that, but the software is freely available so I may try it eventually.

The yuvsoft software I linked actually uses this same technique, along with several others. Still takes a lot of work to clean it up. Also, background reconstruction.

They use a depth from motion implementation as part of the process, but I don't think it's anywhere near the complexity of the algorithms used by this technique. In their case it looks a lot more like a simple depth estimation from a panning camera without geometrical knowledge about the scene.

In their video they even don't show it on a filmed scene but on a 3D rendered scene, with only two single shots of blurry depth maps, which makes me seriously doubt they have anything near the sophistication of what I posted.

And by judging the different 3D conversions I've seen, it seems 3D conversion companies are still very far from what is done in the computer vision research field.

[MaterialDefender]

That sounds interesting, would be great to see results with more complex scenarios.

By the way: 3d reconstruction in post can be near perfect. Not automatically though. The process involves hand painted depth mattes, a lot of time consuming rotoscoping and deep pockets to pay for that. I recently saw a presentation by a rotoscoping artist demonstrating this. About three hours of extremely tedious manual work for a shot a few seconds long. The result was pixel perfect, but the amount of time and money this consumes is hard to swallow even for mid-size productions.

[cybereality]

Even manual conversion is not great (though better than most commercial automatic processes). Just look at the hollywood movies that spent millions on the conversion and the results are sometimes very poor.

[3Tree]

He mentions that he uses some concepts from holography to help with the conversion.

I must have missed that part, do you remember in which video (and at what timecode, as the videos are very long) he said that?

I was hoping you would find it since they gave a lot of information lol.

Here we go: http://youtu.be/AlWckIQNncs?t=27m30s

The whole video is a great watch though and he talks about the rotoscopy method and the lenticular method, what's good/bad about them..etc.

Also, here are a few quotes from one of the articles:

Why did you choose crowdfunding?

We need funding to be able to mass-produce and market and sell the product. The conventional route, which I’ve done before, is you go to investors and you make a presentation to them, you put together a business plan, you raise some private money, and eventually you go public and sell the stock on the stock market. I’ve done all that before.

And if you don’t raise the money do you think you’ll go the traditional route?

Well, I am still going the traditional route already. We already have our business plan… We started in December. We’ve been looking around, talking to potential investors and that’s how I found the Fundable guys. They have an investment membership organization called the Go Big Network and I’m part of that. The idea there is to be introduced to their investors. They tell me they have 20,000 investor members of that company.

I am going that route and I will continue doing that unless I don’t need to."

http://www.killerstartups.com/bootstrap ... vel-again/


And a bit more (although vague) info on the tech:

Can you talk a bit about your technology, and how it's different from other, 3D technology out there?

Gene Dolgoff: There are basically three kinds of 3D converters, besides ours. One is the manual technique, which uses rotoscoping, which requires a graphic artist to sit at a workstation, and convert frame-by-frame into 3D. To convert a single, full length movie, that takes hundreds of graphic artists, four months, and five to 15 million dollars. That's only good for blockbuster movies, and that's prohibitive for television. There also automatic converters. There are two kinds besides ours. One is really fake, and is the "simulated 3D" that most converters use, which essentially offsets different lines on the screen different amounts, which makes a picture look like it has different depth. It doesn't have any correlation to the real depth in the scene. The second kind uses something which are called depth maps, which are algorithms that assign different properties to different depths, for example, you might look at the brightness of an image to try to determine depth, which has a little more do to with actual depth and is more accurate. The problem there, is it requires lots more computing overhead, and requires a much bigger, more expensive machine. That usually causes lag, and that's prohibitive when you're playing a game.

Then, there is our system. Our system uses a different technique, based on the study of the human brain over the past 40 years. We look at two frames at a time, and look at all the 3D factors. You'll notice that most of the time, more depth means a scene has differences in brightness, contrast, and color saturation, and are higher in the frame. Plus, when a camera moves back and forth, the slower and object is, the further back it is. All of these factors are taken into account, and we get an image which is stereoscopically aligned with the algorithm in our brain. The workload of our computing is greatly reduced, and lots of this is done in the human brain. The actual depth information is detected and placed in a lot of the areas of the scene. It's not the actual depth, but when the brain sees actual depth in the areas, it fills in the rest of the missing depth information in the area, by remembering its previous 3D experience. That's how this is able to create good, 3D with accuracy, yet with low compute overhead.

http://www.socaltech.com/gene_dolgoff_s ... 44318.html

[MaterialDefender]

@cyber: I have to admit that I only saw one such movie, as far as I know: Alice in Wonderland, which was produced with the 3d conversion in mind right from the start and was OK for my taste. But I can easily imagine that there are far worse conversions.

[Mel]

Just a friendly thought: would discussion of 2d to 3d video conversion be better off being moved to its own thread?

[3Tree]

I'll go make a thread about the device, if someone hasn't already. And sorry to cyber if my post about the device pressured you into going open source with your project or something.

Alright here is the link for the other thread. http://www.mtbs3d.com/phpbb/viewtopic.php?f=112&t=16126

[Namielus]

I highly doubt your post had anything at all to do with his decisions.

[voliale]

They use a depth from motion implementation as part of the process, but I don't think it's anywhere near the complexity of the algorithms used by this technique. In their case it looks a lot more like a simple depth estimation from a panning camera without geometrical knowledge about the scene.

In their video they even don't show it on a filmed scene but on a 3D rendered scene, with only two single shots of blurry depth maps, which makes me seriously doubt they have anything near the sophistication of what I posted.

And by judging the different 3D conversions I've seen, it seems 3D conversion companies are still very far from what is done in the computer vision research field.

Ok, I had a look at the papers in your link and must admit that you were right and I was wrong.

First, I assumed yuvsoft did 3D tracking, but apparently they don't. And ACTS not only does it, but has developed a really impressive technique. (They still need to correct for moving/deforming/reflective objects though, and this method probably couldn't be used live or in video games as they need to compare images all over the clip.)

I liked the video where they compare their tracking solution to boujou, quite impressive results.

Think I'll need to check it out.

[donkaradiablo]

Doom 3, Hawken and in due time, Doom 4 of course, will be great with their in-game support for the Rift.

Mirror's Edge and Source engined games (like Portal 2 and Half-Life 2, and Black Mesa) are also on my must try with the Oculus Rift list... So I'm grateful for this Vireio Perception project.

and

This may sound awkward coming from a 33 years old man but... I am a Fallout 3 "virgin". And a Battlefield 3 "virgin". I still haven't played them and I'm still waiting for the Oculus Rift dev kit to experience those two games (in singleplayer) for the first time. I really hope someone integrates support for them

[Herod]

just saw this driver for the first time, looks great and i want to use it in the future with the rift, but also now on my 3d tv in sbs mode. my 50 dollars is waiting for you, keep up the good work

[shiva]

This is awesome, can't wait to try it out
Btw, am I the only one thinking DEAD SPACE ?
Playing that game on the Rift would be some serious pants shitting experience

Anyway thanks alot for your hard work, but I do agree with everyone, you should definitly set up a donation page, it's not everyday that people work on truly revolutionary gaming software

[VFRHawk]

How about the Eagle Dynamics Flight Sims, DCS A10 & Black Shark, they sound like pretty natural candidates for this. And Evochron Mercenary too..

And I'd definitely be prepared to donate to support development.

Thanks for all your work.

VFRHawk

[2EyeGuy]

Those would all be good with TrackIR.

[vrvision]

When are you going to release the driver , and does the driver work with DIY oculus rift ?

[Likay]

There is several information through this thread.
The main purpose for the driver is to make a 3d-wrapper specifically and specially for the Rift. Since Cyber decided to make it opensource it means that he is about to release the code free for others to explore and develop (he states at the end of this month but give him some room). I assume that he will probably be around with the coding himself in the future but without the burden of being held liable of the driverfunction. It's free for entusiast to further develop/maintaining and he provides a good starting point for them.

[VFRHawk]

Surely anything that's good with TrackIR's likely to be a good fit with a Rift?

Euro Truck Sim 2 would be pretty damn good too.

VFRHawk

[German]

Surely anything that's good with TrackIR's likely to be a good fit with a Rift?

It depends how much latency the TrackIR has, that's the big thing about making it all work in a way that feels realistic versus feeling drunk.

[cybereality]

I should be making a release next week. Just fixing up a few last things, but it seems to be working good now.

The big thing I fixed was the strange distortion when using the roll in Source engine games. Now the roll works like you would expect.

The thing I am still working on is adjustments for the head-tracking. Right now it only works when you have the tracker in one orientation. But I imagine people might mount it in different ways, so I wanted to have some adjustments. Honestly if its going to take too long I will just release it as is, since this is only really an issue with DIY trackers (won't be a problem with the Rift since the tracker is built-in). But if I can fix it quick, then I will try to do so.

[Mel]

Surely anything that's good with TrackIR's likely to be a good fit with a Rift?

It depends how much latency the TrackIR has, that's the big thing about making it all work in a way that feels realistic versus feeling drunk.

I did/do not have a great experience with my TrackIR 5 pro. I had it mounted on the side of my DIY Rift in such a way as to make the 'dots' easily visible to the sensor across a wide FOV, but it was still quite temperamental. I spent more time worrying about how to move my head so as not upset the TrackIR than paying attention to the game itself, so lag was the least of the worries. Definite immersion-breaker.

It was OK in MS FS-X, but for the FPSes I tried, not so much.

[VFRHawk]

Yes, I had a similar experience with FaceTrackNoIR, great idea, but it seemed to need me to sit unnaturally still and the restricted distance I could turn my head before I couldn't see the screen limited it's utility. Why I have such high hopes for the Rift, especially if support for "non native" games can be made realistic.

[Sinsinful]

This seems like some amazing work your doing cybereality and has actually pushed me to buy a Oculus Rift developer kit purely at the thought someone will be able to make Skyrim work in VR.

I know your most likely busy but I have a few questions if your taking a break from coding and passing the forum maybe you could help enlighten me

1. Did I pick this up right from the previous post: your designing these drivers for Oculus rift - without a developers headset?

2. If 1 is true how do you test them to make sure they work?

3. How do you get round the HUD elements in the likes of Skyrim and the text in the menus? for example will the likes of the health and stamina bars - will they bend with the screen or distort in any way? Will the text in the menus distort?

4. Will the drivers lead the game to require extra processing power from the cpu or gpu?

5. When you release the drivers next week will they be downloaded from your site?

6. I have no coding knowledge in VR but I'm hopefully going to be able to use tutorials to get some use out of my dev kit. Will your drivers be hard to use in conjunction with the rift for non-coders?

Keep up the great work! Cant wait to test your drivers with my dev kit in a few months

[cybereality]

Hey, Sinsinful!

1) Thats right, I don't have a Rift yet. I am just basing my work on the specs of the hardware.

2) I think it should work, but obviously I will have to test it out when I get my hands on a Rift.

3) I am not doing anything special with the HUD elements. They may be difficult to read if they are on the edge of the screen. However they are being pre-warped like everything else, so distortion should not be a big problem.

4) You will lose about 50% of the performance from running in 2D. So you will want to make sure you can get 120FPS in 2D for this driver to work good in 3D.

5) Yes, the driver will be available for download next week.

6) The software is pretty easy to use, but on some games you may need to load the game a certain way (this is explained in the read me).

[Sinsinful]

Great Stuff Cyber.
Cant wait for my Dev kit to get testing!!!
Fun times ahead this year experimenting with this technology