Stereoscopic Fisheye Rendering (WIP) Techdemo

[marbas]

A simple interactive techdemo that showcases stereoscopic fisheye rendering.

Resolution is hardcoded to 1280x800.

If this works as expected on the Rift remains to be seen in December. If someone can test this now, please let me know!

*EDIT 2: Demo updated with controls to fine tune the barrel distortion.
There is now lots of text overlayed on the graphics that show all the KB shortcuts.
Turn this text off with [F1] to increase FPS.

Executable



Note. This is not much optimized yet. Might run slow on older systems.

Controls:
[WASD+Mouse] Freelook/move around
[Space] Toggle normal/fisheye rendering
[F1] Toggle Text
[ESC] Quit

[IGameArt]

That's actually pretty spiffy. I'll test it out for you whenever i get my devkit. Right now it runs at 14 fps on this older computer, 61 with the effect toggled off. I plan on getting a good gaming laptop when i get paid this month. Really excited to see how this stuff pans out. And hopefully someone can test this out for you. Have you messaged Palmer about it at all?

[cybereality]

Cool.

[marbas]

Thanks guys!

@IGameArt
No, asking Palmer didn't cross my mind. I guess he's too busy at the moment to play around with techdemos;)

[IGameArt]

i get about 61 with regular rendering. It's a pretty big difference really. I"m sure it could easily be optimized though. What are your own frame rate experiences?

[marbas]

Yes there is room for improvement here. Both in speed and resolution for fisheye render. Right now it's a bit more blurry than it needs to be:)

On my system I get ~60fps with both render methods (vsync is hardcoded enabled)

My system specs: laptop,win7x64,dualcore 2.1Ghz,4gb ram,gfxcard nvidia 9600M

[cybereality]

Just running a warp pixel shader should not have really much of a performance hit on modern hardware. At least I have not noticed much difference in my experiments.

[marbas]

In this demo I'm dong a render-to-texture operation on quads with high tessellation. I'm sure the warp shader method is faster.

[IGameArt]

You could probably lower the tesselation level as well without too much distortion. That's the method i would have used had i not had the shader to do so.

[marbas]

Hmm lowering the polycount of the quads don't gain much performance as it amounts to only 750 polys. Actually I was planning on adding a few more

[Owen]

Your approach with the vertex distortion is actually a good optimization for mobile, because doing math on texture coords in a pixel shader counts as a dependent texture fetch, which is a significant pipeline stall for an SGX chipset. Its a good method to hang on to.

Sticking to a pixel shader will not hurt you on PC though, and with so few pixels to work with every bit of image quality helps.

[Fredz]

Nice initiative. You should try to add some kind of mipmapping because the lines are very blurry at far distances. Also textures and shadows are very strong depth cues, I think it would be more immersive if you added some. The skybox doesn't seem to be rendered at correct depth, when watching the screenshot cross-eyed it seems to be closer than the scene. You could try to render it at near infinity.

Also I could have swear I saw a junction problem in your skybox, but you may have corrected it already.

[Pyry]

Mipmapping is what causes those blurry lines; without it, you'd see aliasing artifacts rather than blurring. You want anisotropic filtering.

[marbas]

Thank you all for feedback and suggestions for this techdemo. The primary aim of this demo is to test that the fisheye rendering is working using an alternative technique to using a warp shader. I didn't focus to much on the content itself. The "vector" style graphics if for easier observing the distortion effect. The dimension of the fisheye viewports closely matches those shown in the screenshot posted by Mr.Carmack.

I will continue to improve this technique with better resolution and speed. When this is done I will look into improving the overall graphics quality and interactivity.

[Dycus]

Hey, my name is Chris Dycus. I'm actually on the Oculus team as one of the hardware developers.
Now, I realize I can't just say something like that without proof, but Palmer's at Gamescom in Germany right now, so he can't vouch for the validity of this account right now. I'll tell him to check out this thread, but I'm not sure how much free time he has (I was at Quakecon with him; we were so busy it was crazy). Worst case scenario, you'll have to wait a couple days for proof.

Anyway. I don't have any Rift prototypes with me; Palmer had to take them to Gamescom. But I do have a couple of the lenses, and a jig to hold them.
Does the demo need any special requirements? I get an unsupported graphics card error when trying to run it fullscreen and "Unable to create 3D scene" when trying to run in windowed. 'Course, my computer was good in 2003, so... I might have a couple more I could try it on, though.

But, I still have your screenshot. I only have my monitor and two lenses, but I can tell you right now that the warping isn't dramatic enough. Everything still curves in a bit, but not too much. Where's this screenshot from Carmack? I'd like to test that, too.

And we're not too busy to play with techdemos! When Palmer comes back, I'll have him try this. It would probably be better to try this out in an actual headset; I might be doing something wrong.

[marbas]

Hi @Chris and thanks for your interest in this techdemo.

I *think* the warp shape itself is ok. But fov/pr.eye, zoom amount and virtual eye separation is likely to be offset by some amount to match the HMD optical properties. I'm looking into implementing a set of onscreen options to fine tune these properties. Of-course this must be tested om a real HMD. Its weird writing programs for a non existent hardware device

I'm not sure why you should have problems running the demo on your system. May be some incompatibilities with your gfx card. Atleast windowed mode should work. Weird!

Here is the Carmack screenshot: https://p.twimg.com/AtHFjaLCAAAh81G.jpg:large

Glad to hear you guys at Oculus are interested and have time to check out our stuff
No worries about the waiting, Ill totally understand you are very busy right now.

[Dycus]

Hmm, everything still seems to be bent a bit with that screenshot, too. It's most likely just me.

I'll try this on a headset when Palmer gets back, and we'll give you some feedback.

And yeah, I don't know why the demo doesn't run. Like I said, crummy computer. I have a laptop and another desktop I can try it out on, though. I'm sure it'll run fine on Palmer's laptop.

[bobv5]

Dycus you dirty liar!!
Ha only joking. If you look at these links are some pictures from cyberrealitys stereo driver. Would be good if you could test them.

http://www.mtbs3d.com/phpBB/viewtopic.p ... &start=165
http://www.mtbs3d.com/phpBB/viewtopic.p ... &start=150

[Dycus]

Like I said, I don't have a Rift prototype right now. But, when Palmer gets back, I'll be sure to throw those images up on the head mount and see how they look. They look pretty nice so far!

[jis]

I suggest to remove the mouse smoothing. You are just adding more latency without gaining anything.
Maybe you can provide a factor to see how it affects the experience.

Also it seems that windows adds automatically some mouse smoothing when using the standard win32 messages (WM_MOUSEMOVE, DirectX, ...). I am not sure of this, I didn't check myself yet.
Maybe you should try later to use the low level input messages provided by win32: WM_INPUT with RAWMOUSE. This should bypass this issue.

[marbas]

I've added mouselook smoothing by choice, because I like smoothed camera pan I know not everybody like this, but this is just a simple demo to visualize the barrel distortion. If it would be a game or something I would probably add some options to tweak the sensitivity.

On an actual HMD device with camera rotation derived by head tracker data, it would of course make sense to have precise camera pan with no smoothing at all (other then filter out any jitter that may occur)

[jis]

Yeah, I was referring to the testing session on the Rift. Carmack and Palmer were doing whatever they could to reduce latency in order to avoid being disconnected from the world and raising nausea. In that sense, I could deduce that your demo is not Rift ready yet.

edit: Yeah I somehow thought you were going for a full demo. It might be cool to have an open source demo that addresses all the issues at once.
I will certainly try to find a smoothing algorithm that does not increase latency without leaving too many discontinuities when I'll have a hand on the rift.

[marbas]

Well of course it's not Rift ready yet. There will be lots of issues to tackle before you can call a piece of software 100% Rift compatible. I take one step at the time and wait for feedback about the distortion level. That is if the frames align well in the optics and all that.

[PalmerTech]

Dycus has a new batch of parts on the way to him, he should have a few prototypes very soon!

[marbas]

Hi Palmer!

Good news, we need your an Dycus eyes for debugging purposes

[zalo]

Thanks for making this demo, marbas! I haven't had the opportunity to try it on a computer, but the screenshot gives me a very strong 3D effect.

In the mean time, I'd appreciate it if you added keyboard shortcuts that adjusted parameters like distortion strength, interocular separation, and convergence point so we could use this as a 3D sandbox to determine the 3D settings that look best and are the most comfortable. Perhaps Dycus could adjust it himself and relay those numbers back to us so we have a good starting point for more 3D applications.

[marbas]

Thanks for your suggestions Zalo. Yes you can count on those options being implemented soon. The camera rotation I can't do much about until I get the dev kit & sdk. So for now there will be no support for headtracking. Just mouse emulation (unless oculus releases a dll for us to integrate into our demos)

[Dycus]

Like Palmer said, I should be able to put together some more prototypes relatively soon. Once you implement controls for all that stuff, I'd be happy to give it my best calibration. I have some family that I could ask to try it out, too.

[troffmo5]

@marbas do you want to keep the source closed or you plan to put it in some public repository (e.g. github)?
You could find people willing to port it to other platform.

[marbas]

Troffmo, I havent made up my mind yet regarding releasing the source. But I probably will As a set of beginner tutorials on my website.

[marbas]

Hi!
I've updated the demo with controls for fine tuning the barrel distortion. The link is on the first post on page 1.

@Dycus & Palmer
If you have a chance to try this out on the Rift, you may want someone to help you tweaking the values on the keyboard. I have a graphical UI in the works that does not require the keyboard. Which is a lot more hmd-user friendly I believe But for now it's all on the keyboard.

[marbas]

Just done implementing barrel distortion algorithm. That means its now possible to tweak the amount of distortion effect. This was not possible in previous versions of this demo (just transform operations like scale,position & rotation).

One question about convergence.
Does the cameras convergence play any significant role on the Rift? Ain't the eyes supposed to look parallel at inifinity anyway?

[Dycus]

Finally got another prototype built. The demo looks really nice! The warping actually isn't quite right; lines at the edges are still a bit bowed. Is it possible to implement a way to change the bend of the warping?

[marbas]

Thanks Dycus, glad you like the demo

I have uploaded a new version with the following changes:
[Keypad +,-] Distortion
[PgUp,PgDn] Tessellation (hit F11 to visualize this)

All these KB shortcuts are now printed on the screen to make things easier.
Toggle text off with [F1] for higher FPS.

Also I made some changes to increase the FPS (vsync=off and renderskip=on)

Next I will work on increasing graphics quality.

Thanks for testing! I hope the new distortion parameters works for tweaking a good value for the HMD.

[Dycus]

Well dang, now it's a bit too much for my poor laptop. XD When I look around the scene, one eye updates before the other and it feels terrible. It's dizzying. I'm gonna have to wait until I get a better computer on my hands to test this further, sorry.

[marbas]

Well hehe, sorry to have made you dizzy Dycus

You can still change the way it renders without the frame toggling mode (F12), which I agree updating both sides every frame is optimal. Hmm I didn't think that it would have such a terrible impact on the HMD. If I'm not mistaken this frame toggling technique have been used on previous HMD's (like the Vuzix vr920 that I have). Maybe Im doing it wrong?

The most interesting bit for me is the lens distortion. You don't really need to move around to test that (keypad + and -)

This dual render + prewarping stuff makes it hard to keep a high fps. Atleast with a render-to-texture approach.

[Dycus]

When I mean one view updates more slowly than the other, I mean it's slow. On my computer, that is. I'll move the camera, and one eye will change the view, then ~100ms or so later the other view updates. You could imagine how this would be undesirable. But like I said, it's just my slow computer.

Oh yeah, of course I can test the distortion without moving that camera. I'll get to that later, don't worry.

[Pyry]

Dycus, could you provide a warp map or something like that (for the current prototype; I realize it will probably change between now and when the dev kits are released)? I've been playing around with screen space ambient occlusion, and I kind of like the grainy look of the unfiltered output, but I wonder what that will look like in motion in stereo (because the noise pattern is view-dependent), and I'd like to render a test video for you to try.

[gravity360]

Doesn't the warp map differ based on the distance between the pupil and the optic? Or is it just optic to display panel?

[Dycus]

The warp map does indeed change when you move your eyes further from the optics. But that shouldn't be a problem, considering most will be wearing the headset while using it. I'd imagine developers might add a way to tune the warp a bit, too. Hopefully they do.

So, uh, I had to give the Rift I had to somebody else. I'll try to build another one soon. Sorry guys!