Cyber's DIY Stereo Driver [Work Log]

[cybereality]

Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

[Nick3DvB]

Getting both stereo and head-tracking working properly together is the holy-grail,

and pretty fundamental to the HMD experience, so no pressure then...

Keep up the good work!

[ERP]

Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

I think you'll find that won't work for most games.
You'd have to intercept camera position in the game code, not in the GPU calls.
The issue is going to be that the game will have done a basic clip and discarded offscreen geometry based on it's own internal representation before it submits to the GPU. So significant camera motion will have large portions of the scene missing.
This is the reason NVidia allow you to adjust the Frustum slightly in the 3DVision advanced settings. For example certain areas in WOW exhibited missing geometry at the screen edges even with the very slight L/R camera motion required for 3D.

[cybereality]

@ERP: Yes, the more I work with it, the more I think I have going to have to inject into the games themselves. This is surely possible, but will require specialized code for each game. However, it will give me a greater deal of control than with intercepting DirectX. It also opens the door to other stuff I want to do, like haptics/force-feedback which will require specific game knowledge.

[Emerson]

I *really* want to try Mirror's Edge on a Rift... so, last night I started hacking up a prototype of a driver similar to this, just for Mirror's Edge.

It does 3D for now--I'm going to start seeing if I can inject a displacement mapping shader soon to correct for the optics. I'm also thinking it might be possible to hack in roll-axis view support, since that shouldn't require seeing things that have gotten clipped out.

Check it out, the glitchiness around the two minute mark is from me messing with the convergence/separation to try to ham up the 3D-ish-ness.
[youtube]http://www.youtube.com/watch?v=-f8lLnjDK6k[/youtube]

[brantlew]

@Emerson: Woah, that kicks ass! Is that your code? If you get the warping transform working, please share the binary (or source code ) with us. Mirrors Edge would be psychotic on the Rift.

[Emerson]

The only thing I haven't coded so far, that I would change before I release anything, is that I'm leaning on a pretty common off-the-shelf game cheating kit to create the intercept DLL right now. I don't think that's a problem for Mirror's Edge per se, but I'm worried someone would try dropping it into another game and trigger an anti-cheat ban off of the fingerprints. Something else for the to-do list, I suppose.

[brantlew]

Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

[cybereality]

Hey Emerson, that looks awesome. Mirror's Edge is also one of my favorite games of all time.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

[Emerson]

Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

It's Azorbix's "D3D Starter Kit" from gamedeception, but I'm just using it as a proxy dll for now, not a full injection.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

There's a couple cheap tricks to make it work--first off, since I can't actually reproduce the entire render calls for the second viewport, I'm actually just switching off on frames between left and right, and delaying the backbuffer swap until both have rendered (otherwise you get pretty nasty flashes). So, technically, the left and right are slightly temporally shifted, but it still works for the most part.

Secondly, to actually create the 3D effect off the alternating views, I intercept the WorldViewProjection matrix on its way through SetVertexShaderConstantF. I multiply it by an inverted projection matrix I math'd out, then a translation matrix for the eyeball offset, then another projection matrix based on the first that does the assymmetric view frustums.

Lots of trial and error to get it right, so far.

[cybereality]

@Emerson: Ok, thanks man. I have actually gone down a similar road with hooking SetVertexShaderConstantF. But I am having some trouble determining which is the world/view/projection matrix. Currently I am just altering every 4x4 matrix I find but I feel like this has strange side-effects. Any hints on how you determine this?

[Emerson]

I did some testing with logging code that just spits out each matrix that goes through, along with its register--in Mirror's Edge's case the WVP is always in register 0. I also threw a check to see if the [4,1] element is non-zero to try to reject anything that hasn't been through a projection matrix already, but that might be superfluous.

[cybereality]

Good to know, thanks.

[Chriky]

This is great stuff guys. I love Mirror's Edge as well, especially the first few levels then it goes a bit off. I always find myself waiting on the menu for ages because the music is awesome.

[Emerson]

Good progress today! Got the basic optic correction distortion mapping done--though obviously I'm lacking the real data to do the correction for now.

That's it for my weekend hacking--I might try to clean up what I've got this week and get it on github, to see what other people make of it.

[WiredEarp]

Emerson, that looks wicked! I'm impressed. If that can be done for other games as well, then that would eliminate one of the main disadvantages of the Rift.

Are you able to perform basic scaling between different resolutions? IE, do you think it would be possible to have a game rendering to a virtual 720P screen (using 720P in the games settings) but your driver outputting to a different resolution (such as 640x480), scaling as it goes? That would be extremely handy for a very wide range of HMD's that only support a limited number of resolutions, and combined with your warping code would be a perfect solution.

[brantlew]

@emerson: Awesome work!

Carmack posted one in-game still on his Twitter account but that's the only thing I have seen that would allow for any kind of validation. Maybe Palmer or Carmack could supply us with a still image of a warped grid? That would allow us to work out the warp transforms in advance of the Rift release.

[Emerson]

Emerson, that looks wicked! I'm impressed. If that can be done for other games as well, then that would eliminate one of the main disadvantages of the Rift.
Are you able to perform basic scaling between different resolutions? IE, do you think it would be possible to have a game rendering to a virtual 720P screen (using 720P in the games settings) but your driver outputting to a different resolution (such as 640x480), scaling as it goes? That would be extremely handy for a very wide range of HMD's that only support a limited number of resolutions, and combined with your warping code would be a perfect solution.

I think it ought to be feasible--I might give it a shot when I get some spare time this week.

Carmack posted one in-game still on his Twitter account but that's the only thing I have seen that would allow for any kind of validation. Maybe Palmer or Carmack could supply us with a still image of a warped grid? That would allow us to work out the warp transforms in advance of the Rift release.

Yeah, that screenshot's what I've been trying to emulate with my quick-and-dirty photoshopped distortion map, but a grid would at least give a more accurate sense of the internal distortion, or give a basis to try and decipher a rough analytical model of the transform.

[Chriky]

Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

EDIT;

Judging by this screenshot, you look pretty good with the distortion; maybe you are not quite doing enough...

[Emerson]

Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

Good eye. The sides are flipped from what you'd use for an HMD, this works for "cross-eyed" stereoscopy, which is how I've been testing all this so far. Pretty simple to switch once I get the real deal. My eyes are getting pretty fatigued though.

[fader]

Hey guys, thought I'd post this as it may be of use.. from John Carmacks twitter.



Great work, I'm following keenly!

[mm0zct]

Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

Good eye. The sides are flipped from what you'd use for an HMD, this works for "cross-eyed" stereoscopy, which is how I've been testing all this so far. Pretty simple to switch once I get the real deal. My eyes are getting pretty fatigued though.

Thank god, I thought I was going nuts too, I have a naturally lazy eye/divergent eyes so parallel viewing is much more natural to me than cross-eyed. I assumed it would be set up as parallel for the HMD but seemed the wrong eyes.

It's interesting looking at Carmack's latest image (just posted above) on a non distorted system, it makes the flat wall look like a barrel popping out of the screen at you.

Mirrors Edge should be great with the Rift, it's already good to play on my zalman but the Rift (especially if the head tracking works) will be a whole different game.
Personally I'd love to play Dirt3 or a similar racing game though, sadly its cockpit view would benefit most from the parallax effect of shifting your head, rather than just tilting, requiring some additional head tracking.

[brantlew]

sorry cyber, I think your thread has been hijacked. maybe we should start a dedicated thread for Rift 3D Warp Correction Drivers or something

[cybereality]

This is sounding really cool. Looks like I have to step up my game now that there's some competition.

@brantlew: I don't mind the hijack, as this is all very related and helpful to the work I'm doing.

[Emerson]

sorry cyber, I think your thread has been hijacked. maybe we should start a dedicated thread for Rift 3D Warp Correction Drivers or something

Yeah... that's totally my bad. Sorry.

I can start a new thread for my stuff if needed, would it be better in here or the VR/R&D forum?

[Mel]

Wow! Great thread, Cyberreality and Emerson. I just ordered up a copy of Mirror's Edge in anticipation of the Rift and will spend time mastering it on my projection system.

Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

[Emerson]

Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

Honestly, this is my first real dive into DirectX myself--I'm usually more of an OpenGL guy. I've been figuring things out by looking at the DX device interfaces (in the aforementioned D3D Starter Kit from gamedeception), googling lots of MSDN docs, and watching my target game's DX calls with an API Monitor tool http://www.rohitab.com/apimonitor (bit like drinking from a firehose, though).

If you're also an OpenGL type of person, watch out for the matrices, they're effectively transposed between DX and OpenGL.

[Alkapwn]

Yeah, that screenshot's what I've been trying to emulate with my quick-and-dirty photoshopped distortion map, but a grid would at least give a more accurate sense of the internal distortion, or give a basis to try and decipher a rough analytical model of the transform.

Ask and ye shall receive. I can't really develop anything, but with this I can help. (sorry for the huge img, but though it best for you to have it full res.)
I'm assuming you can take a non-distorted half screen, run it through the distortion settings in Photoshop and then have something to shoot for/reference.



Matrix is blue lines every 20 pixels. Steps are on the bottom. That's from a full 640 x 800 red rectangle.

EDIT: thought I should clarify steps.
1. transform, right click warp.
2. in the warp drop down, select inflate (this will make the image much larger than intended final size)
3. apply warp, and then transform with pixel setting on to 800 height

[cybereality]

Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

To be honest I have just been learning DirectX recently mostly for this project. A good book on the subject is Frank Luna's Intro to 3D Programming. Make sure you know your stuff with C++ though first. If you want to learn C++ try C++ Primer Plus by Stephen Prata.

[Mel]

Thanks for the DX advice, CR and Emerson. I'll look into the book and do some reading online and try hooking that one DX call that's been mentioned somewhere in all the Rift related threads, and see what I can see.

[brantlew]

@Alkapwn: One thing I have noticed about that screen shot if you look closely is that the lines tend to converge more towards the right edge of the screen. I don't know if the warp function distorts asymmetrically from side to side or if that is just a bit of head roll causing lateral perspective. Carmack may have just took a quick snap-shot without trying to tilt the head into a perfect downward direction. Also, he mentioned that this was an early example and a bit under-warped. The video images from E3 look a bit more convex to me. Hey Palmer can you give us a little guidance on this? Is there lateral asymmetry in the lens distortion? Are there any hard numbers you can give us with from Nthusim that might help?

[Alkapwn]

@brantlew
Yah I was looking at that when I started to create the distorted one. I think it's because in that screenshot, he's not looking at the wall/floor fully square head-on. I'm hoping that Mr. Carmack will release that info for everyone.

[Emerson]

Okay, I managed to derive an analytical solution based on Alkapwn's grid--it's a lot closer than it was before (ME renders black bars on its UI so the seemingly more squashed look is not the fault of the proxy dll).



If anyone's curious: if your inputs are u and v on the [-1,1] range, then the outputs x=c*u/(v^2+c), y=c*v/(u^2+c), where c=-81/10, over the [-1,1] range. (Gotta scale it from and back to [0,1] in the shader, though)

[brantlew]

Excellent! It's even at the correct resolution. Hey Palmer, if you get a chance can you look at this image and see how close Emerson is getting to the proper warp correction?

[brantlew]

@Emerson: Hey I think you're doing a great job with this, but one thing I notice is aliasing where the convergence is greatest - particularly noticeable in the bottom text. This is how I imagined a post-process warp driver would look since it can only operate from the final rendered 640x800 image. What are the ways to combat this? I assume you could do some type of standard 2D pixel averaging from the 640x800 image. Could you also attack it further up the rendering pipleline? Can you define the projection plane as a curved surface? This would allow supersampling as well. Maybe with an injection driver you cannot gain this granularity of control? Just curious. Excuse my ignorance but I haven't done graphics in 15 years.

[Emerson]

@Emerson: Hey I think you're doing a great job with this, but one thing I notice is aliasing where the convergence is greatest - particularly noticeable in the bottom text. This is how I imagined a post-process warp driver would look since it can only operate from the final rendered 640x800 image. What are the ways to combat this?

I've been thinking about this--I think my first attack on the problem will be to keep it simple: I'll try to render the initial buffer prior to warping at 1280x1600 and effectively try to do an old-school FSAA approach to smooth things out a bit. If that works, great, otherwise I'll have to start digging into the more complicated algos.

[spyro]

@Emerson:

Looks great, but we've to find a solution for the heavy aliasing.

spyro

[Chriky]

I've been using a more hacky solution of rendering each eye out to a 1024x1024 texture, then using them to texture actual 3D distorted planes which are in front of the main camera.

So, two 'eye' cameras with 4:5 aspect ratio rendering to 1024x1024 textures, then one 'main' camera with 16:9 ratio just looking at some sphere segments textured with the 'eye' images.

[cybereality]

@Emerson: Your solution is looking better and better. Thanks a bunch for posting the warping equation, this will help me a lot.

@brantlew: To be honest, I'm not sure the AA is needed. I know at least with the older prototype Palmer sent me, there was a heavy diffusion layer on it and the image had a soft look to it. If the final Rift is anything like this, then it will be a little blurry anyway, so I don't think the aliasing will be a problem. Though you can still notice pixels somewhat, so I guess an AA pass could help fake a higher resolution.

[brantlew]

@cyber: Yeah just like all AA, it's not strictly necessary although on the Rift it's a little different because different parts of the screen will look more aliased than others. Of course the worst aliasing is towards the edge of your vision so less of a concern. If the only solution is to render at high resolution and down sample, I would rather not take the performance hit for it. But if there is a simple way that doesn't add too much overhead, it would be nice.