Cyber's DIY Stereo Driver [Work Log]

[brantlew]

Interesting. Keep us posted.

[ceashure]

Is there a possibility that the DX depth buffer uses only the alpha channel, and the color channels are to remain black?

[Chriky]

A lot of post process shaders use a full screen quad - that's a flat board fitted exactly in front of the camera - which might explain why you get solid black on your depth map.

[cybereality]

So I have attempted to get a dual render stereo working (well right now just moving the camera left and right). I naively though this would be as simple as intercepting "SetTransform" and altering the projection matrix. While this method would have worked with the fixed-function pipeline, its seems no modern games utilize this. Someone on the forum here (forgot who sorry) mentioned this before. What I assume developers are doing is just passing in their own matrix into a vertex shader and doing the computations there. So to get stereo working I would have to intercept this matrix somehow before it gets into the shader. I think its possible, but just a bit harder than I imagined. This also means it probably won't be a generic solution that will work for every game, as each game might be doing something slightly different. Still have to do more research on this aspect, but hopefully it won't be too difficult.

[ERP]

Yes I mentioned it.

I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.
The projection matrix will have none trivial values in the 4th row or column, you have the added complexity of it being possible to write shaders that treat matrices as either row or column major, there is a convention (can't remember which one it is in the shader) but there is no enforcement of it. I'd still expect most games to follow it.

[cybereality]

@ERP: Ok, thanks for the advice.

[phil]

I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.

Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil

[ERP]

Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil

That makes a lot of sense, explains a lot of the common artifacts, anything view dependent computed in the vertex shader will be wrong.
It's likely how the Iz3d drivers work as well, they likely append instructions to the end of the shader.

I'm not sure there is a better way to do it in the world of programable shaders.

Means there is likely no perfect automatic solution for any game.

[brantlew]

We should lobby the major engines like Unreal to include proper stereo support out of the box. Maybe then game developers would just leave it enabled and we would get more game support automatically.

[cybereality]

Well Unreal Engine already has some 3D support (via RealD and TriOViz) and so does CryEngine and probably some others. The issue is that game developers are not utilizing them.

Anyway, even if the major engines supported 3D, it would likely only be the very common stuff (ie HDMI 1.4a, 3D Vision, etc.). Certainly I doubt they would support strange DIY setups (the DOOM 3 thing is the exception). Plus, there is a bunch of other stuff I want to do with hooking DirectX (like full 6DOF headtracking, force-feedback, etc.), stereo 3D is just the first step.

[cybereality]

So I began with intercepting "SetVertexShaderConstantF" and trying to find the right matrix. Right now I am looking at arrays of floats with 16 or greater elements (which I am assuming are matrices). I am able to print the results on screen for testing.

So far in Left4Dead I am getting something, looks like maybe a translation matrix. It stays fairly steady when I am still, but the numbers change when I move or rotate the camera. Certainly looks like something important, will have to investigate further to see how useful this will prove. But I feel like I am on the right direction.

L4D_Matrix.png

[cybereality]

So I started trying to alter the matrices/vectors I'm looking at. Barely working, but I have some control. Just got to find the right matrix.

[youtube]http://www.youtube.com/watch?v=j_4N-Qb_QLA[/youtube]

[brantlew]

This is a cool exercise in reverse engineering. You should consider starting a technical blog - not necessarily giving away the source code but showing the basic hooking mechanisms and functions calls so others can create similar projects. This type of info is often difficult to come by - especially aggregated in a single place.

[Chriky]

You are probably intercepting the matrix that is used to render just that object. Try adding the same vector to the last column of every matrix you come across.

[cybereality]

So I have made some serious progress in just a few days. Managed to hack together camera movement in L4D. Currently it's the only game that's working, since I'm hard-coding the values to modify. Basically looked for matrices with exactly 16 elements, and then did trial and error altering each element in the array. Will have to come up with some generic way to handle this, although I guess creating a profile for each game wouldn't be too hard.

[youtube]http://www.youtube.com/watch?v=JKjXaHP-M-U[/youtube]

Next up I will try getting multiple viewports working for side-by-side 3D. Stay tuned.

[brantlew]

Nice!

[dfiqujkh]

I'm working on something similar for my HMD based on foisi's concept. So far I have head position injection and split image output. I only did TrackIR enabled games so far. These had an additional transformation matrices somewhere in memory. I found these either by looking through disassembly or searching memory with CheatEngine.

Here's a screenshot from F1 2011. It might look stereoscopic at first glance but it's actually just parts of the same image duplicated and aligned for the left/right eyes. It's done by hooking Present, grabbing the backbuffer and using it as a texture for two quads. http://i.imgur.com/qZAea.jpg

I don't plan to do stereo images, I'm not too fond of halving my framerate and fixing all the small issues coming from shoehorning 3D rendering into games not meant for it. Great to see someone up for the challenge, though.

[cybereality]

Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

[dfiqujkh]

After the game is done rendering the frame, there might be all kinds of Direct3D state still set that you don't expect. Try turning off alphablending and shaders before drawing your quad. Here's some stuff I had to do:

Code: Select all

device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_SELECTARG1);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_CONSTANT);
device->SetTextureStageState(0, D3DTSS_CONSTANT, 0xffffffff);
device->SetRenderState(D3DRS_ALPHABLENDENABLE, FALSE);

device->SetVertexShader(NULL);
device->SetPixelShader(NULL);

[ERP]

Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

You'll have to explicitly set any renderstate, including none obvious things like render target, scissor regions and destination blend modes.
There is no telling what state the "state machine" is in when you get called.

[cybereality]

Thanks. That's good to know.

[Nick3DvB]

Just wanted to say well done on the progress so far, excellent work!

All way above my head but it looks like you've got it all under control!

I didn't realize you could control XYZ plane camera movement with an intercept driver,

I assumed it would need to be coded into the game engine, shows what I know...

But it begs the question - why the hell hasn't anyone done this before !!!

[brantlew]

@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work. Thus this thread..

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15027

[Alkapwn]

John Carmack was TOTALLY right! the big companies don't even get how amazing this truly can be! Getting it into the hands of the enthusiast crowd is going to have the development skyrocket by leaps and bounds! The stuff you guys are doing is completely amazing! I think I somewhat understand the theory of what you guys are doing, even though I'm not much of a coder myself. Though I can sure offer some moral support!

So cyber, if you get the side by side output going, you'd pretty much just need to implement the "Rift Warp" into the output and you'd theoretically have another working demo to drop on the Kickstarter market. I wonder if Carmack would be willing to give you the spatial distance needed from left eye camera to right eye camera, since he's mapped it out for Doom already. If whatever he's figured out for Doom even applies to LFD.

[Nick3DvB]

@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work...

I didn't mean for stereo drivers, I mean't for head-tracking kit like the TrackIR etc

seizing controlling the "head" (camera) location in the game and mapping the players actual head location to it,

the position of the head in space, not just the direction the head's eyes are looking in. They really missed a trick there!

[cybereality]

Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

[Nick3DvB]

Getting both stereo and head-tracking working properly together is the holy-grail,

and pretty fundamental to the HMD experience, so no pressure then...

Keep up the good work!

[ERP]

Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

I think you'll find that won't work for most games.
You'd have to intercept camera position in the game code, not in the GPU calls.
The issue is going to be that the game will have done a basic clip and discarded offscreen geometry based on it's own internal representation before it submits to the GPU. So significant camera motion will have large portions of the scene missing.
This is the reason NVidia allow you to adjust the Frustum slightly in the 3DVision advanced settings. For example certain areas in WOW exhibited missing geometry at the screen edges even with the very slight L/R camera motion required for 3D.

[cybereality]

@ERP: Yes, the more I work with it, the more I think I have going to have to inject into the games themselves. This is surely possible, but will require specialized code for each game. However, it will give me a greater deal of control than with intercepting DirectX. It also opens the door to other stuff I want to do, like haptics/force-feedback which will require specific game knowledge.

[Emerson]

I *really* want to try Mirror's Edge on a Rift... so, last night I started hacking up a prototype of a driver similar to this, just for Mirror's Edge.

It does 3D for now--I'm going to start seeing if I can inject a displacement mapping shader soon to correct for the optics. I'm also thinking it might be possible to hack in roll-axis view support, since that shouldn't require seeing things that have gotten clipped out.

Check it out, the glitchiness around the two minute mark is from me messing with the convergence/separation to try to ham up the 3D-ish-ness.
[youtube]http://www.youtube.com/watch?v=-f8lLnjDK6k[/youtube]

[brantlew]

@Emerson: Woah, that kicks ass! Is that your code? If you get the warping transform working, please share the binary (or source code ) with us. Mirrors Edge would be psychotic on the Rift.

[Emerson]

The only thing I haven't coded so far, that I would change before I release anything, is that I'm leaning on a pretty common off-the-shelf game cheating kit to create the intercept DLL right now. I don't think that's a problem for Mirror's Edge per se, but I'm worried someone would try dropping it into another game and trigger an anti-cheat ban off of the fingerprints. Something else for the to-do list, I suppose.

[brantlew]

Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

[cybereality]

Hey Emerson, that looks awesome. Mirror's Edge is also one of my favorite games of all time.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

[Emerson]

Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

It's Azorbix's "D3D Starter Kit" from gamedeception, but I'm just using it as a proxy dll for now, not a full injection.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

There's a couple cheap tricks to make it work--first off, since I can't actually reproduce the entire render calls for the second viewport, I'm actually just switching off on frames between left and right, and delaying the backbuffer swap until both have rendered (otherwise you get pretty nasty flashes). So, technically, the left and right are slightly temporally shifted, but it still works for the most part.

Secondly, to actually create the 3D effect off the alternating views, I intercept the WorldViewProjection matrix on its way through SetVertexShaderConstantF. I multiply it by an inverted projection matrix I math'd out, then a translation matrix for the eyeball offset, then another projection matrix based on the first that does the assymmetric view frustums.

Lots of trial and error to get it right, so far.

[cybereality]

@Emerson: Ok, thanks man. I have actually gone down a similar road with hooking SetVertexShaderConstantF. But I am having some trouble determining which is the world/view/projection matrix. Currently I am just altering every 4x4 matrix I find but I feel like this has strange side-effects. Any hints on how you determine this?

[Emerson]

I did some testing with logging code that just spits out each matrix that goes through, along with its register--in Mirror's Edge's case the WVP is always in register 0. I also threw a check to see if the [4,1] element is non-zero to try to reject anything that hasn't been through a projection matrix already, but that might be superfluous.

[cybereality]

Good to know, thanks.

[Chriky]

This is great stuff guys. I love Mirror's Edge as well, especially the first few levels then it goes a bit off. I always find myself waiting on the menu for ages because the music is awesome.

[Emerson]

Good progress today! Got the basic optic correction distortion mapping done--though obviously I'm lacking the real data to do the correction for now.

That's it for my weekend hacking--I might try to clean up what I've got this week and get it on github, to see what other people make of it.