Meant to be Seen

So I know I have been talking about making a stereo driver for quite some time, and now I have finally gotten some initial work started on it (up until now it has been mostly research). My main motivation right now is to get something ready for Palmer's Oculus Rift HMD project. So initially it will probably only support that HMD, but in the future I plan to support all variety of devices. I also want to do some more advanced things that aren't available on current 3D drivers. For example, pre-warping for the optics on an HMD and things like that. Down the line I may want to integrate things like head-tracking or force-feedback, but that will probably be further out.

So yeah, lots of big plans so I have my work cut out for me. I still have to decided what to do with the project in terms of commercial options. Initially I want to have a free version, but I will have to figure out if I want to go commercial or even open-source in the future. It may end up being closed-source freeware, I don't know. Also, if IZ3D actually does go open-source, that might effect my decision. We shall see.

Anyway, just wanted to start a thread to use as my work log. So far I have just touched on the basics of hooking into DirectX9 and altering some of the functions. Currently I have just injected a red square and some text into a random game (Zombie Driver in this case). So far this is the only game I could get working, since all the other games I have are on Steam and the way Steam launches games messes up the method I'm using. But I'm sure I can work around it with more time.
I know it doesn't look like much, but this is a crucial first step to getting some control over DirectX. Will be updating this thread as I make more progress.

I applaud the effort and hope you can get something useful from it. Maybe the guys behind the Helix mod that improved 3d for games by altering shaders and whatnot can offer you some good advice:
http://helixmod.wikispot.org/

Cool!

Looks good CR!
One place where you might find some interesting code etc that might relate would be in the source of game hacks, such as aimbots and overlays/wallhacks.

WiredEarp wrote:One place where you might find some interesting code etc that might relate would be in the source of game hacks, such as aimbots and overlays/wallhacks.

Yes, some of my research has certainly gone into this area (which is probably the majority of what people would want to do with DirectX hooking).

Looks good. The "warping" driver is going to be an important part of the Rift project. Good luck.

Awesome! Super excited about this, having a 3D driver written with VR in mind would be amazing.

I think that charging for the driver would be totally acceptable, considering that all the other 3D drivers have a cost, too. At the very least, it would cover any flood of people wanting support (I have read several nightmare stories about people getting flooded with emails for freeware software they released, or even worse, pirated copies of paid software).

Have you considered using OpenGL in addition to DirectX? I know most gamers use Windows, but it's a pitty that Mac/Linux users are always forgotten...

davidgutierrezpalma wrote:Have you considered using OpenGL in addition to DirectX? I know most gamers use Windows, but it's a pitty that Mac/Linux users are always forgotten...

Yes, I have considered it, but its a very small market compared to all the DirectX games on Windows. However if this project gets off the ground and becomes successful I could revisit a cross-platform OpenGL solution.

Awesome project Cyber! If you do decide to open source it, I'd be more than happy to contribute to the project.

I was thinking this morning re the warping. Isn't this normally done on a pixel basis (i assume this is what warpalizer does)? So, would intercepting the DX calls really be a valid way to do this?
Are you planning to warp all the vertices, which may achieve the same effect...?

I was thinking that the warping would just be a full-screen post-process effect (pixel-shader). I don't think its necessary to warp the actual geometry (that would get kinda crazy).

Yep, that sounds like it would make much more sense

OT, but this is kind of stupid:

You cannot make another post so soon after your last.

I agree with the warping being a post process effect. It would actually be quite similar to the effects some games use to indicate some altered state, such as a boost in racing games, or some sort of enhanced focus in an FPS.

If you would be willing or interested to share some source I'd be very interested, either along the way or just at the end. Sadly I'm a bit busy just now or I'd be very interested in helping out on the development front.

One avenue that might be worth exploring is tapping into the stereo API's already used in some games, such as the new Deus Ex:HR and Dirt3, to persuade them to render in stereo and then display the results in whatever format the user actually requires.

(For example personally I find it quite annoying that the AMD HD3D, while listing the interlaced Zalmans as "supported", doesn't actually provide an interlaced output mode, only HDMI and DisplayPort frame packed modes)

So I have spent the last few days trying to get access to the depth buffer (z-buffer) in DirectX. The idea was to make a 2D+depth mode, as I figured it might be easy and have good compatibility. Apparently this is not something the API exposes normally (they didn't add this until Dx10). So I found a sort of hack to do this using INTZ textures. This seems to work with a simple test app I made, but doesn't work with commercial games. Below is my test app being hooked into to display the depth buffer.
With real games all I get is a black square. I guess maybe that games are doing something custom with the depth buffer and not using the automatic one provided by DirectX. However DDD seems to have success with their "Virtual 3D" (now "Power 3D") mode. So I know its possible somehow to get the depth from the scene. Will have to investigate this further at a later date.

Now I am thinking I will try to modify the view and projection matrices to start prepping for full dual render stereo. Like everything else, this probably won't be as easy as it seems, but hopefully not too difficult. Will keep you guys updated on the progress.

Interesting. Keep us posted.

Is there a possibility that the DX depth buffer uses only the alpha channel, and the color channels are to remain black?

A lot of post process shaders use a full screen quad - that's a flat board fitted exactly in front of the camera - which might explain why you get solid black on your depth map.

So I have attempted to get a dual render stereo working (well right now just moving the camera left and right). I naively though this would be as simple as intercepting "SetTransform" and altering the projection matrix. While this method would have worked with the fixed-function pipeline, its seems no modern games utilize this. Someone on the forum here (forgot who sorry) mentioned this before. What I assume developers are doing is just passing in their own matrix into a vertex shader and doing the computations there. So to get stereo working I would have to intercept this matrix somehow before it gets into the shader. I think its possible, but just a bit harder than I imagined. This also means it probably won't be a generic solution that will work for every game, as each game might be doing something slightly different. Still have to do more research on this aspect, but hopefully it won't be too difficult.

Yes I mentioned it.

I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.
The projection matrix will have none trivial values in the 4th row or column, you have the added complexity of it being possible to write shaders that treat matrices as either row or column major, there is a convention (can't remember which one it is in the shader) but there is no enforcement of it. I'd still expect most games to follow it.

@ERP: Ok, thanks for the advice.

ERP wrote:I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.

Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil

phil wrote:Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil

That makes a lot of sense, explains a lot of the common artifacts, anything view dependent computed in the vertex shader will be wrong.
It's likely how the Iz3d drivers work as well, they likely append instructions to the end of the shader.

I'm not sure there is a better way to do it in the world of programable shaders.

Means there is likely no perfect automatic solution for any game.

We should lobby the major engines like Unreal to include proper stereo support out of the box. Maybe then game developers would just leave it enabled and we would get more game support automatically.

Well Unreal Engine already has some 3D support (via RealD and TriOViz) and so does CryEngine and probably some others. The issue is that game developers are not utilizing them.

Anyway, even if the major engines supported 3D, it would likely only be the very common stuff (ie HDMI 1.4a, 3D Vision, etc.). Certainly I doubt they would support strange DIY setups (the DOOM 3 thing is the exception). Plus, there is a bunch of other stuff I want to do with hooking DirectX (like full 6DOF headtracking, force-feedback, etc.), stereo 3D is just the first step.

So I began with intercepting "SetVertexShaderConstantF" and trying to find the right matrix. Right now I am looking at arrays of floats with 16 or greater elements (which I am assuming are matrices). I am able to print the results on screen for testing.

So far in Left4Dead I am getting something, looks like maybe a translation matrix. It stays fairly steady when I am still, but the numbers change when I move or rotate the camera. Certainly looks like something important, will have to investigate further to see how useful this will prove. But I feel like I am on the right direction.

So I started trying to alter the matrices/vectors I'm looking at. Barely working, but I have some control. Just got to find the right matrix.

[youtube]http://www.youtube.com/watch?v=j_4N-Qb_QLA[/youtube]

This is a cool exercise in reverse engineering. You should consider starting a technical blog - not necessarily giving away the source code but showing the basic hooking mechanisms and functions calls so others can create similar projects. This type of info is often difficult to come by - especially aggregated in a single place.

You are probably intercepting the matrix that is used to render just that object. Try adding the same vector to the last column of every matrix you come across.

So I have made some serious progress in just a few days. Managed to hack together camera movement in L4D. Currently it's the only game that's working, since I'm hard-coding the values to modify. Basically looked for matrices with exactly 16 elements, and then did trial and error altering each element in the array. Will have to come up with some generic way to handle this, although I guess creating a profile for each game wouldn't be too hard.

[youtube]http://www.youtube.com/watch?v=JKjXaHP-M-U[/youtube]

Next up I will try getting multiple viewports working for side-by-side 3D. Stay tuned.

Nice!

I'm working on something similar for my HMD based on foisi's concept. So far I have head position injection and split image output. I only did TrackIR enabled games so far. These had an additional transformation matrices somewhere in memory. I found these either by looking through disassembly or searching memory with CheatEngine.

Here's a screenshot from F1 2011. It might look stereoscopic at first glance but it's actually just parts of the same image duplicated and aligned for the left/right eyes. It's done by hooking Present, grabbing the backbuffer and using it as a texture for two quads. http://i.imgur.com/qZAea.jpg

I don't plan to do stereo images, I'm not too fond of halving my framerate and fixing all the small issues coming from shoehorning 3D rendering into games not meant for it. Great to see someone up for the challenge, though.

Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

After the game is done rendering the frame, there might be all kinds of Direct3D state still set that you don't expect. Try turning off alphablending and shaders before drawing your quad. Here's some stuff I had to do:

cybereality wrote:Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

You'll have to explicitly set any renderstate, including none obvious things like render target, scissor regions and destination blend modes.
There is no telling what state the "state machine" is in when you get called.

Thanks. That's good to know.

Just wanted to say well done on the progress so far, excellent work!

All way above my head but it looks like you've got it all under control!

I didn't realize you could control XYZ plane camera movement with an intercept driver,

I assumed it would need to be coded into the game engine, shows what I know...

But it begs the question - why the hell hasn't anyone done this before !!!

@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work. Thus this thread..

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15027

John Carmack was TOTALLY right! the big companies don't even get how amazing this truly can be! Getting it into the hands of the enthusiast crowd is going to have the development skyrocket by leaps and bounds! The stuff you guys are doing is completely amazing! I think I somewhat understand the theory of what you guys are doing, even though I'm not much of a coder myself. Though I can sure offer some moral support!

So cyber, if you get the side by side output going, you'd pretty much just need to implement the "Rift Warp" into the output and you'd theoretically have another working demo to drop on the Kickstarter market. I wonder if Carmack would be willing to give you the spatial distance needed from left eye camera to right eye camera, since he's mapped it out for Doom already. If whatever he's figured out for Doom even applies to LFD.

brantlew wrote:@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work...

I didn't mean for stereo drivers, I mean't for head-tracking kit like the TrackIR etc

seizing controlling the "head" (camera) location in the game and mapping the players actual head location to it,

the position of the head in space, not just the direction the head's eyes are looking in. They really missed a trick there!