It is currently Mon Jul 22, 2019 10:11 am



Reply to topic  [ 333 posts ]  Go to page 1, 2, 3, 4, 5 ... 9  Next
 Cyber's DIY Stereo Driver [Work Log] 
Author Message
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I know I have been talking about making a stereo driver for quite some time, and now I have finally gotten some initial work started on it (up until now it has been mostly research). My main motivation right now is to get something ready for Palmer's Oculus Rift HMD project. So initially it will probably only support that HMD, but in the future I plan to support all variety of devices. I also want to do some more advanced things that aren't available on current 3D drivers. For example, pre-warping for the optics on an HMD and things like that. Down the line I may want to integrate things like head-tracking or force-feedback, but that will probably be further out.

So yeah, lots of big plans so I have my work cut out for me. I still have to decided what to do with the project in terms of commercial options. Initially I want to have a free version, but I will have to figure out if I want to go commercial or even open-source in the future. It may end up being closed-source freeware, I don't know. Also, if IZ3D actually does go open-source, that might effect my decision. We shall see.

Anyway, just wanted to start a thread to use as my work log. So far I have just touched on the basics of hooking into DirectX9 and altering some of the functions. Currently I have just injected a red square and some text into a random game (Zombie Driver in this case). So far this is the only game I could get working, since all the other games I have are on Steam and the way Steam launches games messes up the method I'm using. But I'm sure I can work around it with more time.

Attachment:
ZombieDriver_Hook.jpg

I know it doesn't look like much, but this is a crucial first step to getting some control over DirectX. Will be updating this thread as I make more progress.


You do not have the required permissions to view the files attached to this post.

_________________
check my blog - cybereality.com


Sat May 19, 2012 6:20 pm
Profile
Two Eyed Hopeful

Joined: Fri Apr 15, 2011 8:30 am
Posts: 88
Reply with quote
I applaud the effort and hope you can get something useful from it. Maybe the guys behind the Helix mod that improved 3d for games by altering shaders and whatnot can offer you some good advice:
http://helixmod.wikispot.org/


Sat May 19, 2012 7:03 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Apr 07, 2007 4:34 pm
Posts: 2908
Location: Sweden
Reply with quote
Cool!

_________________
Mb: Asus P5W DH Deluxe
Cpu: C2D E6600
Gb: Nvidia 7900GT + 8800GTX
3D:100" passive projector polarized setup + 22" IZ3D
Image


Sun May 20, 2012 1:43 am
Profile
Golden Eyed Wiseman! (or woman!)

Joined: Fri Jul 08, 2011 11:47 pm
Posts: 1498
Reply with quote
Looks good CR!
One place where you might find some interesting code etc that might relate would be in the source of game hacks, such as aimbots and overlays/wallhacks.


Sun May 20, 2012 6:43 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
WiredEarp wrote:
One place where you might find some interesting code etc that might relate would be in the source of game hacks, such as aimbots and overlays/wallhacks.

Yes, some of my research has certainly gone into this area (which is probably the majority of what people would want to do with DirectX hooking).

_________________
check my blog - cybereality.com


Sun May 20, 2012 1:26 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
Looks good. The "warping" driver is going to be an important part of the Rift project. Good luck.


Sun May 20, 2012 3:21 pm
Profile
Golden Eyed Wiseman! (or woman!)

Joined: Fri Aug 21, 2009 9:06 pm
Posts: 1644
Reply with quote
Awesome! Super excited about this, having a 3D driver written with VR in mind would be amazing.

I think that charging for the driver would be totally acceptable, considering that all the other 3D drivers have a cost, too. At the very least, it would cover any flood of people wanting support (I have read several nightmare stories about people getting flooded with emails for freeware software they released, or even worse, pirated copies of paid software).


Sun May 20, 2012 5:32 pm
Profile
One Eyed Hopeful

Joined: Tue Apr 24, 2012 4:14 am
Posts: 23
Reply with quote
Have you considered using OpenGL in addition to DirectX? I know most gamers use Windows, but it's a pitty that Mac/Linux users are always forgotten...


Mon May 21, 2012 3:15 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
davidgutierrezpalma wrote:
Have you considered using OpenGL in addition to DirectX? I know most gamers use Windows, but it's a pitty that Mac/Linux users are always forgotten...

Yes, I have considered it, but its a very small market compared to all the DirectX games on Windows. However if this project gets off the ground and becomes successful I could revisit a cross-platform OpenGL solution.

_________________
check my blog - cybereality.com


Mon May 21, 2012 5:50 pm
Profile
Two Eyed Hopeful
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 91
Location: Internet
Reply with quote
Awesome project Cyber! If you do decide to open source it, I'd be more than happy to contribute to the project. :)

_________________
₪ 3D Solution⁞ ASUS VG278H 27" w/ Nvidia 3D Vision 2


Tue May 22, 2012 12:47 pm
Profile YIM WWW
Golden Eyed Wiseman! (or woman!)

Joined: Fri Jul 08, 2011 11:47 pm
Posts: 1498
Reply with quote
I was thinking this morning re the warping. Isn't this normally done on a pixel basis (i assume this is what warpalizer does)? So, would intercepting the DX calls really be a valid way to do this?
Are you planning to warp all the vertices, which may achieve the same effect...?


Tue May 22, 2012 6:27 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
I was thinking that the warping would just be a full-screen post-process effect (pixel-shader). I don't think its necessary to warp the actual geometry (that would get kinda crazy).

_________________
check my blog - cybereality.com


Tue May 22, 2012 8:35 pm
Profile
Golden Eyed Wiseman! (or woman!)

Joined: Fri Jul 08, 2011 11:47 pm
Posts: 1498
Reply with quote
Yep, that sounds like it would make much more sense ;-)

OT, but this is kind of stupid:

Quote:
You cannot make another post so soon after your last.


Tue May 22, 2012 10:38 pm
Profile
Cross Eyed!

Joined: Thu Nov 19, 2009 9:35 am
Posts: 140
Reply with quote
I agree with the warping being a post process effect. It would actually be quite similar to the effects some games use to indicate some altered state, such as a boost in racing games, or some sort of enhanced focus in an FPS.

If you would be willing or interested to share some source I'd be very interested, either along the way or just at the end. Sadly I'm a bit busy just now or I'd be very interested in helping out on the development front.

One avenue that might be worth exploring is tapping into the stereo API's already used in some games, such as the new Deus Ex:HR and Dirt3, to persuade them to render in stereo and then display the results in whatever format the user actually requires.

(For example personally I find it quite annoying that the AMD HD3D, while listing the interlaced Zalmans as "supported", doesn't actually provide an interlaced output mode, only HDMI and DisplayPort frame packed modes)


Fri May 25, 2012 6:46 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I have spent the last few days trying to get access to the depth buffer (z-buffer) in DirectX. The idea was to make a 2D+depth mode, as I figured it might be easy and have good compatibility. Apparently this is not something the API exposes normally (they didn't add this until Dx10). So I found a sort of hack to do this using INTZ textures. This seems to work with a simple test app I made, but doesn't work with commercial games. Below is my test app being hooked into to display the depth buffer.

Attachment:
DX9_Box_Z.jpg

With real games all I get is a black square. I guess maybe that games are doing something custom with the depth buffer and not using the automatic one provided by DirectX. However DDD seems to have success with their "Virtual 3D" (now "Power 3D") mode. So I know its possible somehow to get the depth from the scene. Will have to investigate this further at a later date.

Now I am thinking I will try to modify the view and projection matrices to start prepping for full dual render stereo. Like everything else, this probably won't be as easy as it seems, but hopefully not too difficult. Will keep you guys updated on the progress.


You do not have the required permissions to view the files attached to this post.

_________________
check my blog - cybereality.com


Sat May 26, 2012 10:14 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
Interesting. Keep us posted.


Sat May 26, 2012 10:18 pm
Profile
Two Eyed Hopeful
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 91
Location: Internet
Reply with quote
Is there a possibility that the DX depth buffer uses only the alpha channel, and the color channels are to remain black?

_________________
₪ 3D Solution⁞ ASUS VG278H 27" w/ Nvidia 3D Vision 2


Sun May 27, 2012 7:41 am
Profile YIM WWW
Binocular Vision CONFIRMED!
User avatar

Joined: Fri Jan 27, 2012 11:24 am
Posts: 228
Reply with quote
A lot of post process shaders use a full screen quad - that's a flat board fitted exactly in front of the camera - which might explain why you get solid black on your depth map.


Sun May 27, 2012 12:39 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I have attempted to get a dual render stereo working (well right now just moving the camera left and right). I naively though this would be as simple as intercepting "SetTransform" and altering the projection matrix. While this method would have worked with the fixed-function pipeline, its seems no modern games utilize this. Someone on the forum here (forgot who sorry) mentioned this before. What I assume developers are doing is just passing in their own matrix into a vertex shader and doing the computations there. So to get stereo working I would have to intercept this matrix somehow before it gets into the shader. I think its possible, but just a bit harder than I imagined. This also means it probably won't be a generic solution that will work for every game, as each game might be doing something slightly different. Still have to do more research on this aspect, but hopefully it won't be too difficult.

_________________
check my blog - cybereality.com


Mon May 28, 2012 1:59 pm
Profile
Cross Eyed!

Joined: Sat Jul 31, 2010 12:08 pm
Posts: 101
Reply with quote
Yes I mentioned it.

I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.
The projection matrix will have none trivial values in the 4th row or column, you have the added complexity of it being possible to write shaders that treat matrices as either row or column major, there is a convention (can't remember which one it is in the shader) but there is no enforcement of it. I'd still expect most games to follow it.


Wed May 30, 2012 11:22 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
@ERP: Ok, thanks for the advice.

_________________
check my blog - cybereality.com


Wed May 30, 2012 5:36 pm
Profile
Cross Eyed!
User avatar

Joined: Sat Apr 26, 2008 4:23 pm
Posts: 160
Location: Montréal, Canada
Reply with quote
ERP wrote:
I suspect what the NVidia and other 3D drivers do is use some heuristic to identify the projection matrix.
It's got a very distinctive "shape".
I'd start by intercepting calls to SetVertexShaderConstantF with counts of 16 or more and looking to see if you can see the projection matrix, and whether it's been combined with the other matrices on the CPU.

Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil


Wed May 30, 2012 7:08 pm
Profile WWW
Cross Eyed!

Joined: Sat Jul 31, 2010 12:08 pm
Posts: 101
Reply with quote
phil wrote:
Hi, I read that the NVidia drivers work by appending some code onto the vertex shaders to shift the projected positions that have been calculated there.

The problem with doing that, rather than altering the matrices at the start, is that it leads to incorrect specular lighting, environment mapping, etc. (since the vertex shader is unaware of the view & projection offsets, it can't take them into account, nor can the pixel shader that it feeds into). The gloss and reflections will then look painted-on (like in a converted film, yuck). In fact everything view-dependent that the shaders produce will be wrong, apart from the vertex positions!

On the other hand, if you try to intercept the view & projection matrices, I expect you'll find it's more common to see a combined view-and-projection (world-to-clip) matrix used instead, or a world-view-projection (model-to-clip). In the latter case, the fact that the matrix already contains the orientation of the object being drawn might be a problem. You might need to know the view orientation in order to apply the view offset in the right direction, I'm not sure about that.

Good luck!
phil



That makes a lot of sense, explains a lot of the common artifacts, anything view dependent computed in the vertex shader will be wrong.
It's likely how the Iz3d drivers work as well, they likely append instructions to the end of the shader.

I'm not sure there is a better way to do it in the world of programable shaders.

Means there is likely no perfect automatic solution for any game.


Thu May 31, 2012 5:10 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
We should lobby the major engines like Unreal to include proper stereo support out of the box. Maybe then game developers would just leave it enabled and we would get more game support automatically.


Thu May 31, 2012 5:21 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
Well Unreal Engine already has some 3D support (via RealD and TriOViz) and so does CryEngine and probably some others. The issue is that game developers are not utilizing them.

Anyway, even if the major engines supported 3D, it would likely only be the very common stuff (ie HDMI 1.4a, 3D Vision, etc.). Certainly I doubt they would support strange DIY setups (the DOOM 3 thing is the exception). Plus, there is a bunch of other stuff I want to do with hooking DirectX (like full 6DOF headtracking, force-feedback, etc.), stereo 3D is just the first step.

_________________
check my blog - cybereality.com


Thu May 31, 2012 5:30 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I began with intercepting "SetVertexShaderConstantF" and trying to find the right matrix. Right now I am looking at arrays of floats with 16 or greater elements (which I am assuming are matrices). I am able to print the results on screen for testing.

So far in Left4Dead I am getting something, looks like maybe a translation matrix. It stays fairly steady when I am still, but the numbers change when I move or rotate the camera. Certainly looks like something important, will have to investigate further to see how useful this will prove. But I feel like I am on the right direction.

Attachment:
L4D_Matrix.png


You do not have the required permissions to view the files attached to this post.

_________________
check my blog - cybereality.com


Thu May 31, 2012 10:13 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I started trying to alter the matrices/vectors I'm looking at. Barely working, but I have some control. Just got to find the right matrix.


_________________
check my blog - cybereality.com


Fri Jun 01, 2012 6:20 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
This is a cool exercise in reverse engineering. You should consider starting a technical blog - not necessarily giving away the source code but showing the basic hooking mechanisms and functions calls so others can create similar projects. This type of info is often difficult to come by - especially aggregated in a single place.


Fri Jun 01, 2012 7:13 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Fri Jan 27, 2012 11:24 am
Posts: 228
Reply with quote
You are probably intercepting the matrix that is used to render just that object. Try adding the same vector to the last column of every matrix you come across.


Sat Jun 02, 2012 5:17 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
So I have made some serious progress in just a few days. Managed to hack together camera movement in L4D. Currently it's the only game that's working, since I'm hard-coding the values to modify. Basically looked for matrices with exactly 16 elements, and then did trial and error altering each element in the array. Will have to come up with some generic way to handle this, although I guess creating a profile for each game wouldn't be too hard.



Next up I will try getting multiple viewports working for side-by-side 3D. Stay tuned.

_________________
check my blog - cybereality.com


Mon Jun 04, 2012 9:03 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
Nice!


Mon Jun 04, 2012 11:55 pm
Profile
One Eyed Hopeful

Joined: Thu Sep 22, 2011 9:41 am
Posts: 2
Reply with quote
I'm working on something similar for my HMD based on foisi's concept. So far I have head position injection and split image output. I only did TrackIR enabled games so far. These had an additional transformation matrices somewhere in memory. I found these either by looking through disassembly or searching memory with CheatEngine.

Here's a screenshot from F1 2011. It might look stereoscopic at first glance but it's actually just parts of the same image duplicated and aligned for the left/right eyes. It's done by hooking Present, grabbing the backbuffer and using it as a texture for two quads. http://i.imgur.com/qZAea.jpg

I don't plan to do stereo images, I'm not too fond of halving my framerate and fixing all the small issues coming from shoehorning 3D rendering into games not meant for it. Great to see someone up for the challenge, though. :)


Wed Jun 06, 2012 5:48 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

_________________
check my blog - cybereality.com


Wed Jun 06, 2012 6:11 pm
Profile
One Eyed Hopeful

Joined: Thu Sep 22, 2011 9:41 am
Posts: 2
Reply with quote
After the game is done rendering the frame, there might be all kinds of Direct3D state still set that you don't expect. Try turning off alphablending and shaders before drawing your quad. Here's some stuff I had to do:

Code:
device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_SELECTARG1);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_CONSTANT);
device->SetTextureStageState(0, D3DTSS_CONSTANT, 0xffffffff);
device->SetRenderState(D3DRS_ALPHABLENDENABLE, FALSE);

device->SetVertexShader(NULL);
device->SetPixelShader(NULL);


Thu Jun 07, 2012 4:14 am
Profile
Cross Eyed!

Joined: Sat Jul 31, 2010 12:08 pm
Posts: 101
Reply with quote
cybereality wrote:
Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?


You'll have to explicitly set any renderstate, including none obvious things like render target, scissor regions and destination blend modes.
There is no telling what state the "state machine" is in when you get called.


Thu Jun 07, 2012 3:46 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11393
Reply with quote
Thanks. That's good to know.

_________________
check my blog - cybereality.com


Thu Jun 07, 2012 5:32 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Reply with quote
Just wanted to say well done on the progress so far, excellent work!

All way above my head but it looks like you've got it all under control! :D

I didn't realize you could control XYZ plane camera movement with an intercept driver,

I assumed it would need to be coded into the game engine, shows what I know...

But it begs the question - why the hell hasn't anyone done this before !!! :o


Wed Jun 13, 2012 8:03 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2218
Location: Menlo Park, CA
Reply with quote
@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work. Thus this thread..

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15027


Last edited by brantlew on Wed Jun 13, 2012 8:27 pm, edited 1 time in total.



Wed Jun 13, 2012 8:24 pm
Profile
Cross Eyed!

Joined: Thu Jun 07, 2012 9:28 am
Posts: 171
Reply with quote
John Carmack was TOTALLY right! the big companies don't even get how amazing this truly can be! Getting it into the hands of the enthusiast crowd is going to have the development skyrocket by leaps and bounds! :o The stuff you guys are doing is completely amazing! I think I somewhat understand the theory of what you guys are doing, even though I'm not much of a coder myself. Though I can sure offer some moral support!

So cyber, if you get the side by side output going, you'd pretty much just need to implement the "Rift Warp" into the output and you'd theoretically have another working demo to drop on the Kickstarter market. I wonder if Carmack would be willing to give you the spatial distance needed from left eye camera to right eye camera, since he's mapped it out for Doom already. If whatever he's figured out for Doom even applies to LFD.


Wed Jun 13, 2012 8:26 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Reply with quote
brantlew wrote:
@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work...

I didn't mean for stereo drivers, I mean't for head-tracking kit like the TrackIR etc

seizing controlling the "head" (camera) location in the game and mapping the players actual head location to it,

the position of the head in space, not just the direction the head's eyes are looking in. They really missed a trick there!


Wed Jun 13, 2012 9:15 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 333 posts ]  Go to page 1, 2, 3, 4, 5 ... 9  Next

Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.