Nice!

I'm working on something similar for my HMD based on foisi's concept. So far I have head position injection and split image output. I only did TrackIR enabled games so far. These had an additional transformation matrices somewhere in memory. I found these either by looking through disassembly or searching memory with CheatEngine.

Here's a screenshot from F1 2011. It might look stereoscopic at first glance but it's actually just parts of the same image duplicated and aligned for the left/right eyes. It's done by hooking Present, grabbing the backbuffer and using it as a texture for two quads. http://i.imgur.com/qZAea.jpg

I don't plan to do stereo images, I'm not too fond of halving my framerate and fixing all the small issues coming from shoehorning 3D rendering into games not meant for it. Great to see someone up for the challenge, though.

Good stuff. Do you have any advice on rendering the split screen part?

I tried rendering a quad straight into screen space, which works sometimes, but I noticed it doesn't show up in a number of games (or it will show at the beginning and then disappear once the game starts). Any advice?

After the game is done rendering the frame, there might be all kinds of Direct3D state still set that you don't expect. Try turning off alphablending and shaders before drawing your quad. Here's some stuff I had to do:

You'll have to explicitly set any renderstate, including none obvious things like render target, scissor regions and destination blend modes.
There is no telling what state the "state machine" is in when you get called.

Thanks. That's good to know.

Just wanted to say well done on the progress so far, excellent work!

All way above my head but it looks like you've got it all under control!

I didn't realize you could control XYZ plane camera movement with an intercept driver,

I assumed it would need to be coded into the game engine, shows what I know...

But it begs the question - why the hell hasn't anyone done this before !!!

@Nick3DvB: They have. This is how nVidia / Tridef / IZ3D drivers work. Thus this thread..

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15027

John Carmack was TOTALLY right! the big companies don't even get how amazing this truly can be! Getting it into the hands of the enthusiast crowd is going to have the development skyrocket by leaps and bounds! The stuff you guys are doing is completely amazing! I think I somewhat understand the theory of what you guys are doing, even though I'm not much of a coder myself. Though I can sure offer some moral support!

So cyber, if you get the side by side output going, you'd pretty much just need to implement the "Rift Warp" into the output and you'd theoretically have another working demo to drop on the Kickstarter market. I wonder if Carmack would be willing to give you the spatial distance needed from left eye camera to right eye camera, since he's mapped it out for Doom already. If whatever he's figured out for Doom even applies to LFD.

I didn't mean for stereo drivers, I mean't for head-tracking kit like the TrackIR etc

seizing controlling the "head" (camera) location in the game and mapping the players actual head location to it,

the position of the head in space, not just the direction the head's eyes are looking in. They really missed a trick there!

Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

Getting both stereo and head-tracking working properly together is the holy-grail,

and pretty fundamental to the HMD experience, so no pressure then...

Keep up the good work!

I think you'll find that won't work for most games.
You'd have to intercept camera position in the game code, not in the GPU calls.
The issue is going to be that the game will have done a basic clip and discarded offscreen geometry based on it's own internal representation before it submits to the GPU. So significant camera motion will have large portions of the scene missing.
This is the reason NVidia allow you to adjust the Frustum slightly in the 3DVision advanced settings. For example certain areas in WOW exhibited missing geometry at the screen edges even with the very slight L/R camera motion required for 3D.

@ERP: Yes, the more I work with it, the more I think I have going to have to inject into the games themselves. This is surely possible, but will require specialized code for each game. However, it will give me a greater deal of control than with intercepting DirectX. It also opens the door to other stuff I want to do, like haptics/force-feedback which will require specific game knowledge.

I *really* want to try Mirror's Edge on a Rift... so, last night I started hacking up a prototype of a driver similar to this, just for Mirror's Edge.

It does 3D for now--I'm going to start seeing if I can inject a displacement mapping shader soon to correct for the optics. I'm also thinking it might be possible to hack in roll-axis view support, since that shouldn't require seeing things that have gotten clipped out.

Check it out, the glitchiness around the two minute mark is from me messing with the convergence/separation to try to ham up the 3D-ish-ness.

@Emerson: Woah, that kicks ass! Is that your code? If you get the warping transform working, please share the binary (or source code ) with us. Mirrors Edge would be psychotic on the Rift.

The only thing I haven't coded so far, that I would change before I release anything, is that I'm leaning on a pretty common off-the-shelf game cheating kit to create the intercept DLL right now. I don't think that's a problem for Mirror's Edge per se, but I'm worried someone would try dropping it into another game and trigger an anti-cheat ban off of the fingerprints. Something else for the to-do list, I suppose.

Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

Hey Emerson, that looks awesome. Mirror's Edge is also one of my favorite games of all time.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

It's Azorbix's "D3D Starter Kit" from gamedeception, but I'm just using it as a proxy dll for now, not a full injection.


There's a couple cheap tricks to make it work--first off, since I can't actually reproduce the entire render calls for the second viewport, I'm actually just switching off on frames between left and right, and delaying the backbuffer swap until both have rendered (otherwise you get pretty nasty flashes). So, technically, the left and right are slightly temporally shifted, but it still works for the most part.

Secondly, to actually create the 3D effect off the alternating views, I intercept the WorldViewProjection matrix on its way through SetVertexShaderConstantF. I multiply it by an inverted projection matrix I math'd out, then a translation matrix for the eyeball offset, then another projection matrix based on the first that does the assymmetric view frustums.

Lots of trial and error to get it right, so far.

@Emerson: Ok, thanks man. I have actually gone down a similar road with hooking SetVertexShaderConstantF. But I am having some trouble determining which is the world/view/projection matrix. Currently I am just altering every 4x4 matrix I find but I feel like this has strange side-effects. Any hints on how you determine this?

I did some testing with logging code that just spits out each matrix that goes through, along with its register--in Mirror's Edge's case the WVP is always in register 0. I also threw a check to see if the [4,1] element is non-zero to try to reject anything that hasn't been through a projection matrix already, but that might be superfluous.

Good to know, thanks.

This is great stuff guys. I love Mirror's Edge as well, especially the first few levels then it goes a bit off. I always find myself waiting on the menu for ages because the music is awesome.

Good progress today! Got the basic optic correction distortion mapping done--though obviously I'm lacking the real data to do the correction for now.

That's it for my weekend hacking--I might try to clean up what I've got this week and get it on github, to see what other people make of it.

Emerson, that looks wicked! I'm impressed. If that can be done for other games as well, then that would eliminate one of the main disadvantages of the Rift.

Are you able to perform basic scaling between different resolutions? IE, do you think it would be possible to have a game rendering to a virtual 720P screen (using 720P in the games settings) but your driver outputting to a different resolution (such as 640x480), scaling as it goes? That would be extremely handy for a very wide range of HMD's that only support a limited number of resolutions, and combined with your warping code would be a perfect solution.

@emerson: Awesome work!

Carmack posted one in-game still on his Twitter account but that's the only thing I have seen that would allow for any kind of validation. Maybe Palmer or Carmack could supply us with a still image of a warped grid? That would allow us to work out the warp transforms in advance of the Rift release.

I think it ought to be feasible--I might give it a shot when I get some spare time this week.


Yeah, that screenshot's what I've been trying to emulate with my quick-and-dirty photoshopped distortion map, but a grid would at least give a more accurate sense of the internal distortion, or give a basis to try and decipher a rough analytical model of the transform.

Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

EDIT;

Judging by this screenshot, you look pretty good with the distortion; maybe you are not quite doing enough...

Good eye. The sides are flipped from what you'd use for an HMD, this works for "cross-eyed" stereoscopy, which is how I've been testing all this so far. Pretty simple to switch once I get the real deal. My eyes are getting pretty fatigued though.

Hey guys, thought I'd post this as it may be of use.. from John Carmacks twitter.



Great work, I'm following keenly!

Thank god, I thought I was going nuts too, I have a naturally lazy eye/divergent eyes so parallel viewing is much more natural to me than cross-eyed. I assumed it would be set up as parallel for the HMD but seemed the wrong eyes.

It's interesting looking at Carmack's latest image (just posted above) on a non distorted system, it makes the flat wall look like a barrel popping out of the screen at you.

Mirrors Edge should be great with the Rift, it's already good to play on my zalman but the Rift (especially if the head tracking works) will be a whole different game.
Personally I'd love to play Dirt3 or a similar racing game though, sadly its cockpit view would benefit most from the parallax effect of shifting your head, rather than just tilting, requiring some additional head tracking.

sorry cyber, I think your thread has been hijacked. maybe we should start a dedicated thread for Rift 3D Warp Correction Drivers or something

This is sounding really cool. Looks like I have to step up my game now that there's some competition.

@brantlew: I don't mind the hijack, as this is all very related and helpful to the work I'm doing.

Yeah... that's totally my bad. Sorry.

I can start a new thread for my stuff if needed, would it be better in here or the VR/R&D forum?

Wow! Great thread, Cyberreality and Emerson. I just ordered up a copy of Mirror's Edge in anticipation of the Rift and will spend time mastering it on my projection system.

Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

Honestly, this is my first real dive into DirectX myself--I'm usually more of an OpenGL guy. I've been figuring things out by looking at the DX device interfaces (in the aforementioned D3D Starter Kit from gamedeception), googling lots of MSDN docs, and watching my target game's DX calls with an API Monitor tool http://www.rohitab.com/apimonitor (bit like drinking from a firehose, though).

If you're also an OpenGL type of person, watch out for the matrices, they're effectively transposed between DX and OpenGL.

Ask and ye shall receive. I can't really develop anything, but with this I can help. (sorry for the huge img, but though it best for you to have it full res.)
I'm assuming you can take a non-distorted half screen, run it through the distortion settings in Photoshop and then have something to shoot for/reference.



Matrix is blue lines every 20 pixels. Steps are on the bottom. That's from a full 640 x 800 red rectangle.

EDIT: thought I should clarify steps.
1. transform, right click warp.
2. in the warp drop down, select inflate (this will make the image much larger than intended final size)
3. apply warp, and then transform with pixel setting on to 800 height

To be honest I have just been learning DirectX recently mostly for this project. A good book on the subject is Frank Luna's Intro to 3D Programming. Make sure you know your stuff with C++ though first. If you want to learn C++ try C++ Primer Plus by Stephen Prata.

Thanks for the DX advice, CR and Emerson. I'll look into the book and do some reading online and try hooking that one DX call that's been mentioned somewhere in all the Rift related threads, and see what I can see.