It is currently Fri Nov 22, 2019 12:01 am



Reply to topic  [ 333 posts ]  Go to page Previous  1, 2, 3, 4, 5 ... 9  Next
 Cyber's DIY Stereo Driver [Work Log] 
Author Message
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.

_________________
check my blog - cybereality.com


Wed Jun 13, 2012 9:24 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Reply with quote
Getting both stereo and head-tracking working properly together is the holy-grail,

and pretty fundamental to the HMD experience, so no pressure then...

Keep up the good work! :)


Wed Jun 13, 2012 9:36 pm
Profile
Cross Eyed!

Joined: Sat Jul 31, 2010 12:08 pm
Posts: 101
Reply with quote
cybereality wrote:
Well stereo is only part of what I am doing. Once I have control over the camera I can do other cool stuff like 6DOF headtracking.


I think you'll find that won't work for most games.
You'd have to intercept camera position in the game code, not in the GPU calls.
The issue is going to be that the game will have done a basic clip and discarded offscreen geometry based on it's own internal representation before it submits to the GPU. So significant camera motion will have large portions of the scene missing.
This is the reason NVidia allow you to adjust the Frustum slightly in the 3DVision advanced settings. For example certain areas in WOW exhibited missing geometry at the screen edges even with the very slight L/R camera motion required for 3D.


Thu Jun 14, 2012 2:05 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
@ERP: Yes, the more I work with it, the more I think I have going to have to inject into the games themselves. This is surely possible, but will require specialized code for each game. However, it will give me a greater deal of control than with intercepting DirectX. It also opens the door to other stuff I want to do, like haptics/force-feedback which will require specific game knowledge.

_________________
check my blog - cybereality.com


Thu Jun 14, 2012 5:29 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
I *really* want to try Mirror's Edge on a Rift... so, last night I started hacking up a prototype of a driver similar to this, just for Mirror's Edge.

It does 3D for now--I'm going to start seeing if I can inject a displacement mapping shader soon to correct for the optics. I'm also thinking it might be possible to hack in roll-axis view support, since that shouldn't require seeing things that have gotten clipped out.

Check it out, the glitchiness around the two minute mark is from me messing with the convergence/separation to try to ham up the 3D-ish-ness.


Sun Jun 17, 2012 12:49 am
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
@Emerson: Woah, that kicks ass! Is that your code? If you get the warping transform working, please share the binary (or source code :) ) with us. Mirrors Edge would be psychotic on the Rift.


Sun Jun 17, 2012 8:23 am
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
The only thing I haven't coded so far, that I would change before I release anything, is that I'm leaning on a pretty common off-the-shelf game cheating kit to create the intercept DLL right now. I don't think that's a problem for Mirror's Edge per se, but I'm worried someone would try dropping it into another game and trigger an anti-cheat ban off of the fingerprints. Something else for the to-do list, I suppose.


Sun Jun 17, 2012 10:26 am
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.


Sun Jun 17, 2012 10:56 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
Hey Emerson, that looks awesome. Mirror's Edge is also one of my favorite games of all time.

Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

_________________
check my blog - cybereality.com


Sun Jun 17, 2012 10:57 am
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
brantlew wrote:
Mind me asking what the "game kit" is? Is it a linked library or open source? It would be great to get a repository of these injection samples and techniques into the open source to build upon.

It's Azorbix's "D3D Starter Kit" from gamedeception, but I'm just using it as a proxy dll for now, not a full injection.

cybereality wrote:
Do you think you could explain a little about how you are doing the 3D? I don't need to see any code, but just a brief overview would be really helpful to me. Thanks.

There's a couple cheap tricks to make it work--first off, since I can't actually reproduce the entire render calls for the second viewport, I'm actually just switching off on frames between left and right, and delaying the backbuffer swap until both have rendered (otherwise you get pretty nasty flashes). So, technically, the left and right are slightly temporally shifted, but it still works for the most part.

Secondly, to actually create the 3D effect off the alternating views, I intercept the WorldViewProjection matrix on its way through SetVertexShaderConstantF. I multiply it by an inverted projection matrix I math'd out, then a translation matrix for the eyeball offset, then another projection matrix based on the first that does the assymmetric view frustums.

Lots of trial and error to get it right, so far.


Sun Jun 17, 2012 11:16 am
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
@Emerson: Ok, thanks man. I have actually gone down a similar road with hooking SetVertexShaderConstantF. But I am having some trouble determining which is the world/view/projection matrix. Currently I am just altering every 4x4 matrix I find but I feel like this has strange side-effects. Any hints on how you determine this?

_________________
check my blog - cybereality.com


Sun Jun 17, 2012 12:43 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
I did some testing with logging code that just spits out each matrix that goes through, along with its register--in Mirror's Edge's case the WVP is always in register 0. I also threw a check to see if the [4,1] element is non-zero to try to reject anything that hasn't been through a projection matrix already, but that might be superfluous.


Sun Jun 17, 2012 12:53 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
Good to know, thanks.

_________________
check my blog - cybereality.com


Sun Jun 17, 2012 2:00 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Fri Jan 27, 2012 11:24 am
Posts: 228
Reply with quote
This is great stuff guys. I love Mirror's Edge as well, especially the first few levels then it goes a bit off. I always find myself waiting on the menu for ages because the music is awesome.


Sun Jun 17, 2012 3:31 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
Good progress today! Got the basic optic correction distortion mapping done--though obviously I'm lacking the real data to do the correction for now.

That's it for my weekend hacking--I might try to clean up what I've got this week and get it on github, to see what other people make of it.


Image


Mon Jun 18, 2012 12:45 am
Profile
Golden Eyed Wiseman! (or woman!)

Joined: Fri Jul 08, 2011 11:47 pm
Posts: 1498
Reply with quote
Emerson, that looks wicked! I'm impressed. If that can be done for other games as well, then that would eliminate one of the main disadvantages of the Rift.

Are you able to perform basic scaling between different resolutions? IE, do you think it would be possible to have a game rendering to a virtual 720P screen (using 720P in the games settings) but your driver outputting to a different resolution (such as 640x480), scaling as it goes? That would be extremely handy for a very wide range of HMD's that only support a limited number of resolutions, and combined with your warping code would be a perfect solution.


Mon Jun 18, 2012 1:29 am
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
@emerson: Awesome work!

Carmack posted one in-game still on his Twitter account but that's the only thing I have seen that would allow for any kind of validation. Maybe Palmer or Carmack could supply us with a still image of a warped grid? That would allow us to work out the warp transforms in advance of the Rift release.


Mon Jun 18, 2012 7:50 am
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
WiredEarp wrote:
Emerson, that looks wicked! I'm impressed. If that can be done for other games as well, then that would eliminate one of the main disadvantages of the Rift.
Are you able to perform basic scaling between different resolutions? IE, do you think it would be possible to have a game rendering to a virtual 720P screen (using 720P in the games settings) but your driver outputting to a different resolution (such as 640x480), scaling as it goes? That would be extremely handy for a very wide range of HMD's that only support a limited number of resolutions, and combined with your warping code would be a perfect solution.

I think it ought to be feasible--I might give it a shot when I get some spare time this week.

brantlew wrote:
Carmack posted one in-game still on his Twitter account but that's the only thing I have seen that would allow for any kind of validation. Maybe Palmer or Carmack could supply us with a still image of a warped grid? That would allow us to work out the warp transforms in advance of the Rift release.

Yeah, that screenshot's what I've been trying to emulate with my quick-and-dirty photoshopped distortion map, but a grid would at least give a more accurate sense of the internal distortion, or give a basis to try and decipher a rough analytical model of the transform.


Mon Jun 18, 2012 1:18 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Fri Jan 27, 2012 11:24 am
Posts: 228
Reply with quote
Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

EDIT;

Judging by this screenshot, you look pretty good with the distortion; maybe you are not quite doing enough...
Image


Mon Jun 18, 2012 2:12 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
Chriky wrote:
Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

Good eye. The sides are flipped from what you'd use for an HMD, this works for "cross-eyed" stereoscopy, which is how I've been testing all this so far. Pretty simple to switch once I get the real deal. My eyes are getting pretty fatigued though. :shock:


Mon Jun 18, 2012 3:35 pm
Profile
One Eyed Hopeful

Joined: Sun May 20, 2012 7:28 pm
Posts: 11
Reply with quote
Hey guys, thought I'd post this as it may be of use.. from John Carmacks twitter.

Image

Great work, I'm following keenly! :)


Mon Jun 18, 2012 3:46 pm
Profile
Cross Eyed!

Joined: Thu Nov 19, 2009 9:35 am
Posts: 140
Reply with quote
Emerson wrote:
Chriky wrote:
Looking brilliant.

Am I going mad or are the left and right eyes the wrong way around?

Good eye. The sides are flipped from what you'd use for an HMD, this works for "cross-eyed" stereoscopy, which is how I've been testing all this so far. Pretty simple to switch once I get the real deal. My eyes are getting pretty fatigued though. :shock:



Thank god, I thought I was going nuts too, I have a naturally lazy eye/divergent eyes so parallel viewing is much more natural to me than cross-eyed. I assumed it would be set up as parallel for the HMD but seemed the wrong eyes.

It's interesting looking at Carmack's latest image (just posted above) on a non distorted system, it makes the flat wall look like a barrel popping out of the screen at you.

Mirrors Edge should be great with the Rift, it's already good to play on my zalman but the Rift (especially if the head tracking works) will be a whole different game.
Personally I'd love to play Dirt3 or a similar racing game though, sadly its cockpit view would benefit most from the parallax effect of shifting your head, rather than just tilting, requiring some additional head tracking.


Mon Jun 18, 2012 5:32 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
sorry cyber, I think your thread has been hijacked. maybe we should start a dedicated thread for Rift 3D Warp Correction Drivers or something


Mon Jun 18, 2012 6:02 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
This is sounding really cool. Looks like I have to step up my game now that there's some competition.

@brantlew: I don't mind the hijack, as this is all very related and helpful to the work I'm doing.

_________________
check my blog - cybereality.com


Mon Jun 18, 2012 6:17 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
brantlew wrote:
sorry cyber, I think your thread has been hijacked. maybe we should start a dedicated thread for Rift 3D Warp Correction Drivers or something


Yeah... that's totally my bad. Sorry. :(

I can start a new thread for my stuff if needed, would it be better in here or the VR/R&D forum?


Mon Jun 18, 2012 6:20 pm
Profile
Binocular Vision CONFIRMED!

Joined: Sat Nov 10, 2007 7:45 am
Posts: 287
Reply with quote
Wow! Great thread, Cyberreality and Emerson. I just ordered up a copy of Mirror's Edge in anticipation of the Rift and will spend time mastering it on my projection system. :-)

Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.


Mon Jun 18, 2012 6:37 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
Mel wrote:
Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

Honestly, this is my first real dive into DirectX myself--I'm usually more of an OpenGL guy. I've been figuring things out by looking at the DX device interfaces (in the aforementioned D3D Starter Kit from gamedeception), googling lots of MSDN docs, and watching my target game's DX calls with an API Monitor tool http://www.rohitab.com/apimonitor (bit like drinking from a firehose, though).

If you're also an OpenGL type of person, watch out for the matrices, they're effectively transposed between DX and OpenGL.


Mon Jun 18, 2012 6:50 pm
Profile
Cross Eyed!

Joined: Thu Jun 07, 2012 9:28 am
Posts: 171
Reply with quote
Emerson wrote:
Yeah, that screenshot's what I've been trying to emulate with my quick-and-dirty photoshopped distortion map, but a grid would at least give a more accurate sense of the internal distortion, or give a basis to try and decipher a rough analytical model of the transform.

Ask and ye shall receive. I can't really develop anything, but with this I can help. (sorry for the huge img, but though it best for you to have it full res.)
I'm assuming you can take a non-distorted half screen, run it through the distortion settings in Photoshop and then have something to shoot for/reference.

Image

Matrix is blue lines every 20 pixels. Steps are on the bottom. That's from a full 640 x 800 red rectangle.

EDIT: thought I should clarify steps.
1. transform, right click warp.
2. in the warp drop down, select inflate (this will make the image much larger than intended final size)
3. apply warp, and then transform with pixel setting on to 800 height


Mon Jun 18, 2012 6:57 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
Mel wrote:
Question: I've been writing code for over twenty years, and while I've read many books on 3D graphics programming theory for the shear joy of it, I've sadly never had occasion to do any 3D work that involved DirectX. Can either of you recommend a good book or two on the subject? Id like to take a shot at some code hooking and injection. Thanks.

To be honest I have just been learning DirectX recently mostly for this project. A good book on the subject is Frank Luna's Intro to 3D Programming. Make sure you know your stuff with C++ though first. If you want to learn C++ try C++ Primer Plus by Stephen Prata.

_________________
check my blog - cybereality.com


Mon Jun 18, 2012 8:29 pm
Profile
Binocular Vision CONFIRMED!

Joined: Sat Nov 10, 2007 7:45 am
Posts: 287
Reply with quote
Thanks for the DX advice, CR and Emerson. I'll look into the book and do some reading online and try hooking that one DX call that's been mentioned somewhere in all the Rift related threads, and see what I can see.


Mon Jun 18, 2012 10:34 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
@Alkapwn: One thing I have noticed about that screen shot if you look closely is that the lines tend to converge more towards the right edge of the screen. I don't know if the warp function distorts asymmetrically from side to side or if that is just a bit of head roll causing lateral perspective. Carmack may have just took a quick snap-shot without trying to tilt the head into a perfect downward direction. Also, he mentioned that this was an early example and a bit under-warped. The video images from E3 look a bit more convex to me. Hey Palmer can you give us a little guidance on this? Is there lateral asymmetry in the lens distortion? Are there any hard numbers you can give us with from Nthusim that might help?


Tue Jun 19, 2012 7:29 am
Profile
Cross Eyed!

Joined: Thu Jun 07, 2012 9:28 am
Posts: 171
Reply with quote
@brantlew
Yah I was looking at that when I started to create the distorted one. I think it's because in that screenshot, he's not looking at the wall/floor fully square head-on. I'm hoping that Mr. Carmack will release that info for everyone.


Tue Jun 19, 2012 9:00 am
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
Okay, I managed to derive an analytical solution based on Alkapwn's grid--it's a lot closer than it was before (ME renders black bars on its UI so the seemingly more squashed look is not the fault of the proxy dll).

Image

If anyone's curious: if your inputs are u and v on the [-1,1] range, then the outputs x=c*u/(v^2+c), y=c*v/(u^2+c), where c=-81/10, over the [-1,1] range. (Gotta scale it from and back to [0,1] in the shader, though)


Tue Jun 19, 2012 11:23 am
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
Excellent! It's even at the correct resolution. Hey Palmer, if you get a chance can you look at this image and see how close Emerson is getting to the proper warp correction?


Tue Jun 19, 2012 11:32 am
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
@Emerson: Hey I think you're doing a great job with this, but one thing I notice is aliasing where the convergence is greatest - particularly noticeable in the bottom text. This is how I imagined a post-process warp driver would look since it can only operate from the final rendered 640x800 image. What are the ways to combat this? I assume you could do some type of standard 2D pixel averaging from the 640x800 image. Could you also attack it further up the rendering pipleline? Can you define the projection plane as a curved surface? This would allow supersampling as well. Maybe with an injection driver you cannot gain this granularity of control? Just curious. Excuse my ignorance but I haven't done graphics in 15 years.


Tue Jun 19, 2012 12:58 pm
Profile
One Eyed Hopeful

Joined: Wed Jun 13, 2012 11:16 pm
Posts: 34
Reply with quote
Quote:
@Emerson: Hey I think you're doing a great job with this, but one thing I notice is aliasing where the convergence is greatest - particularly noticeable in the bottom text. This is how I imagined a post-process warp driver would look since it can only operate from the final rendered 640x800 image. What are the ways to combat this?

I've been thinking about this--I think my first attack on the problem will be to keep it simple: I'll try to render the initial buffer prior to warping at 1280x1600 and effectively try to do an old-school FSAA approach to smooth things out a bit. If that works, great, otherwise I'll have to start digging into the more complicated algos.


Tue Jun 19, 2012 1:43 pm
Profile
One Eyed Hopeful

Joined: Tue Jun 19, 2012 1:56 pm
Posts: 19
Reply with quote
@Emerson:

Looks great, but we've to find a solution for the heavy aliasing.

spyro


Tue Jun 19, 2012 1:59 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Fri Jan 27, 2012 11:24 am
Posts: 228
Reply with quote
I've been using a more hacky solution of rendering each eye out to a 1024x1024 texture, then using them to texture actual 3D distorted planes which are in front of the main camera.

So, two 'eye' cameras with 4:5 aspect ratio rendering to 1024x1024 textures, then one 'main' camera with 16:9 ratio just looking at some sphere segments textured with the 'eye' images.


Tue Jun 19, 2012 3:13 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
@Emerson: Your solution is looking better and better. Thanks a bunch for posting the warping equation, this will help me a lot.

@brantlew: To be honest, I'm not sure the AA is needed. I know at least with the older prototype Palmer sent me, there was a heavy diffusion layer on it and the image had a soft look to it. If the final Rift is anything like this, then it will be a little blurry anyway, so I don't think the aliasing will be a problem. Though you can still notice pixels somewhat, so I guess an AA pass could help fake a higher resolution.

_________________
check my blog - cybereality.com


Tue Jun 19, 2012 5:30 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
@cyber: Yeah just like all AA, it's not strictly necessary although on the Rift it's a little different because different parts of the screen will look more aliased than others. Of course the worst aliasing is towards the edge of your vision so less of a concern. If the only solution is to render at high resolution and down sample, I would rather not take the performance hit for it. But if there is a simple way that doesn't add too much overhead, it would be nice.


Last edited by brantlew on Tue Jun 19, 2012 10:20 pm, edited 1 time in total.



Tue Jun 19, 2012 7:06 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 333 posts ]  Go to page Previous  1, 2, 3, 4, 5 ... 9  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.