Cyber's DIY Stereo Driver [Work Log]

Post by **Nick3DvB** » Tue Jun 19, 2012 9:31 pm

I think I remember reading Palmer ditched the diffuser because of the loss of contrast?

Good call IMHO, people can add one themself if they really want to give their Rift cataracts!

Maybe Nvidia's new FXAA modes could help us? Interesting bit on temporal AA here:

http://timothylottes.blogspot.co.uk/201 ... -info.html

Post by **Emerson** » Tue Jun 19, 2012 11:42 pm

One quick fix that I found--I was using point sampling on the intermediary texture, not lerping--it helps a bit:

Post by **brantlew** » Wed Jun 20, 2012 12:12 am

That looks a lot better. Perfectly playable as far as I'm concerned.

Now I just need to grab a Hillcrest tracker, whip up the FreePIE integration code, and we're all set for 2 games once the Rift bundle comes out.

Post by **CyberVillain** » Wed Jun 20, 2012 2:54 am

Emerson wrote:One quick fix that I found--I was using point sampling on the intermediary texture, not lerping--it helps a bit:

Hmm, too me it looks like only the "frame" thats warped?
Look at the ATM machine in that pic, it's lines are still straight?

Post by **Emerson** » Wed Jun 20, 2012 9:13 am

Nah, it's just subtle (moreso because the area of highest distortion is inside the black-bar area in ME).

Same area, just with the camera pointed further down:

Post by **brantlew** » Wed Jun 20, 2012 9:21 am

@Emerson: So are you done with this mod? (beyond testing and verification on an actual Rift) I don't know how much time you put in, but it seemed really fast. Amazing work! Do you have plans for conversion of any other titles?

Post by **Alkapwn** » Wed Jun 20, 2012 9:56 am

That looks awesome Emerson! I just tried the cross-eye test, holding face about 1 foot away from the monitor and

this is going to be unREAL! So pumped!

Post by **Emerson** » Wed Jun 20, 2012 10:10 am

brantlew wrote:@Emerson: So are you done with this mod? (beyond testing and verification on an actual Rift) I don't know how much time you put in, but it seemed really fast. Amazing work! Do you have plans for conversion of any other titles?

I think I'm gonna call this roughly feature complete for the first round--I'll probably try to spend the rest of the week cleaning up some of the more dodgy stuff I did in the code, and maybe throw it against some other games to see how much work it would be to support a few more. Hopefully I can get it out and let other people play with it this upcoming weekend.

Once that's out of the way I'll start poking around with additional functionality (maybe the screen-resolution-adaptor behavior mentioned earlier in the thread, or see if I can create a usable roll-axis).

Post by **brantlew** » Wed Jun 20, 2012 11:09 am

Sounds great.

One problem I see with roll is that there is no native command interface for it. Yaw and Pitch can be mouse emulated, but if you provide roll you will also need to provide a method for controlling it. If you are just supporting the Rift kit then you could poll the Hillcrest tracker directly, but I would suggest providing a generic interface (socket/pipe/Window Messaging,etc..) so that other trackers could be attached to it. I plan on putting Hillcrest support into FreePIE in the next few weeks so in theory we could come up with a scheme to drive the "roll interface" via FreePIE and support a wide range of trackers including the Rift.

Post by **CyberVillain** » Wed Jun 20, 2012 3:35 pm

Emerson wrote:Nah, it's just subtle (moreso because the area of highest distortion is inside the black-bar area in ME).

Same area, just with the camera pointed further down:

Yeah, clearly visible there, cool!

Post by **mm0zct** » Thu Jun 21, 2012 4:05 pm

I've noticed so far that the images are rendered at traditional 640x800 resolution then distorted, with the widest point aligned to the narrowest part of the Rift FOV, leaving the large black borders.

Perhaps we should be rendering to a larger FOV at higher resolution and crop some image off the top and sides but fill in more of the black area? Given the whole point in the Rift is the large FOV I don't think those corner pixels should go to waste (although I understand some will be physically obscured to improve the immersion).

Post by **cybereality** » Thu Jun 21, 2012 9:32 pm

Know I've been kind of quiet lately, but I've still be hacking away at the driver. Now I have fixed the camera separation controls (previously it was skewing the view at a strange angle) and I have also got convergence working. Will probably add the side-by-side viewing over the weekend and finally get to try this out with my HMZ-T1. I'm excited.

[youtube]http://www.youtube.com/watch?v=HEtJ31fJlcc[/youtube]

Post by **brantlew** » Thu Jun 21, 2012 10:26 pm

Rockin! Almost there. We may have several Rift-ready titles by August.

Post by **Nick3DvB** » Thu Jun 21, 2012 10:53 pm

Nice work Cyber.

Could the camera control code work with other Source games, HL2 etc?

Post by **WiredEarp** » Fri Jun 22, 2012 12:49 am

Sounds great CyberReality! Exciting to think we will actually be able to use our Rifts for more than one game when we get them!

Projects Backed · Post by **LeeN** » Fri Jun 22, 2012 11:10 am

This is excellent Cyber! ( why do we have so many separate DIY forums: p)

I found an interesting hack that some one did. They modify Diablo 3's camera directly in memory to zoom and rotate. It's an interesting way to solve this since, besides not disrupting shaders, it can potentially take advantage of frustrum culling, allowing for head tracking. Of course it would be difficult to render twice.

http://www.youtube.com/watch?v=0J22sIG- ... ata_player

Post by **Emerson** » Fri Jun 22, 2012 12:30 pm

While I think the direct memory editing might end up being the best looking solution for rigging direct linkage of camera and a 6DOF tracker, due to the above mentioned culling issues, I also think it's likely to be the most fragile. The memory locations for this sort of thing are going to be game specific, and probably even different patch-to-patch on the same game.

I'm hoping my DirectX intercept solution will generalize to a wider variety of games with only minor tweaks necessary. I've been trying to get it to work with something other than Mirror's Edge, and there's certainly differences from engine to engine, but I'm making progress on a solution that should adapt well. (Even commercial offerings like IZ3D and DDD need game-specific profiles, though, and I'd expect the same here.)

Ideally, one could use both--the DirectX intercepts would render whatever camera views actually come down in 3D, and the direct memory editing would make sure that that camera is tracking the HMD view accurately.

Post by **Chriky** » Fri Jun 22, 2012 12:37 pm

Why is it so hard to do roll? Surely if you have access to the matrix you can do any transformation?

Post by **Emerson** » Fri Jun 22, 2012 1:42 pm

Roll is feasible because it mostly doesn't expose any new geometry that you weren't already seeing--just rotating the existing stuff (there's a little new geometry from circling the square, so to speak). That still leaves all the head-translations, too, though, which might run into culling problems (most games will dump a significant portion of their geometry, as much as they can, before even talking to DirectX at all).

From an optimal experience perspective, it'd be best to be able to manipulate the camera data structures directly, because you could skip the sensor->mouse emulation->direct input->game layers and dive straight into the rendering loop, sensor orientation->camera orientation, minimizing the latency and accuracy drift.

If I recall, Carmack's video showed him doing just that, for D3BFG--though it's of course easier when you have the source code for your game.

3DOF mouse-emulation x/y and directx-injected z would probably be a good first effort, though.

Post by **brantlew** » Fri Jun 22, 2012 2:10 pm

Direct memory injected roll is the only way you could implement it because there is no I/O device support for roll. It would be handy though to have the tracker poll be generic instead of tied to a specific device. Instead of polling the Hillcrest unit directly using their API, you could poll a pipe/socket/etc. The mouse emulation utility would then be put in charge of yaw, pitch, and roll which can be advantageous (code-wise), and also allow a variety of tracking devices to be utilized.

Post by **rajveer** » Fri Jun 22, 2012 2:22 pm

I thought Emerson is intercepting the render matrix so he can apply any transformation to it he wants, including roll, the game doesn't need an interface for this. It will go sensor->emerson's software->directx matrix transformation, or whatever. The only reason why modifying the camera in memory would be better is so that there isn't any culled geometry, but with roll it probably won't be much of an issue as you're still looking in the same direction (so the orientation of the frustum along the z-axis that the game uses to cull geometry won't be the same as the camera's orientation, but it's still looking along the same z-axis, so most geometry that needs to be shown will probably be shown).

Post by **Emerson** » Fri Jun 22, 2012 4:30 pm

rajveer wrote:I thought Emerson is intercepting the render matrix so he can apply any transformation to it he wants, including roll, the game doesn't need an interface for this. It will go sensor->emerson's software->directx matrix transformation, or whatever. The only reason why modifying the camera in memory would be better is so that there isn't any culled geometry, but with roll it probably won't be much of an issue as you're still looking in the same direction (so the orientation of the frustum along the z-axis that the game uses to cull geometry won't be the same as the camera's orientation, but it's still looking along the same z-axis, so most geometry that needs to be shown will probably be shown).

That's basically accurate. Modifying the camera in memory might lower the overall latency, if the rendering loop runs at a faster rate than the game's input polling.

Post by **cybereality** » Fri Jun 22, 2012 6:12 pm

Roll is perfectly possible, in fact I just added it real quick to my driver right now:

[youtube]http://www.youtube.com/watch?v=krZ6JsoPGmk[/youtube]

Post by **Chriky** » Sat Jun 23, 2012 5:30 am

Awesome

Do you know if you're going to get a RIFT slightly early Cyber? Or maybe get another go with the PR4? Would be a big win to get L4D2 RIFTracking at launch. Looks like you basically just need to get the warping nailed and the double rendering, then its just cleaning up your code and making a nice interface for different trackers. Great stuff.

BTW if you set the game to 4:3 aspect you should cut down on seeing culled stuff when you roll. TBH though nobody is going to roll more than about 15deg each way.

Post by **Emerson** » Sat Jun 23, 2012 11:21 am

I'd be worried about running my driver against a Source game since I've made absolutely no effort to dodge VAC...

Post by **cybereality** » Sat Jun 23, 2012 11:28 am

@Chriky: Not sure if I can get a pre-release Rift, but I will try to have Palmer test the driver as soon as I get something working good. Even if I get the kit the same time as everyone else, it would probably only add another week or two to get it optimized for the Rift and the tracker it comes with.

@Emerson: It should be fine as long as you don't play online.

Post by **Emerson** » Sat Jun 23, 2012 11:36 am

Been working on compatibility a bit, the past few days. Made some progress!

[youtube]http://www.youtube.com/watch?v=TjV0jLVC9MU[/youtube]

Post by **cybereality** » Sat Jun 23, 2012 11:41 am

Awesome work Emerson!

Post by **brantlew** » Sat Jun 23, 2012 2:18 pm

Hell yeah - this is the title I want the most! You the man Emerson.

Post by **Nick3DvB** » Sat Jun 23, 2012 2:59 pm

Very cool, almost the full 9 yards now, but someones got to say it... "Can it run Crysis?"

The stereoscopic stuff is going to be problematic, but I'd love to try the Wiisis tracking mod with he Rift.

Any takers?

Post by **rajveer** » Sat Jun 23, 2012 4:38 pm

Nick3DvB wrote:Very cool, almost the full 9 yards now, but someones got to say it... "Can it run Crysis?"

I hope so, I'd love to use this with Mechwarrior Living Legends!

Post by **cybereality** » Sat Jun 23, 2012 6:51 pm

I've gotten side-by-side mode mostly working. The main issue now is crashing sometimes when the game starts. First it was working only in windowed mode, now it only works in full-screen (though I didn't change any code). Not sure what the issue is, have to investigate further. But if you get in the game its fine. Also there are some small glitches with things like fire, but I'm sure I can fix it.

L4D_Cyber3D.jpg

Post by **Emerson** » Sat Jun 23, 2012 6:52 pm

Very cool!

Post by **cybereality** » Sat Jun 23, 2012 7:33 pm

[youtube]http://www.youtube.com/watch?v=Ovf5TLiIfZ8[/youtube]

Watch on YouTube for the 3D options.

Post by **brantlew** » Sat Jun 23, 2012 8:25 pm

@cyber: Great work! Really impressed with the speed that you and Emerson were able to get to this point. At this rate we may have half a dozen Rift-ready games on release. Who needs Tridef?

(FYI: I ordered a Hillcrest tracker last Thursday and I hope to get support for it working within a week or two from now)

The Roll is also a very exciting prospect. If you have that much control I assume it might be possible to do variable height and some sort of head-translation as well. Head translation is a little weird though because essentially it's the same vectors as walking - just limited to a few centimeters around the primary "body" location. So for instance, something like peeking around a corner could be handled both by a direct memory translation matrix or by a mouse emulated strafe. I guess the difference would be that the translation matrix approach would not change the character position and bounding box in the simulation. Also - a lateral head motion on a "rolled" head would be a different motion. True "neck simulation" would certainly need to be handled carefully.

Post by **Emerson** » Sat Jun 23, 2012 9:49 pm

I'm sick of being super-paranoid about breaking my code and flying-by-the-seat-of-my-pants, so I started a github repo and tossed up what I have, so far.

I started a new thread as to not spam this one with people trying it out and finding out how buggy a v0.1 release can be.

http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=15086

Post by **PalmerTech** » Sat Jun 23, 2012 10:02 pm

Wow! I get busy for a few days working on business stuff and I come back to this, crazy! I have a small order of parts on the way so I can make some pre-release units to get into the hands of people who need them.

Cyber, that is great that you have roll working. It will make a huge difference compared to just standard mouselook! Doing proper head translation really needs something like magnetic or optical tracking, but looking around and roll is as good as you can easily get on the Hillcrest trackers.

Emerson, your warping looks great. I just tried your video out on the test unit I have here, it looks good.

On a related note, TriDef is not interested in working with Oculus at this point. A bit sad, but oh well. With the way things are going, it looks like the Rift will have two ways to support games:

1) Rift Native games, using an SDK provided by Oculus to have full support built in
2) Standard PC games using a third party community or commercial driver

With proper support, things can go beyond just FPS games. Imagine an RTS where you can look down at the world as if you really are god sized, easily looking off into the distance to see enemy units!

EDIT: Missed Brantlew's post. The distortion is indeed symmetric for each lens, so Carmack's screenshot is not perfect. I will try to get a good grid, though it does not matter all that much until the Rift is closer to being sent out.

Post by **Emerson** » Sat Jun 23, 2012 10:15 pm

Good to hear the warping is close!

I'm hoping with this much free information flowing on how to do 3D game adaptations, we can make it easy to support all sorts of games going forward.

Post by **brantlew** » Sat Jun 23, 2012 10:31 pm

PalmerTech wrote:TriDef is not interested in working with Oculus at this point.

Bad move on their part. Now with Emerson essentially open-sourcing the stereoscopic hacking technique, people are just going to go ahead and work stereoscopic Rift support for all the popular games. Turn off warping and you have a free version of Tridef so everybody that wants 3D (not just Rift users) will use it. They just lost a bunch of sales.

PalmerTech wrote:the Rift will have two ways to support games:

1) Rift Native games, using an SDK provided by Oculus to have full support built in

Whoa - back up! Where did that come from? What are the capabilities of this SDK? Warping transform, Hillcrest tracker access, external control of viewpoint ??? And you're no programmer so this must be Carmack libs.

Post by **PalmerTech** » Sun Jun 24, 2012 12:07 am

I cannot spill details at this moment, but I can tell you that it not just Carmack. I will get to work with some people who have a LOT of experience with SDKs/middleware, can't name them till everything is nailed down though.

Warping, per-user adjustments for things like IPD and relief, viewpoint and weapon hooks for trackers, and maybe some other fun stuff. It will probably be months before it is ready to show around to game devs, and probably months more before support starts going into games on any kind of scale, so driver hacks for existing games have a ton of value.

Really wish I could say more, I promise that I will as soon as I can.