Head Tracking/Gun controller help and suggestions required!!

Post by **brantlew** » Mon Feb 27, 2012 1:04 pm

Yes, but many games support thumbsticks on XBox360 controller - don't they?

This is what I am currently working on with FriiSpace. Translating real world motion into emulated 360 thumbsticks in SkyRim.

Post by **GSG9LIGHTNING** » Mon Feb 27, 2012 4:45 pm

Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try

)
http://www.youtube.com/watch?v=9wXx3vMy_AQ

Post by **ERP** » Mon Feb 27, 2012 5:37 pm

cybereality wrote:Well right now I am just planning on getting the camera orientation via DirectX. We can probably assume most every game uses the same up vector, so this should be helpful in re-centering the pitch. In terms of knowing which way is forward (for yaw), this may be harder (and/or impossible) but I think it could be done with some work. And this would most certainly be considered a hack, and get you banned online. I have no doubts about that. But I mostly play single-player so I don't mind.

Getting the camera matrix out of directX is not simple, there is no real concept of camera matrix, it's just an input to a shader, and worse several can be used in a single frames worth of rendering. I'm actually not sure how NVidia deals with this in the 3D Drivers, best guess is they identify the matrix based on the contents.
It would probably be doable on a game by game basis, by doing a shader capture (like Helix' shader mod for 3DVision) and figuring out which input it is for a specific commonly used game shader.
Doing it automatically the only way I can see it looking at the contents of the matrix, and identifying the usual camera matrix pattern, but your back to trying to determine if it's the camera for the main view or an offscreen surface, NVidia doesn't have to worry about this because they use other mechanisms to determine if they need two cameras or not.

Everything falls out of the camera matrix if you have it.

Post by **WiredEarp** » Mon Feb 27, 2012 7:19 pm

Its been an extremely long time since I did any Direct3D stuff, but surely this is just an intercept to SetTransform or the modern equivalent when its doing the view transform. Since you pump that all your transforms, you should be able to intercept the one you want.

It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

Post by **cybereality** » Mon Feb 27, 2012 8:02 pm

To be honest, I am still somewhat of a noob when it comes to DirectX. Much of the 3D work I've done up until this point has been with higher-level engines (like OGRE) which hide much of the nitty-gritty details. From what I get so far, there are certain functions that will always be called around the place where the rendering happens. For example "BeginScene()" "EndScene()" and "SetTransform()" in particular. So I think I am on the right track, but its probably not as easy as it looks. Though surely if people like iz3D or DDD can have a somewhat generic solution it must not be impossible.

Post by **brantlew** » Mon Feb 27, 2012 8:03 pm

It seems like if you used some type of hooking or DLL replacement, you could intercept the SetTransform call. The first argument is an enum and you could just check for D3DTS_VIEW (0x2).

Post by **pierreye** » Mon Feb 27, 2012 11:06 pm

GSG9LIGHTNING wrote:Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try )
http://www.youtube.com/watch?v=9wXx3vMy_AQ

ARMA 2 is using TrackIR for headtracking. Not sure if GlovePIE TrackIR support ARMA 2 as few years back, TrackIR encrypt their API and interface so most third party solution doesn't work.

Post by **CyberVillain** » Tue Feb 28, 2012 1:40 am

pierreye wrote:
GSG9LIGHTNING wrote:Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try )
http://www.youtube.com/watch?v=9wXx3vMy_AQ
ARMA 2 is using TrackIR for headtracking. Not sure if GlovePIE TrackIR support ARMA 2 as few years back, TrackIR encrypt their API and interface so most third party solution doesn't work.

*Commercial begin* It supports FreeTrack too, which FreePIE supports *Commercial ends*

Post by **ERP** » Tue Feb 28, 2012 12:25 pm

WiredEarp wrote:Its been an extremely long time since I did any Direct3D stuff, but surely this is just an intercept to SetTransform or the modern equivalent when its doing the view transform. Since you pump that all your transforms, you should be able to intercept the one you want.

It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

The problem is in DX9+ there is no concept of a model/view transform, it's just a matrix input to a shader, there is no context. It's not like the old fixed function stuff where you set the transforms explicitly.
You'd have to look for things that look like camera matrices being passed to shaders, that's not particularly hard, they have a very specific pattern of entries, but you have no idea if it's just the camera transform (or if some of all of the model transform is included), or if it's the camera transform for a shadow map rather than for the primary render target.
Though I guess you can track which render target is selected for the latter.
My guess is you could get it to work for a lot of games pretty quickly, but you'll have outliers that are harder to get working.

Post by **ERP** » Tue Feb 28, 2012 12:28 pm

WiredEarp wrote: It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

Sorry I missed this, I think the reason NVidia and DDD can do it is the modification doesn't require they isolate the camera transform explicitly, they just have to produce the same result as adding an additional translation into the final matrix.

Post by **brantlew** » Tue Feb 28, 2012 9:11 pm

@ERP: Sure once the matrices all get multiplied together in the render-pipeline it would be difficult to disentangle them reliably. But what if you trapped the view matrix way back up at the initial input function? (ie. SetTransform(D3DTS_VIEW, ...) ) I guess some games might pre-compute the combined matrix and input it all at once, but surely lots of games input the view matrix independently using this method. What am I missing here?

PS. My graphics is rusty and mostly OpenGL based.

Post by **WiredEarp** » Tue Feb 28, 2012 9:28 pm

I don't know enough to comment here, my DX experience ended with V6 I think.

At a glance tho Brantlew's concept seems workable...

Post by **pierreye** » Mon Mar 19, 2012 5:36 pm

Here's another VR demo for Mass Effect 3. This clip is longer around 12 mins of unedited gameplay to prove that the whole VR setup is playable.

[youtube-hd]http://www.youtube.com/watch?v=lMIWAV5h4EI[/youtube-hd]

Post by **brantlew** » Mon Mar 19, 2012 8:08 pm

Cool video!

Is it odd playing in 3rd person view like that?

Post by **cybereality** » Mon Mar 19, 2012 8:24 pm

Sick man! Very nice.

Post by **pierreye** » Mon Mar 19, 2012 9:04 pm

brantlew wrote:Cool video!

Is it odd playing in 3rd person view like that?

It's playable once you adapt. The view is much better than DeadSpace 2 as when you turn, the character will turn together so the back is always facing you. DeadSpace 2 when you turn, your view actually shift to the front of the character. I can aim and shoot better than using a mouse/keyboard combo as no longer need to lift the mouse when doing a quick turn and fine aiming feel much more natural.

Post by **CyberVillain** » Tue Mar 20, 2012 1:51 am

Nice!

I tried Mass effect 3, but my eyes got tired of all the halo effects around the characters, to bad ME3 is not capable of correct stereo 3d :/

Post by **pierreye** » Tue Mar 20, 2012 4:39 am

Try out Helix DirectX 9 3D Vision fix.
http://helixmod.wikispot.org/gamelist

Out of topic, ancjob try to spam my youtube video, damn arsehole.

Post by **CyberVillain** » Tue Mar 20, 2012 5:37 am

pierreye wrote:Try out Helix DirectX 9 3D Vision fix.
http://helixmod.wikispot.org/gamelist

Out of topic, ancjob try to spam my youtube video, damn arsehole.

Will do, thanks! Is ME3 a DX9 game? Thought most games where 10 - 11 these days

about ancjob, unemployed people with too much free time are dangerousness...

Post by **pierreye** » Tue Mar 20, 2012 6:51 am

You will be surprise most game still using DirectX 9. ME3 and Skyrim both are directx 9.

Post by **CyberVillain** » Tue Mar 20, 2012 7:02 am

pierreye wrote:You will be surprise most game still using DirectX 9. ME3 and Skyrim both are directx 9.

Its because of all the f*ing console ports :/

Post by **pierreye** » Wed Mar 21, 2012 6:08 am

Try out Battlefield 3 and the 3D is good. With the config.ini tweaking the depth and convergence, the 3D had lot of pop and depth. The headtracking in the Jet fighter is just damn realistic. Only hate the dirty lens effect.

Post by **CyberVillain** » Wed Mar 21, 2012 6:24 am

pierreye wrote:Try out Battlefield 3 and the 3D is good. With the config.ini tweaking the depth and convergence, the 3D had lot of pop and depth. The headtracking in the Jet fighter is just damn realistic. Only hate the dirty lens effect.

Yeah, its really nice, really love how you cna make bf3 take away the stereo 3d when aiming down the scope... I hate it that they didnt implement proper freelook.. That you cant stear the helcopter while freelooking...

Post by **CyberVillain** » Wed Mar 21, 2012 12:15 pm

btw, try Alan Wake, its really nice after the latest patch

Post by **space123321** » Thu Mar 29, 2012 7:16 pm

Can you guys suggest best 3d games that can also incorporate mouse view tracking via the wiiremote on the HMZ.

Obviously the FPS type games - COD, Skyrim, MAss Effect, etc that I see in with Pierreye's setup however wondering if there are some good racing, sims, etc that can also use the HMZ/wiimote setup. Thanks!

Post by **WiredEarp** » Thu Mar 29, 2012 8:29 pm

FSX and Black Shark are two of my faves if you are into sims...

Post by **space123321** » Thu Mar 29, 2012 8:52 pm

Ah yes - I do have FSX - I shoudl reinstall that for the HMZ

I do remember that you had to hold down the space bar to activate mouse look... is there away around this? I do recall when I used the Vuzix glasses the driver automatically set this...

Post by **WiredEarp** » Thu Mar 29, 2012 9:12 pm

Hmm, not sure, I vaguely remember coming across that issue, but theres an option or something like that that lets you turn it on permanently. Thats what I have done. Can't remember what, but if you google it you'll find the answer (probably modifying a .ini file).

Post by **pierreye** » Fri Mar 30, 2012 12:05 am

Going to try recently release Wing Commander Saga developed by fans using Open Freespace engine. Too bad it is in OpenGL so no 3D support. FS2Open should support headtracking through TrackIR API. I love Wing Commander and FreeSpace series.

Post by **CyberVillain** » Fri Mar 30, 2012 1:14 am

Grid race driver and latest Dirt both have TrackIR support

Post by **PalmerTech** » Thu Apr 05, 2012 1:58 pm

Wanted to let you guys know that the motion tracking components in the Razer Hydra can be fit into a very, very small tracking cube, about 10x15x20mm perhaps? I will take pictures tomorrow, I have a cable coming out of the right hand controller (Which now has no tracking) to a small box containing the tracking components. Mounts to an HMD perfectly!

Here is a picture (Borrowed from WiredEarp) that I put lines on showing where I made cuts:

All three coils share a common ground, and each also has a positive line, for a total of just 4 wires that need to be relocated. Soldering to the tracker side is very easy, but be careful on the main circuit board end! You have to scrape the enamel off the traces and solder directly to that, and the traces are very thin and easy to lift off the board. I accidentally did that with one of my connections, but I was able to wire directly to where the trace led on the board, phew! My advice would be to hot-glue the wires to the back of the board for strain relief, and then solder the connections.

Tracking still works flawlessly, even extended with 4 feet of wire! It might just be the placebo effect, but it seems like it might work even better than before.

Post by **WiredEarp** » Thu Apr 05, 2012 7:33 pm

Hey PalmerTech, What did you do about the small 'daughterboard' that contains the buttons, triggers etc? Just leave that connected?
Be extremely careful with the cable that connects that to the main board, it has failed in mine now with all the moving around when testing, and I haven't been able to figure out a replacement yet (way too tiny for me to solder I think). If you are leaving it in the existing Hydra shell that is probably a good idea to protect it from movement.

Glad to hear you can reduce the size further doing that! It looked possible (as did merely bending the board so it reads 'flatter', as mine had to be mounted on a slight upwards angle to read 'flat') but I was afraid it would be timing the signals from those coils and it could throw things off. Great to hear it doesn't. I'll try your technique here when I get my Hydra going again. It already makes an awesome head tracker, but reducing it like you have mentioned will mean it will fit on my P5 glove much better.

Only issue with using the Hydra is you need the buttons etc to initialize it. I REALLY wish Sixense would just release a version of the Hydra tracker designed for tracking specifically.

Post by **PalmerTech** » Thu Apr 05, 2012 8:21 pm

I left all the other components in the controller. I can either build it into my motion tracked gun to use as extra buttons and analog stick, or just put it in the backpack. I think this is the best solution, since it leaves the controller fully usable aside from motion tracking. Next, I want to put some kind of connector on the module. That way, I can either use it for headtracking with an extension cable, or plug it right into the end of the Hydra for "normal" use.

Post by **WiredEarp** » Fri Apr 06, 2012 12:50 am

Great idea. Should be easy to use any sort of micro connector to do that.

Where would you put the module bit when you try to use it like a standard Hydra again? Would you just cut out a section of the Hydra controller handle so you can stick it in when you need it, or mount it some other way?

Post by **fireslayer26** » Wed Apr 18, 2012 12:31 pm

@Pierreye Have you heard anything from this guy who claims to have got the Move to work fully work without the wiimote? He was also suppose to post an executable for it.: [youtube]http://www.youtube.com/watch?v=X7ZFTO6H-GA[/youtube]

Post by **pierreye** » Wed Apr 18, 2012 8:56 pm

Not yet. He previously mention that MoveOnPC is conflicting with MotionInJoy driver so I'm not sure how he hook PS Move and PS Navigator to the PC. I'll check with him.

Post by **fireslayer26** » Thu Apr 19, 2012 8:55 am

Ok great. Looking forward to what you find out.

Post by **pierreye** » Fri Apr 20, 2012 9:47 pm

Managed to tie the wiimote to the leg as detection for prone/jump and it works. I'll shoot a battlefield 3 demo soon with headtracking, gun tracking and leg motion detection. It's also possible to disable movement if you don't move your leg.

Post by **fireslayer26** » Wed Jun 06, 2012 7:29 pm

Did you ever get to make this demo? Get the setup to work as he did without the wiimote?

Post by **pedroagodoy** » Thu Jul 12, 2012 5:42 am

Hello guys!!!! My name is Pedro, I´m from Brazil!!! Work in game industry as level designer and 3d modeler!!!
I´ve been following this topic for some time and i have to say its great!!!!

I dream about this for at least 8 years!!!! But here in Brazil its so dificult to get an HMD (too much expensive!!! eaven the simple models!!!) I started to build my on setup months ago. I´m not really gonna use this for games but mostly for realtime archviz (games are a bonnus!!!!

) I´m making my house in UDK and I want to walk in it before I start building.

This is what I did already.

Manage to get wiimote + motion plus and ps3 navigation working great with pierre scripts in Glovepie + MotionnJoy and 2 bluetooth adapters.
Vuzix Wrap 920VR (just got mine and I´m really enjoying it) using mouse emulation app from cybereality.

The problem is that can't get then working together!!!! Carl is still banned from Vuzix!!!! Does it work with older version of glovepie???? 0.3???

@Cybereality how did you get it working????? You wrote other app for this???? Can you please upload it????

And the 3D effects work with nvidia stereo drivers???? I only get the anaglyf mode working the others uses nvidia glasses right?

Thank you all!!!!! Keep up the good work!!! If you guys need anything that I can do i´ll be happy to help.