Head Tracking/Gun controller help and suggestions required!!

[FingerFlinger]

I'm looking forward to your new video.

Yes, I would think Skyrim is ideal for a wearable computing situation, since 'aiming' with your head feels natural with melee weapons.

Another big project that I'm looking at is a voice interface for game menus and dialogue. A voice interface removes the need to have an analog stick on hand, or to use your gun as a pointing device. I really feel like it would help the illusion if you always treat your gun as a gun, instead of "a gun, but sometimes a mouse". Maybe it's not that big of a deal, but if I can find a good voice recognition library, I think it would be worth trying.

OPENNI looks very cool. I wonder if it is precise enough for self-skinning. Meaning, if you look at your hand through the HMD, and it's wearing a leather gauntlet and tracking your actual movement.

[donkaradiablo]

Anybody willing to shed some light on the use of Logitech Mx Air mouse vs other alternatives for head tracking in FPS gaming? I'm thinking no need for other layers of software or emulation would make it a better candidate for the job but I have no first hand experience so I might be missing something. And the on-mouse "back" button that can be used to center view without any need for such an ingame keybinding should be a plus.

anyone?

[pierreye]

Here's 3rd person shooter VR using Dead Space 2 as example. This setup include head tracker using Motion Plus on the head. I need to re-shoot Skyrim video due to some problem to the picture quality.

http://www.youtube.com/watch?v=moYHZOhXTfk

[youtube-hd]http://www.youtube.com/watch?v=moYHZOhXTfk[/youtube-hd]

[cybereality]

Very cool.

[pierreye]

Here's Skyrim VR. A great game to try out in VR.

[youtube-hd]http://www.youtube.com/watch?v=jBP0q_kNdMY[/youtube-hd]

[CyberVillain]

Nice!, you can be a test case when FreePIE supports WiiMote

[pierreye]

Sure. No problem to be on the software testing team. Can't contribute much on programming nowadays as I stop for few years. Now only work as Project Manager getting specs, draft out the framework design and ensure my team deliver the application.

[fireslayer26]

Great work Pierreye!

[cybereality]

Cool.

[pierreye]

Pretty much any first person view/third person view games can be turn into a VR gaming with some scripting. Now only hope that developer would start to support dual mouse input, one for view and another for aiming cursor. This would push VR into next level.

[WiredEarp]

Now only hope that developer would start to support dual mouse input, one for view and another for aiming cursor.

I'd rather hope that they start to support dual input of multiple types, not just dual mouse input. Tying it to mice would mean its not absolute, and thats where its at for real VR imho.

[pierreye]

That's true but the problem is no standard input device for absolute view which make it difficult to support.

I had seen a demo on HL2 mod that support mouse aiming crosshair and independent view from sixense.

[CyberVillain]

Joysticks are absolute, the problem is that most games use them in a relative manner

[cybereality]

The main issue is that developers always want to support the bottom-line, and for 99% of customers that means a mouse and keyboard. Even when better options exist (ie Razer Hydra) there usually isn't enough of a critical mass to justify support. Even though adding support could take only a couple of days (or less) developers don't seem to think its worth it. Not sure that will change any time soon unless there was a massively popular VR peripheral for the PC.

[WiredEarp]

I dont think most joystick ranges have sufficient precision, compared to things like TrackIR etc. Most potentiometer ones only have 1024 resolution, so even if you wrapped the coordinates into them, the accuracy would not be as great. Of course, if we are mapping into a virtual joystick, we could possibly increase the precision to any level.

I'm thankful that most games use joysticks relatively, don't know if you remember the hell of Apple 2 joysticks when heaps of game developers wrote them using absolute addressing instead of relative.
Having to put the joystick 1/4 of the way so my spaceship is 1/4 of the way across the screen? lame!!!

I think half the problem is the lack of an easy interface. Sure, we have VRPN, but its not well known or used. What is really needed is just a very simple generic input interface, that game devs can read, and use in their games. Then, other systems can just plug into this. I guess FreePIE could be a contender here - what is needed is NOT 'compatibility with TrackIR, Hydra, etc' but just 'generic input' compatibility, so that game devs can just read the interface, and modify their view/aiming appropriately.

[CyberVillain]

I just stated that there already is a general software interface that is absolute... DirectX has no limitations when it comes to precision... Its up to the hardware, or emulation in this case.... The problem is still that game developers use Joysticks in a relative manner, this is sad otherwise you could use ppJoy as a bridge between lets say FreeTrack and any game....

[WiredEarp]

Ok, so even if the precision isn't a problem, and we can use virtual joysticks with multiple axis to represent the different points, a standard for which axis represents what and what ranges are allowed, etc, would still be required (or at least, be very handy). This would allow configuration to be set once for each device, and after that, everything would work smoothly, with every game.

[CyberVillain]

Well ranges are not a problem, DX has range from -1 to 1 so your FreePIE or GlovePIE script just has to convert 360 degrees to that range, but, since the game takes that value and uses it reletive there is no point :/

The games does x += value instead of doing x = value

[cybereality]

If you could somehow "know" what the angle is in the game, then I think you could hack it so the joystick was absolute without changing anything in the game. This is one of the things I hope to accomplish with my DirectX driver (if I ever pull it off).

[CyberVillain]

Some kind of memory hack? Then you need to know the address for each game :/
Are going to write directly to the memory or just read the value and set it using conventional joystick / mouse emulating?

If your doing the first changes are that online games will ban you with punkbuster

It would be really cool if you pull this off

[cybereality]

Well right now I am just planning on getting the camera orientation via DirectX. We can probably assume most every game uses the same up vector, so this should be helpful in re-centering the pitch. In terms of knowing which way is forward (for yaw), this may be harder (and/or impossible) but I think it could be done with some work. And this would most certainly be considered a hack, and get you banned online. I have no doubts about that. But I mostly play single-player so I don't mind.

[pierreye]

Another issue to solve is when you hit the limit of the pitch, you can keep on increasing DirectInput Y value but the coordinate just doesn't move. This is where I hit the hurdle for auto centering script.

[brantlew]

If you could somehow "know" what the angle is in the game, then I think you could hack it so the joystick was absolute without changing anything in the game. This is one of the things I hope to accomplish with my DirectX driver (if I ever pull it off).

Sounds interesting but tricky. You would need to determine on a per-game basis the scaling between joystick ticks and turning speed. Once you have that rate you could sort of hack absolute motion by simulating a joystick push and release. For example, if your absolute yaw went from 0 degrees to 45 degrees in 100 milliseconds you would have to simulate a joystick push of a magnitude that caused a 450 degree per second turn and then recenter the simulated joystick after 100 ms.

A couple things that could be problematic. There may be a limit to the turning speed supported in the game in which case some motions might feel a bit laggy. Also I wonder how the joystick polling rate would tie in. It might be difficult to pin down an exact joystick turning rate if it is dependent on the frame rate. Different areas of the game might have different turning behavior.

Still, it would be an interesting exercise. For yaw, I don't think it would matter if your calculations were off. So what if your real world yaw is out of sync with your in-game yaw? Pitch errors would be noticeable though.

[PalmerTech]

Yeah, as long as the yaw matches up to what you are seeing during the turn, it does not matter if it eventually drifts off a bit.

[CyberVillain]

I dont think you can use Joysticks in most games today Brantlew, the game implementation is just too slow.. You'll have to use the mouse :/

I wished all games had a switch for relative or absolute joystick use :/

[brantlew]

Yes, but many games support thumbsticks on XBox360 controller - don't they?

This is what I am currently working on with FriiSpace. Translating real world motion into emulated 360 thumbsticks in SkyRim.

[GSG9LIGHTNING]

Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try )
http://www.youtube.com/watch?v=9wXx3vMy_AQ

[ERP]

Well right now I am just planning on getting the camera orientation via DirectX. We can probably assume most every game uses the same up vector, so this should be helpful in re-centering the pitch. In terms of knowing which way is forward (for yaw), this may be harder (and/or impossible) but I think it could be done with some work. And this would most certainly be considered a hack, and get you banned online. I have no doubts about that. But I mostly play single-player so I don't mind.

Getting the camera matrix out of directX is not simple, there is no real concept of camera matrix, it's just an input to a shader, and worse several can be used in a single frames worth of rendering. I'm actually not sure how NVidia deals with this in the 3D Drivers, best guess is they identify the matrix based on the contents.
It would probably be doable on a game by game basis, by doing a shader capture (like Helix' shader mod for 3DVision) and figuring out which input it is for a specific commonly used game shader.
Doing it automatically the only way I can see it looking at the contents of the matrix, and identifying the usual camera matrix pattern, but your back to trying to determine if it's the camera for the main view or an offscreen surface, NVidia doesn't have to worry about this because they use other mechanisms to determine if they need two cameras or not.

Everything falls out of the camera matrix if you have it.

[WiredEarp]

Its been an extremely long time since I did any Direct3D stuff, but surely this is just an intercept to SetTransform or the modern equivalent when its doing the view transform. Since you pump that all your transforms, you should be able to intercept the one you want.

It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

[cybereality]

To be honest, I am still somewhat of a noob when it comes to DirectX. Much of the 3D work I've done up until this point has been with higher-level engines (like OGRE) which hide much of the nitty-gritty details. From what I get so far, there are certain functions that will always be called around the place where the rendering happens. For example "BeginScene()" "EndScene()" and "SetTransform()" in particular. So I think I am on the right track, but its probably not as easy as it looks. Though surely if people like iz3D or DDD can have a somewhat generic solution it must not be impossible.

[brantlew]

It seems like if you used some type of hooking or DLL replacement, you could intercept the SetTransform call. The first argument is an enum and you could just check for D3DTS_VIEW (0x2).

[pierreye]

Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try )
http://www.youtube.com/watch?v=9wXx3vMy_AQ

ARMA 2 is using TrackIR for headtracking. Not sure if GlovePIE TrackIR support ARMA 2 as few years back, TrackIR encrypt their API and interface so most third party solution doesn't work.

[CyberVillain]

Awsome work Pierreye !
An impressive HMD Setup you got there !

Thats incredible ! Must be totally immersive for you to play FPS Games/Simulations etc.
Hopefully we can try that too with the ST1080 soon.

By the way, as far as i know the Infantry Simulation ARMA 2 and ARMA 3 support dual input ( headtracking and mouse input )
So you can look around with your Headtracking device and aim with your Mouse.

Watch this Video please ( i know its old , may be you already saw it, but its worth a try )
http://www.youtube.com/watch?v=9wXx3vMy_AQ

ARMA 2 is using TrackIR for headtracking. Not sure if GlovePIE TrackIR support ARMA 2 as few years back, TrackIR encrypt their API and interface so most third party solution doesn't work.

*Commercial begin* It supports FreeTrack too, which FreePIE supports *Commercial ends*

[ERP]

Its been an extremely long time since I did any Direct3D stuff, but surely this is just an intercept to SetTransform or the modern equivalent when its doing the view transform. Since you pump that all your transforms, you should be able to intercept the one you want.

It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

The problem is in DX9+ there is no concept of a model/view transform, it's just a matrix input to a shader, there is no context. It's not like the old fixed function stuff where you set the transforms explicitly.
You'd have to look for things that look like camera matrices being passed to shaders, that's not particularly hard, they have a very specific pattern of entries, but you have no idea if it's just the camera transform (or if some of all of the model transform is included), or if it's the camera transform for a shadow map rather than for the primary render target.
Though I guess you can track which render target is selected for the latter.
My guess is you could get it to work for a lot of games pretty quickly, but you'll have outliers that are harder to get working.

[ERP]

It can't be that hard to do, or 3D drivers such as DDD and Nvidia wouldn't be able to do it so well?

Sorry I missed this, I think the reason NVidia and DDD can do it is the modification doesn't require they isolate the camera transform explicitly, they just have to produce the same result as adding an additional translation into the final matrix.

[brantlew]

@ERP: Sure once the matrices all get multiplied together in the render-pipeline it would be difficult to disentangle them reliably. But what if you trapped the view matrix way back up at the initial input function? (ie. SetTransform(D3DTS_VIEW, ...) ) I guess some games might pre-compute the combined matrix and input it all at once, but surely lots of games input the view matrix independently using this method. What am I missing here?

PS. My graphics is rusty and mostly OpenGL based.

[WiredEarp]

I don't know enough to comment here, my DX experience ended with V6 I think.

At a glance tho Brantlew's concept seems workable...

[pierreye]

Here's another VR demo for Mass Effect 3. This clip is longer around 12 mins of unedited gameplay to prove that the whole VR setup is playable.

[youtube-hd]http://www.youtube.com/watch?v=lMIWAV5h4EI[/youtube-hd]

[brantlew]

Cool video!

Is it odd playing in 3rd person view like that?

[cybereality]

Sick man! Very nice.