Independent head and weapon movement (Research Thread)

[PalmerTech]

As I have said far too often, I am not exactly a programmer. Not at all, really. I know ARMA sort of supports independent head and weapon movement, here are a few more things I have seen that seem promising:

SharpeYe: This is a mod that allows for all kinds of cool things. The best part is the "Focus Mode" which allows you to aim anywhere within the camera viewport, though it also has other things, like head bobbing, breathing that varies with stamina and damage, etc. Another thing I find interesting is that it has a "lean" customization that rotates the camera, so it seems like having the camera work on a rolling axis is possible to some degree. This video shows the aiming coolness near the end:

[youtube]http://www.youtube.com/watch?v=JD6bepTpNEQ[/youtube]

Another very cool development is that the Razer Hydra is getting integrated support in all Valve titles. One of the modes allows you to aim within the camera viewport using the Hydra! There is an older "Hybrid" mode that allows you to move the camera with one controller and aim with the other, but the weapon aiming takes over your camera movement. A perfect solution would be an Integrated/Hybrid combo that let you move the camera with the left Hydra, and aim your weapon with the right. Here is a video showing TF2 integration:

[youtube]http://www.youtube.com/watch?v=ca5E67vW4nw#t=183s[/youtube]

A "Free Aim" mod. He provides all the code and explanation in a thread linked in the description of this video:

[youtube]http://www.youtube.com/watch?v=vFzJ-RD_ASc[/youtube]

Anything else like this that people have found, and any ideas on how to make something that works for our setups?

[brantlew]

Don't know the details, but my gut reaction is that it would be engine specific. The Half-Life engine is fairly wide-spread so the same mod code might be compatible across several games, but it wouldn't transfer to other engines like Crytek or id Tech, etc.. I have no experience with game mod'ing though so I couldn't tell you how much control the engines give you over fundamental properties like viewport and weapon aiming.

[cybereality]

Well, there is the mod Vuzix did for Unreal 2004. It had a number of different control types, one that allowed fully independent look and aiming. I did toy with it a bit, it certainly added a lot of immersion but it became difficult to aim (mostly since the crosshair did not update correctly). Though it worked good enough to show promise for this type of control scheme. I wish I had made a video of it, can't seem to find any on YouTube.

The Razer Hydra is really promising. The controls used on TF2 look similar to what they did with Portal 2, and those controls were great. It felt more intuitive than mouse controls, even if you sacrificed some accuracy. This really is probably the best kit we have to work with today at that price, even if it has short-comings (wires, interference, etc.).

Also, the control scheme I used for my BacktopVR video worked decent. Basically both the headtracker and the controller would map to the mouse emulator, but the controller would be disabled while the headtracker was moving. This had the effect of shutting off hand motion when moving (turning, etc.) but allowing fine-tuned aiming while standing still.

[youtube]http://www.youtube.com/watch?v=ni-QFyfZ8rY[/youtube]

Not perfect, but the advantage is that it works out-of-box with literally any type of first person computer game. Otherwise we will need to mod specific games individually.

This is exactly the kind of stuff I hope to be able to do with my DirectX driver, but the project seems a little more complex than I initially thought. I haven't given up yet, just haven't had too much time recently to dedicate to it. But its still on the roadmap.

[CyberVillain]

More game dev's should take after Bohemia and integrate independent aiming and viewport, are there any other games then Flashpoint and Arma that has this feature? Seems strange since its such a nice feature

[WiredEarp]

Good thread PalmerTech, much needed, and thanks for the info re mods. Its an area i've always wanted to investigate but always had other priorities.

I think the Hydra is a wonderful device for VR, and definitely one we should all be throwing our weight behind us. To have available magnetic sensing technology at consumer prices (and 2 sensors as well!), and also software support from a big company, is amazing, and something I hope we will all reap benefits from. Finally, with this and the Sony HMD, the dawn of VR is here. Not the glimmer seen by the pioneers, but the bright light of the new day of consumer VR.

It seems Source/HL2 based games are going to be the easiest candidates to mod seeing as so much is already done for them. However, lets not think that this is anything but a stopgap. We really need custom VR games, or at least some serious mods, before we can take a game from a game with a HMD, to VR. Such things as absolute and unrestricted hand/gun movement and real acting virtual sights are essential to the immersion we all really crave - even if we don't know it yet.

[AsGoodAsReality]

We've been playing around with independent head and weapon movement in our immersive VR game experimental rig - using ARMA 2. To be honest, it can be really difficult to get used to when you are in the game. I've spent quite a bit of time immersed in the game and I never really got to grips with independent weapon movement - even though I wanted to experience it and had high expectations.

We have now gone back to HL2 based scenarios where the head is tracked to weapon movement (the tracker is on the weapon) and, perhaps because of the user's prior experience of FPS, this seems to be more natural and easier to grasp. It is certainly a lot more fun than getting frustrated in ARMA 2.

[PalmerTech]

What is difficult to get used to about it?

[AsGoodAsReality]

Two things:

1: I found it difficult to constantly keep track of where your weapon is pointing (and hence be able to bring it naturally into action) - obviously you are immersed in the game, so can't see where it is in reality,
But worse:
2: It is difficult to make movement natural. We locked movement to the gun direction, so movement tended to take you in unexpected directions and the only natural or usable approach seemed to be to move the gun back into your FOV, point it straight ahead, and then move along its axis.

Having said all this, we did not attempt to optimise our implementation (it wasn't and isn't the focus of our work) and I did not spend long enough in it to really grasp and understand the full interaction between the weapon direction, the view direction and movement direction. With lots of practice this should be possible.

[CyberVillain]

Two things:

1: I found it difficult to constantly keep track of where your weapon is pointing (and hence be able to bring it naturally into action) - obviously you are immersed in the game, so can't see where it is in reality,
But worse:
2: It is difficult to make movement natural. We locked movement to the gun direction, so movement tended to take you in unexpected directions and the only natural or usable approach seemed to be to move the gun back into your FOV, point it straight ahead, and then move along its axis.

Having said all this, we did not attempt to optimise our implementation (it wasn't and isn't the focus of our work) and I did not spend long enough in it to really grasp and understand the full interaction between the weapon direction, the view direction and movement direction. With lots of practice this should be possible.

Arma only support absolute position of the Head movement, if also the weapon was absolute it would be much easier because the weapon would aim correctly 1:1 mapped to the headtracker (Hard to explain but I hope you get the idea). I've used ARMA with Freetrack and a HMZ-T1 and standard mouse for wapon amaing with nice result, but I had to turn of the Z tracking in Freetrack, with that turned on the feeling was horrible, Freetracking is tracking Z movement but Arma is using that info for Zooming, and it does not feel natural in any way

[brantlew]

I can see how tying motion to the gun direction would feel really awkward. Without three independent controllers and direction vectors I think just about everything is going to feel like a compromise.

[BillRoeske]

Back in college (2003, SUNY Buffalo), a friend and I hacked together a setup with one 6DOF tracker on the HMD and another attached to the user's belt, centered in back. His or her forward vector for movement was whatever direction their hips were facing, independent of where the HMD was looking. We never got to the point of adding a weapon, but an independently tracked wand would have filled the role, there.

For view-independent aiming with something like a PlayStation Move or Wii Remote Plus mounted in a Zapper, I'd be strongly tempted to try the approach Nintendo used in the Legend of Zelda: Skyward Sword. Assume that the gun is always a fixed distance away from the user and pointing outward along its forward vector, basically mapping it to a sphere. Zelda does some nice IK to move the origin of that sphere along with its best guess of where the user's elbow is, but I think rooting the sphere near the user's origin would be an acceptable starting point.

This approach wouldn't ever let you do something like blind firing around a corner, but you'd need fancier hardware than just an accelerometer and orientation sensor on the gun for that, anyway.

For anyone interested in the movement details of our 2003 setup:

For small movement corrections, the user could just move around by one or two steps. For longer journeys, the user could walk forward by standing upright and walking in place, sidestep by leaning to the side and walking in place, and backpedal by leaning back slightly and walking in place. We constructed a pretty Frankenstein set of two-part "sandals" that would strap under the user's shoes. They had contact sensors in the ball and heel areas, which were wired up to the button sensors on an old Thrustmaster wireless gamepad. We did some rudimentary gait detection based on the cadence of the foot switches being turned on and off when someone was walking.

We never got to the point of refining the setup, but it worked pretty okay for what it was. By that, I mean most users didn't almost fall flat on their face as soon as they tried moving around. For those that did, I blame the fact that we built the foot sandals out of 1/8" plate aluminum. I have some fabric ones sewn together, but they never were put to use.

For a cheap DIY home set-up, it wouldn't surprise me if a cell phone with a magentometer and accelerometer (pretty much all smart phones) could replace the hip sensor and also help detect gait. Not sure how you'd replicate detecting lean without trackers with absolute position, though.

[brantlew]

There's been some discussion of a system like this before. It's cool that you actually implemented it. These days a lot of the ideas revolve around using the Kinect.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 784#p64474

What did you learn from the experiment - or what would you do differently if you were to do it again?

[BillRoeske]

Since it seemed to be the more relevant place, I migrated my post over to that thread, added some pictures, and tried to answer your questions. Feel free to ask any more if I missed anything!

[zalo]

If the VR system is back-mounted, and the Hydra is USB powered, what's stopping you from back-mounting the Hydra base, and attaching a controller to your HMD and Gun (or waist)? This way, you'd have drift free 6DOF data for your gun and head relative to your back. Or am I missing something? A normal gyro sensor could do the whole body rotations. Unfortunately, this would all require insanely specialized software and hard-core maths.

[brantlew]

Bingo! I think this is one of WiredEarp's ideas as well.

[Chriky]

If the VR system is back-mounted, and the Hydra is USB powered, what's stopping you from back-mounting the Hydra base, and attaching a controller to your HMD and Gun (or waist)? This way, you'd have drift free 6DOF data for your gun and head relative to your back. Or am I missing something? A normal gyro sensor could do the whole body rotations. Unfortunately, this would all require insanely specialized software and hard-core maths.

I am trying to make a system like that at the moment!