What's lacking in VR controls and stereo support for gaming?

[brantlew]

I'm launching this thread to call out and clarify the hardware and software technology that is lacking in the gaming industry in order to support the VR gaming experience. These would include things like high FOV/sensory blocking HMD's, driver support and API hooks for stereoscopic display, standardized control interfaces for precise analog motion mapping and head-tracking, and others...

My current area of interest is in motion interfaces, so that is where I will direct the first comments and let others fill in the blanks. Right now for PC gaming the only real "standard" interface is the WASD/mouse input. You can accomplish quite a bit with mouse emulation head tracking, but it is not perfect since it uses a relative coordinate system and can drift. The WASD keys (plus running) provide a very limited form of motion control offering only 8 discrete directions and 2 speeds to map to. Additionally, height is typically confined to 3 discrete states - standing, crouching, and jumping. The XBox 360 gamepad controller is sometimes supported and offers a different but equally frustrating control scheme. It supports analog direction and speed of motion at the expense of imprecise turning. Very few (if any) games support simultaneous gamepad + mouse control so you are left with the tough decision of choosing either well mapped motion OR head turning, but not both at the same time.

What is needed is a standardized and generic, forward-compatible control interface that would allow for free and accurate motion mapping along several dimensions. This would include (but not be limited to):
- control of the viewport yaw, pitch, and roll in absolute angular coordinates
- analog control of direction and speed of motion (relative to the forward yaw vector)
- analog control of height
- separation of the head and other body parts (ie. separate look and gun aiming)

Now obviously it's not very pragmatic for game developers to support non-existing control hardware, and I'm sure there are many technical and software management issues that would be involved. So it's not surprising that these things don't exist. But I would love to hear the industry's stance and approach toward these interfaces (hint: that means you id software). Is this ever a discussion about this? How would this effect game design and what problems would it cause? Is there even the beginnings of a standard in the industry or the academic community for VR control interfaces?

[PalmerTech]

Just a quick thing to throw out there: The Razer Hydra hardware can support all of these things, and the new driver update at the end of the month has integration with all Valve games that lets you move the aiming crosshair around within the camera viewport, and another "hybrid" mode that lets you aim with one controller and move the camera with another. Unfortunately, these two modes do not work in tandem. I asked about it on the Sixense forums, and they said that doing so would require ratcheting with your head to play. Not a problem if you plan on standing up and spinning around! One of the guys from Sixense also said there it should be possible to map the Hydra 1:1 to in-game cameras and aiming.

The Hydra also allows for analog movement. Basically, it has pretty strong driver support already, and seems like the best bet for getting what we want. A "Hybrid/Integrated" mode for the Hydra could get us pretty far! Perhaps someone from the community could modify an open source game to support the Hydra for everything we want, and that could be used to drive interest from the industry.

As far as a standard VR control interface, VRPN all the way.

[cybereality]

I would just like to see a game support a decent data-glove for control. Currently I don't think a suitable glove exists (at least on the consumer market) so I doubt there will be much interest from game devs to support vaporware. However, the Razer Hydra does have most of this functionality, so data-glove like interfaces could be designed. I think the game that came closest to this was Penumbra, where you would have to open drawers and doors and things like that. They even had Novint Falcon support at one point. Also, the game Trespasser had a dynamic hand like control (never played it though) which seems like it would work great with a glove. Currently mouse controls just don't cut it for immersive experiences.

[WiredEarp]

Unless i'm mistaken, isn't this exactly what I proposed in another thread? An open source interface definition so that multiple controllers etc can write to the same format (or use conversion tools such as FreePIE to convert to the format) and have a game read it?

If so, I totally think this is needed to progress games/sims further. I believe MS tried with SimConnect etc (I know nothing really about simconnect tho), but something lightweight would be great.

I suggest that some base definitions should exist (head tracker, local mobile tracker, etc) with specific types/ranges. That way, a game coder can write his interface to read the amounnt of (eg) head turn in radians (or whatever is chosen as the data type) and doesn't have to rewrite his code to deal with what another device might be pushing to the interface - which if its not constrained, could be degrees instead of radians etc, which would require the game dev to do more conversion work. I think for this idea to succeed, it would need to be as easy to implement as possible in any software.

@ CyberReality: I think the Razer is great but I think we need an open definition, not just one based around the current 'hot' device. This way, game coders could easily implement support for head trackers etc, without having to do much work. The Razer is awesome tho, i'm about to go and try and fix mine, or else I'll be buying a new one soon

[brantlew]

@WiredEarp: True. There are no novel concepts here - just an aggregation of ideas put forth many times before. I did however want to accentuate the need for analog motion controls, since that gets discussed much less than head-tracking and independent gun aiming.

@PalmerTech: A reference design could be useful as a way to clarify the problem and work through some of the technical details. I would suggest modifying a pre-existing open source game. Without industry backing however, it is doubtful that the idea would gain much traction. We've got a double-whammy problem of being a small group of people with very few hardware implementations, so it's hard to see where any game developer could allocate resources for something like this unless they were independent and very progressive.

[ERP]

As big an issue if wide FOV HMD's ever become available will be the games HUD's.
There is just too much convention sticking bits of information at the edges of the screen.

[Bishop51]

The game ARMA has the single best utilization of desktop VR control out there and its one I wish more games would emulate. Your head control can be independent of WASD and mouse body control and your gunsight is locked down with ironsights. Your head control can be locked to the Ironsight when necessary but other than that you can look around freely. This is of course dependent on having a head tracker set up so that your mouse controls your body rotation and your head/neck control viewpoint. https://www.youtube.com/watch?v=haCfLdFvJ8M

So for me the ideal implementation should take two key operating modes:

• Desktop VR (90% of general usage): Seated position. Body rotation, movement and general interaction controlled by WASD & Mouse. Head viewpoint controlled by HMD with head tracker.
• Console VR (10% of general usage): Standing position. Body rotation, movement, controlled by Kinect full body tracking. Button and solid state interactions activated by wireless controller. Head viewport controlled by HMD with tracker.

[morgg]

As big an issue if wide FOV HMD's ever become available will be the games HUD's.
There is just too much convention sticking bits of information at the edges of the screen.

Thanks to surround gaming (eyefinity and such), that is changing. Some games can detect extra-wide FOV and they position HUD accordingly. Some other games are "hacked", like those based on source engines.

[brantlew]

As big an issue if wide FOV HMD's ever become available will be the games HUD's.
There is just too much convention sticking bits of information at the edges of the screen.

That's a good point. Customizable HUD placement or at least HUD placement within a confined forward direction instead of far off in the periphery.

[FingerFlinger]

How hard would it be to grab the HUD pixels out of video memory and paste them where we want? Seems like it would be a PITA but possible, at least. (Not a graphics programmer.)

[brantlew]

@FingerFlinger: I think you would need to intercept it within the rendering pipeline rather than after the final frame has been rendered.

[WiredEarp]

I'd just ignore the HUD issue. They can be useful, but they kill immersion, so I don't really see them as being that necessary for VR.

[pierreye]

It's possible to hide commercial games HUD without hacking the code. Look for Helix 3D Vision fix that allow you to look at the shader and hide the effects. There is a possibility to either disable the HUD or push the depth using the directx overide dll.

http://forums.nvidia.com/index.php?s=b1 ... pic=222386

[WiredEarp]

Yep, many games nowadays have HUD hiding modes as well built in.

So what base interfaces would be required for a generic interface?

I can think of:

Headtracker
Hand trackers
Generic analog motion controls
Gun trackers

Almost all of which could really use the same generic class. What would be good would be if the device can 'register' with the interface, and then any game could read it. IE, a game could 'register' the head tracker and gun tracker, and the game coder could just do an 'if exist headtracker' to determine if there is a headtracker he should bother reading the data from...

[CyberVillain]

All games support direct input, we need to influence Microsoft to make a general tracker interface in DX