- My goal is to focus development effort toward a specific demonstration platform in order to provide friends, coworkers, and acquaintances with a great VR experience. The content below is my effort to distill a small part of the vast knowledge stored on these forums into cohesive work, such that a hardware / software baseline is established making a VR platform available for this experience.
- In reading through many of this forum’s threads over the past few months, it's apparent that we are approaching a convergence point for a number of technologies that will soon put virtual reality within the reach of the common consumer. Until recently, the biggest technical obstacle was the availability of a high quality HMD, but with the availability of the HMZ-T1, and more recently the ST1080, we've now cleared this hurdle. The next logical step is to integrate tracking with the HMD, of which there are a multitude of approaches to head / body / weapon tracking. Each approach has its advantages and challenges, along with varying degrees of software integration. Hardware for tracking is now more affordable than ever with cameras, IR, solid state, and more available to anyone with a modest budget. Software platforms are also becoming more open to tracking integration with studios like Valve and id software taking the lead with SDKs and the like. With the coming advent of the Oculus Rift (http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=14777), we have the first steps towards a tracking technology with some momentum (as well as the first high quality unified VR gaming experience). While there rightly will be developers pushing the envelope on what is possible, I think its time to draw a line in the sand and define a "base" specification for tracking that will demonstrate the power of immersion while maintaining relative simplicity. The tracking hardware chosen for the Oculus Rift, the Hillcrest FSRK-USB-2, is an excellent launching point for this base specification.
(A similar opintion was voiced by Nick3DvB here: http://www.mtbs3d.com/phpBB/viewtopic.p ... 040#p74347)
- The Hillcrest FSRK-USB-2 kit offers a 6DOF tracker with a simple USB port for power and output. This offers some excellent features in a compact and easy to integrate package. Tracking info is provided by (3) integrated accelerometers and (3) gyros. Results collected by various mtbs3d members seem to point at some drift in x, y, z, tracking (http://www.mtbs3d.com/phpBB/viewtopic.p ... 128#p76756) but quite accurate results on Pitch / Yaw / Roll. Also of note is a modified firmware for this reference kit that will be made available with the Oculus Rift that should double sampling rate, increasing responsiveness from this hardware, and cutting latency. At this point it is unclear if this firmware will be made available to private parties who purchase the Hillcrest kit separately.
- In looking at how to integrate this head tracker into a computer gaming setup, three main configurations come to mind.
- In this model, the user is situated at a desktop interface, wearing an HMD with the Hillcrest tracker integrated. The HMD and tracker are directly connected to the computer which is in close proximity to the player.
Advantages:- Easy transition for existing gamers
Minimal required physical space
No wiring concerns- distance to computer is constant
No latency concerns, all devices are wired
Any computer will work
- Physical limitation on Yaw, therefore a secondary look device must be employed to view 360 degrees
- Easy transition for existing gamers
- In this model, the user is free standing in an empty space, wearing an HMD with the Hillcrest tracker integrated. The HMD and tracker are directly connected to the computer. The computer would need to be portable (laptop / tablet / etc), and directly worn by the user.
Advantages:- 360 Degree view of environment
No secondary pointing device required, only x,y,z movement
No wiring concerns- distance to computer is constant
No latency concerns, all devices are wired
- More physical space required (so the user doesn’t hit physical objects)
Possibly more difficult initial transition for the player
Computer must be portable
Playtime is limited by practical battery life
- 360 Degree view of environment
- In this model, the user is free standing in an empty space, wearing an HMD with a wireless Hillcrest tracker integrated (FSRK-BT-1 or FSRK-2.4G-1). The HMD would be connected to a wireless HDMI bridge that would provide video signals from a computer
Advantages:- 360 Degree view of environment
No secondary pointing device required, only x,y,z movement
No wiring concerns- all connections are wireless
Any computer will work
- More physical space required (so the user doesn’t hit physical objects)
Possibly more difficult initial transition for the player
Some proximity must be maintained for wireless reception
Playtime is limited by practical battery life
- Additionally incurred latency due to wireless tracker use
Additionally incurred latency due to wireless video use (Miracast would not work)
Possible wireless video reception issues for 60Ghz solutions
Possibly more difficult to troubleshoot problems due to multiple wireless connections
- 360 Degree view of environment
- All three usage models are worth pursuing and should be taken into account when attempting to integrate software with the tracking data.
- In these early days, growing momentum for VR as a platform will be done in small peer groups, with new users gaining experience through one pioneering friend in their circle. These new users may or may not be experienced gamers, and their first experience should have a gentle learning curve. In the near term I believe the best established platform to show off this technology is Portal2. Portal2 requires no real arcade style reflexes, but awards precision nonetheless. Users will notice their ability to control the in game avatar with increasing dexterity; this alone offering a feeling of achievement.
Further, Portal2 oftentimes deals in great height changes, with falling and acceleration playing into key aspects of the game. This vertical dimension gives the game a certain vivacity with little risk of failure.
Finally, the in game portals provide secondary vanishing points from a first person perspective, which further enhances the feeling of immersion, especially when portals exist on different geometric planes.
With its full integration with the Oculus Rift, Doom 3 BFG Edition will be the first real platform to provide full VR immersion. While this will clearly give a very immersive experience, Portal2 may offer an alternative that may be less intense and therefore more accessible to more casual gamers while maintaining a high degree of immersion.
- We have an HMD, we have the tracking hardware, we’ve defined the usage models, we have a target test platform, now comes the tricky part, the software integration. While I’ve tried to do my homework here, this is the area in which I have the most questions and need the most help.
First, it looks like the FSRK-USB-2 allows for simple mouse emulation out of the box:
- http://hillcrestlabs.com/products/downl ... asheet.pdf - Section 1, Paragraph 3The FSM supports full speed operation (~ 12Mbps) and initially enumerates as a mouse
The unit is also capable of sending tracking data over the HID protocol (same section, next paragraph) which opens up integration with all 6DOF.
Software integration with the Hillcrest tracking hardware would help build a foundation for both the Rift and other HMDs with Hillcrest kits integrated. Given the two data modes available (mouse / tracking data) I assert there are at least three different levels of integration that can be achieved, with a large gap in effort between them. The following comparisons will be presented in the context of a 1st person perspective game.
- With near universal compatibility and the lowest overall effort, 2DOF tracking using mouse emulation is the ground floor. Game engines would not need to be altered, and almost all titles would be immediately compatible. There are some downsides to this approach however.
First, mouse acceleration could cause major problems. If the speed of the response using a tracker (or mouse) changes the rate of movement of the cursor (or camera view), then it will be almost impossible to maintain a consistent origin point. So if you snap your head to the right, and then slowly track it back to center, the field of view will likely be offset to the right with mouse acceleration turned on. This would be near impossible to use with the Desktop usage model above, and would remain a problem in the freestanding usage models for pitch tracking. Turning off mouse acceleration should help, but others will have to chime in on to what degree drift is still a factor with non accelerated mouse emulation using the FSRK-USB-2. A possible solution for this method would be a dead reckoning key that would allow an occasional reset of the in game view to dead center.
The second problem is tracking speed. With mouse emulation, a manual calibration of the speed of tracking would be required to ensure that a physical 45 degree head turn would be mapped as a 45 degree camera movement inside the game. This calibration would not need to be *exact* (especially for the freestanding usage models) but a close analogous would help immersion.
- A larger jump in integration effort, 2/3DOF would be a significant improvement over mouse tracking. For 2DOF, yaw and pitch angle would be accurately tracked and therefore no movement scaling would need to be calibrated. Integrating this movement into a game engine that was designed for mouse control seems like an easy adaptation. 3DOF would be more work, since the camera would need to roll, which may or may not be supported as feature on a given gaming engine. Once we move from 2 to 3 DOF, the discussion of what to do with the camera becomes more important as well. Currently this discussion seems to be captured well regarding the disembodied eye vs head and neck approach: http://www.mtbs3d.com/phpBB/viewtopic.p ... 598#p74598
- This approach has the potential for the greatest level of immersion, but may also require a fundamental change to the underlying game engine. The in game camera would track not only the natural motion of the head on the neck (Pitch / Yaw / Roll) but would allow for additional actions such as ducking, jumping, peering around corners, etc. Based on the drift of x,y,z discussed earlier, I’m not especially optimistic for subtle movements in those vectors, but ducking and jumping may be possible if there was enough of a threshold for z to differentiate between these actions and simple head jerks.
- Although simple mouse emulation will be great for compatibility, game integration with 2DOF tracking data should be the baseline. If more DOF become possible, that’s great, but 2DOF integrated with a game’s mouselook will go a long way toward immersing the user while keeping the overall software integration effort to a manageable size.
Although I have many aspects of my base demonstration platform established, I still need help to achieve my goal. I'd appreciate any input on this project, including corrections, tips, or answers to the questions below.
- Does the FSM-USB-2 indeed work as a mouse emulator right out of the box? Can tracking speed / acceleration be adjusted using standard windows control panel settings?
What does the software stack look like for attempting to get tracking data into Portal2? Where does libfreespace fit in?
Brantlew linked to the Valve how-to for 6DOF motion tracker integration (https://developer.valvesoftware.com/wiki/Head_Tracking), but it seems this is only necessary for head positioning using all 6DOF. If we’re only attempting to integrate 2/3DOF is any of this necessary?
Are there any current examples, sample code, etc for some user that has already attempted this?
