Full Body Motion VR Ideas

[Zaptruder]

Hi guys, I'm new to the forum. I've been lurking for a bit, just trying to soak up all the info I can. VR is a topic that I've been thinking about on a number of levels for quite a while now, and although I don't have the technical skills, I'd like to be able to contribute something worthwhile to the community and the technology nonetheless. Also I'm putting this out there because ideally, I'd like someone else to implement the idea to see how well it works. If they're no takers, then I'll do what I can myself after I get to grips with programming/scripting/etc in general.

Aside from a wide FOV and in-game POV meshing with your own POV, one of the big draws of the VR idea is the ability to use your body to interact with the world - specifically as it relates to moving around it.

There have been a number of solutions for this difficult problem, ranging from exotic hardware like roller skates with sensors, to ball bearing treadmills, to even more exotic hardware like 'holodecks' - specialized rooms that allow for free action within the room. Some have taken VR outdoors, onto large open areas like basketball courts.

Been a VR geek, I think all that is very cool. But been a pragmatist, I can't see any of that stuff been realistically workable in a mass market way that'll actually appeal enough to developers and audience to want to develop for, and spend money on that equipment and gear.

I mean... irrespective of how accurate a sensation of movement you can get with the tech - be it a roller ball with some sort of movement limiting exoskeleton - I can't see this technology been something that will appeal to enough people to get the world excited about it - not like the Occulus Rift is doing.


To a large extent, I think the problem is like HMDs before the Occulus Rift - we're implicitly assuming a certain element of full body motion control that isn't necessary - and is causing us to look down the wrong path. That implicit assumption is simply - that you have to push your legs forward.

Drop that assumption - and suddenly EVERYTHING becomes easy. What's the solution to walking forwards? Walk on the spot. Running? Run on the spot. What about colliding into things? Well, you're jogging and walking on the spot.

While on one hand, it's not quite as immersive as actual movement - on the other hand, it is still engaging our sense of proprioception in a very big way. That's pretty much what VR is - a synchrony of sensory elements that enable the digital to replicate believable sensory experiences... that's exactly what VR HMDs do - it synchronizes the proprioception of the head and neck with the movement of the camera.

The big up to this method is that you suddenly transform any game involving walking/running around into a low/moderate level physical simulation, reversing a multi-decade long crippling health trend among the developed world - where our excess(ively good) entertainment is increasing sedentary lifestyles at a frighteningly unhealthy pace.

And you do so in a way that is easy to get into and get out of - requiring very little in the way of specialized equipment - the technology to do so is already available at a consumer level (HMDs w/ head tracking + kinect motion tracking) - it simply needs the appropriate software to allow this to happen.


To be fair... what I'm saying may not be terribly revelatory - in some ways, it's pretty obvious after all - with at least one guy doing something like it for Skyrim (although the video has been taken down for some reason). But humour me if you will... just stand up for a moment, close your eyes, and walk and jog on the spot - imagining that you're moving forward in space.

It's not that bad is it? As long as there is sufficient synchrony between the movement of your limbs and the movement of the view port, it could be very immersive, and very practical.


So the details of the proposed solution are simple enough.

Get the kinect - track the leg joints and their relationship - the faster and higher the knee moves up, the faster the player moves in game.

Functionally, this would take the form of a piece of software that translated that movement into a game pad style input - the faster you pump your legs, the greater the vector of the direction of movement.

But it's not sufficient for a single vector of movement - free motion requires the ability to allow the player to move in all directions at any time. Games achieve this by allowing you to strafe and back pedal.

The key to this working then is a straightforward intuitive design of how to translate 'on the spot' movement into strafing.

Leaning is... not desirable - it doesn't feel natural, and it can be ergonomically disastrous given enough time. Neither is using your arm to indicate direction of travel - you want to leave those free for other gestures and movement and or control.

You could link it upto a controller - have the player indicate the direction of movement with an analogue stick, and have the walk on the spot/run on the spot movement generate the magnitude of the vector.

But maybe the better way to do it would simply be to have a centre point where the player stands on indicate forward movement - while a circle around him represents the direction of travel - if the player walks with at least one foot in that direction of travel, then they strafe in that direction. The central area should comfortably encompass an average size person - with a radius of about 1.2 feet. The outer circle should be easily accessibly simply by having the player step to the side - that way, changing direction is never more than a couple of short steps or a larger step away.

One problem this does introduce is that of drift - all this side stepping can cause a person to drift from the centerpoint if they're not careful - even trip over things that aren't quite where they expected to be. The solution to this would be to have an on screen display element - that shows where the user is relative to the centerpoint; when they're out of the centerpoint.

I've got a logitech keyboard that has quite a nice OSD that goes over everything any time anywhere when I hit the capslock/numlock functions. I wonder if it'd be possible to do the same for a little circular indicator telling you where you are.


The upside is that this solution can work quite well for both when you're tethered, and untethered - if you're tethered (allowing you to use the full power of a desktop PC; which I imagine we're going to want to do for at least a few more years), the heading would have to be controlled with a traditional controller (which according to preliminary impressions with the occulus rift, isn't too bad - still an extremely immersive experience) - such as a 360 gamepad (hold one side in one hand, not that bad), or more ergonomic devices like a PS Move Navigation controller, or the razer hydra.

If you're untethered, then the experience would be even more straightforward - the circle moves relative to the direction that your torso is pointed in - to be honest, I'm not sure how well the Kinect camera tracks the sides and back of people - but assuming it's reliable, then it should be still a relatively straight forward experience. Circle strafing would literally be like running around in a little circle!


To be honest, the gist of what I'm saying seems kinda obvious to me. It almost makes me feel like I'm missing something - is there a reason that this approach hasn't been done yet? The pros are quite significant, and the cons... well, not that bad.

Pros:
Low cost, not much equipment required (1x kinect + software).
Accessible to everyone.
Easy to startup and use.
Relatively intuitive.
Allows a full body motion framework to develop that could reliably and easily be supported by software and content developers.
Easy enough to add to existing games.
Minimal space requirements.
Good fun exercise.

Cons:
Reduced sense of proprioception - no bodily sense of actual forward movement - but leg pumping with visual forward movement still gets you a long way there.
Requires a little training (centerpoint and strafing).


But that's not all - the above covers leg movement, and could be feasibly implemented by a talented coder. The next part of this post covers arm gesture/body movement.

So the ideal for VR arm movement is that it tracks our arms 1 to 1. Which with a kinect isn't too bad. Even with latency, it's still a very workable solution. When the leap motion detector is released, it may be possible to further augment this sort of 1 to 1 arm control.

Only problem is, there's no software to support it! On the software side, it's a much more complicated process than what I've proposed above (i.e. you can't map complex arm/hand movements to a control stick) - which is also a large part of why I've seperated out this discussion from the leg movement part.

To be honest; without direct developer support, I would largely stick with hand held control tech like the move navigation controller and the Razer Hydra. The efficacy of quick button responses largely outweigh the novelty of gesture controls (as opposed to 1:1 arm mapping).

The problem with gesture controls is that they rely on the user to gesture first - recognize the gesture, then play the action - there's a good 200-500ms gap between action and effect; enough to create a cognitive disconnect - enough for people to think that, that's not me performing the action (and it isn't really). In the end, it feels clunky, even foolish to play like that.

But with developer support - where the they build an arm/physics model and gameplay/interactions that is relevant to that full range of arm/hand movement, then the sky is the limit there...

The main problem with this though is collision and tactile feedback or lack thereof - should your arm or a weapon you're holding collide with geometry in virtual space - how should the game respond? Just clip through the object and pretend it's not there? With clashing swords, do they just phase through each other?

Some people think that this is an intractable problem with VR if you don't have the appropriate exoskeleton arm or some such that can limit range of motion. Sure that'd be pretty good if you can get hold of one - but assuming that you can't, but still think that we should push gaming into this more immersive and natural method of interaction, then what other options are available?

I think without exception, even if you can't simulate the tactile feedback of the virtual world, you still want to respect the reality of the virtual world - in this sense, you can't have characters, arms, weapons, etc clipping through each other, even when the player input is demanding it to!

But if there's a disjunct between the player's motion and the game's representation of that motion, then the illusion of reality that you've been attempting to cultivate with all this technology is lost.

So there obviously needs to be some sort of feedback - while tactile is difficult to achieve on such a macro scale (at the size of the arms, rather than hands/finger tips (which I believe is termed haptic anyway)), you can use the sensory elements that the existing technology already stimulates - providing the player with visual and auditory feedback.

In practice, this means that as you wave your arm about, your in game arm should wave about as freely as you do. When you're in game arm collides with something - it should be restricted by that object - but your actual arm still requires representation to let you know where it is.

The trick then is, is to represent the physical arm that is out of sync with the virtual arm - in a manner that motivates the player to 'reengage' the virtual arm - to resynchronize the arms in both spaces.

Although the method is upto the developer - one example of this is the idea of a red phantom limb - as the arms lose synchrony, the representation of the player's actual limb location grows stronger - the further away from the virtual limb it is, the more it should glow red, perhaps even with visual indicators to show the player that they need to 'grab' the virtual arm.

The consequence of having it out of sync too long is that the virtual limb loses control - falls flatly to the side, at which time the player needs to re-engage the virtual limb by placing their arm at their side.

You can of course use audio cues to reinforce this loss of syncrhony between the virtual and the real arm.

The effect of this would be to allow the virtual arm to be affected by game physics - if your character is carrying a large heavy item, and you're swinging your arm all over the place, they drop the item, and you'll have to re-engage your arms to get them to move again. Similarly, if you continue swinging your arms wildly after your weapon has been parried in a sword fight - you lose control of the arm and the weapon.

On the flipside, if you keep your arm roughly in line with your virtual arm, there is a convincing proprioception sensation of weight and physical boundaries - try it for yourself - imagine a wall, punch it, and stop your arm as it meets this imaginary wall.

There is no wall, and yet the sudden abrupt jerking sensation is there as you'd expect it to be (even when it's you controlling your arm, and not because there's a wall stopping you from moving further).

I'd expect initially, it may not be super convincing - but with not all that much training to get used to the idea, your brain will be cued to halt your arms and extremities from collisions as convincingly as if it were there - the red phantom limbs and loss of control providing you with the mental incentive to stop the collision in place of pain and actual physical occlusion.

This solution like the movement solution seems like a pretty obvious one to me - I might be missing something again, but I figure this is more just a case of it's a tricky software thing to do, with little reason or need to do it yet.

Combined, these ideas represent a low cost, but nonetheless effective solution to the problem of full body motion simulation that can be used to both enhance the immersion factor of virtual reality, and provide developers with incentives to create next generation gameplay and interaction ideas for games that increasingly bumping up against the boundaries of diminishing returns.

Hopefully my explanations are fairly clear. I intend to sketch some diagrams to include with this post to clarify things further, but I'd thought I'd get some feedback at this stage on these ideas.

*edit* Text wall crits forum for 9999. I'll definetly have to draw up the diagrams and get them in there for clarity.

[brantlew]

I agree in principle with much of what you have said. I argued for a simple Kinect gesture-control system a while back over a complex mechanical device. However there are still a couple of potential issues to keep in mind.

One - the Kinect has a substantial amount of lag associated with it's skeletal tracking. Now this may not be a big problem for locomotion. Walking is a really long sequence of events without a really clearly defined starting point. You've got to lift your leg, then lean forward and fall, then put the leg down. All of this takes up to 1/2 a second to accomplish and as a result, the VR walking sensation is very lenient. I've done a good bit of free-walking in VR and you can honestly handle a good 1/4 to 1/2 second latency from the mental initiation of walking to movement on screen and still feel connected. So that works in the Kinect's favor. But for hand gestures I think you would need a more immediate response. I don't think 100 or 200 milliseconds of lag is going to cut-it for hand/weapon motion. Hopefully the Kinect 2 will improve on that substantially since it will supposedly have on-board processing.

Two - Walking in place certainly contains some of the motor skills and weight displacement sensations of real walking and I think would be superior to many proposals that I have seen using harnesses, skates, etc. But in the end - walking in place is not really like walking. Especially since it's impossible to differentiate directional motion. So you end up having to do what you have suggested - either using a positional system (turning yourself into a sort of joystick) or some type of gesture system (crossing your legs for side movement, putting one foot back for backwards movement, etc..) I think both scenarios end up feeling a bit fake and you remain conscious that you are using a "gesture" system. You are one step removed from the natural intuitive sense of "I want to go over there so I just go over there". Instead its like "I want to go over there, so I position myself appropriately and go through this animation sequence". Admittedly, I haven't done a lot of walking in place. But it's not clear to me that it's any better (more immersive) than button mashing to accomplish the same goal. Is there anyone out there that can comment on this? I bet Palmer has done a bunch of in-place VR walking. On the flip-side: real uninhibited walking adds a whole new level of immersion to the VR experience, which is why I'm such a big advocate for that type of system.

[Zaptruder]

I agree in principle with much of what you have said. I argued for a simple Kinect gesture-control system a while back over a complex mechanical device. However there are still a couple of potential issues to keep in mind.

One - the Kinect has a substantial amount of lag associated with it's skeletal tracking. Now this may not be a big problem for locomotion. Walking is a really long sequence of events without a really clearly defined starting point. You've got to lift your leg, then lean forward and fall, then put the leg down. All of this takes up to 1/2 a second to accomplish and as a result, the VR walking sensation is very lenient. I've done a good bit of free-walking in VR and you can honestly handle a good 1/4 to 1/2 second latency from the mental initiation of walking to movement on screen and still feel connected. So that works in the Kinect's favor. But for hand gestures I think you would need a more immediate response. I don't think 100 or 200 milliseconds of lag is going to cut-it for hand/weapon motion. Hopefully the Kinect 2 will improve on that substantially since it will supposedly have on-board processing.

Two - Walking in place certainly contains some of the motor skills and weight displacement sensations of real walking and I think would be superior to many proposals that I have seen using harnesses, skates, etc. But in the end - walking in place is not really like walking. Especially since it's impossible to differentiate directional motion. So you end up having to do what you have suggested - either using a positional system (turning yourself into a sort of joystick) or some type of gesture system (crossing your legs for side movement, putting one foot back for backwards movement, etc..) I think both scenarios end up feeling a bit fake and you remain conscious that you are using a "gesture" system. You are one step removed from the natural intuitive sense of "I want to go over there so I just go over there". Instead its like "I want to go over there, so I position myself appropriately and go through this animation sequence". Admittedly, I haven't done a lot of walking in place. But it's not clear to me that it's any better (more immersive) than button mashing to accomplish the same goal. Is there anyone out there that can comment on this? I bet Palmer has done a bunch of in-place VR walking. On the flip-side: real uninhibited walking adds a whole new level of immersion to the VR experience, which is why I'm such a big advocate for that type of system.

Thanks for the critique.

While you're right in that the ability to actually walk and have that tracked naturally is the height of immersion... the practicalities of it all doom it to failure for anything outside of very specialist uses.

Even if you have sufficient space to move about freely without limitation (i.e. a large open field without obstacles), you'll still encounter problems like height changes not meshing with what you're seeing. And in practical situations for the forseeable future - where you're wirelessly tethered to high end rendering equipment, you'd be inside a building, which has very limited spatial bounds - you can only walk so far in one direction before bumping into a wall.

Space sensitive solutions ranging from roller balls to omni directional treadmills are not vastly more convincing in those movement terms - even though your legs will enjoy a greater range of motion, because of a host of other small sensory factors (including but not limited to the feel of the terrain under foot, and movement of air on skin) - it won't provide a wholly cohesive movement solution.

Personally, I don't think these problems will ever be satisfactorily addressed until we have the kind of technology that allows us to stimulate sensory nerves directly, while at the same time muting our actual sensory organ impulses.

Still, beyond the immersion factor, there are compelling physical fitness reasons to want to be able to use your own locomotion to drive movement in the virtual space. Also, while not a wholly accurate indicator, simply standing in place and walking and running and visualizing a change in view point would suggest that this method, even with its limitations is significantly more engaging than been seated and using a keyboard, or even standing and using a controller. I use the entry of Whiterun in Skyrim to the Jarl's longhouse as a test area as I'm quite familiar with it (it's rather more difficult to just visualize some stretch of long road that extends straight for yonks) - and it feels good imo.

[pierreye]

I had done the walk in place using wiimote tie to my left leg and using the joystick for direction. I write a simple script in GlovePIE that if I detect changes in leg angle, then the joystick movement will register. The only con is if you are going to play Skyrim for 3 hours straight, it's quite tiring. Also, I had problem mixing crouch/prone with walking motion using wiimote so in the end, I decided that it's better to just program it for crouch/prone motion.

[JMX]

I've tried using leg/leaning gestures for movement with the Kinect and FAAST. It's interesting but it does feel unintuitive. You have to move enough to trigger the gesture, but you also have to have enough of a deadzone so you don't trigger it unintentionally, which means you always have to be aware of how far you are moving and intentionally move the right amount. It doesn't let you forget you are trying to work through the gestures to just play.

The walking in place gesture is too slow to be recognized and is triggered incorrectly too much, so it feels even worse.

The Kinect is neat to use for gestures but it's also inaccurate in a lot of ways. You can get it close to working correctly but it fails frustratingly often. When a limb gets too close to another limb or object, or is occluded it can lose track of it and the virtual skeleton's limb seems to flail around triggering random gestures and then goes limp until the limb is fully separated from the obstruction. These could be fixed by improving the tracking to handle occluded limbs better, it needs more polish and "playtesting". When they were developing it they probably didn't have any real software to connect it to for extensive testing so it would be hard to notice these things.

Using a gamepad/joystick can actually feel more natural for movement because it's so easy for you to directly map your intention into the input that you can basically forget you are using a controller.

Maybe for tactile feedback it could be something like a glove with output nodes that can expand/contract to give a feeling of pressing on something. Maybe some kind of inflation or hardening or magnetic force to move the node. http://en.wikipedia.org/wiki/Haptic_perception
http://www.roblesdelatorre.com/gabriel/ ... re2001.pdf
In this study they say that this kind of feedback can seem real.

http://en.wikipedia.org/wiki/Adaptiv
Electronic control of the temperature of a surface is also possible as the adaptiv system shows.

Palmer say's he's worked with VR stuff and people rarely challenge the reality of the walls they see in VR. So if you reach out to touch something and you feel the pressure and temperature of it you will probably "buy" that it's really there and not try to push through it. Even in real life you don't usually press too hard on things just to see if they're actually solid, you just touch enough to feel the surface and how much give there is. The number of nodes would determine the granularity of how detailed the feel of the surface could be.

[Zaptruder]

That's some good info.

In theory, the things I discussed above aren't invalidated - it is one of a number of solutions towards the full body motion control for a greater degree of immersion in VR... but it is heavily reliant on a piece of technology that currently isn't all that reliable itself.

I can only hope competing technologies (unlikely) or the kinect 2 itself can remedy the janky experience provided by the kinect.

[Zaptruder]

I had done the walk in place using wiimote tie to my left leg and using the joystick for direction. I write a simple script in GlovePIE that if I detect changes in leg angle, then the joystick movement will register. The only con is if you are going to play Skyrim for 3 hours straight, it's quite tiring. Also, I had problem mixing crouch/prone with walking motion using wiimote so in the end, I decided that it's better to just program it for crouch/prone motion.

Yes, but now when you tell people that you had a 6 hour long gaming session, they'll be impressed, rather than roll their eyes in disgust

Still, would you be willing to share that script? I wouldn't mind trying it myself sometime.

[cybereality]

I did some experiments with walking I'm place. Well, it's not great, but it does add a little bit of immersion. Not just because of you feet, but other subtle cues like the way your head bobs. I think it could help.

Back years ago with one of my first VR setups I was using a DDR dance pad hooked up as a USB game controller. I mapped the WASD to the 4 directions on the pad. So if I stepped forward it would move forward. If I stepped to the side it would strafe. I also walked in place to add further immersion. Ghetto, I know, but it showd some promise. Certainly more immersive then sitting in a seat.

I think a similar system could work like was suggest by the OP. Basically there would be an invisible ring floating around the user. Stepping forward and walking in place would walk forward. Stepping to the side would strafe. The key part is that as the user turned, the orientation of the ring would follow. So forward would always be forward to the player. I think it could work.

[druidsbane]

Palmer say's he's worked with VR stuff and people rarely challenge the reality of the walls they see in VR. So if you reach out to touch something and you feel the pressure and temperature of it you will probably "buy" that it's really there and not try to push through it. Even in real life you don't usually press too hard on things just to see if they're actually solid, you just touch enough to feel the surface and how much give there is. The number of nodes would determine the granularity of how detailed the feel of the surface could be.

The paper referenced is really cool. There are no glove devices like that on the market are there? Also, that means you'd need some fairly detailed collision detection in software to get the right force and angle right for solid objects. I really like the concept, though I think it will be hard to get it supported easily unless something like a common physics library like Havok, and clues as to where the user's extremities are (using something like the Leap!) then the library can take care of 'colliding' with the fingers or hand of the glove.

[Zaptruder]

Palmer say's he's worked with VR stuff and people rarely challenge the reality of the walls they see in VR. So if you reach out to touch something and you feel the pressure and temperature of it you will probably "buy" that it's really there and not try to push through it. Even in real life you don't usually press too hard on things just to see if they're actually solid, you just touch enough to feel the surface and how much give there is. The number of nodes would determine the granularity of how detailed the feel of the surface could be.

The paper referenced is really cool. There are no glove devices like that on the market are there? Also, that means you'd need some fairly detailed collision detection in software to get the right force and angle right for solid objects. I really like the concept, though I think it will be hard to get it supported easily unless something like a common physics library like Havok, and clues as to where the user's extremities are (using something like the Leap!) then the library can take care of 'colliding' with the fingers or hand of the glove.

Ideally this is the direction VR takes us - with haptic feedback gloves (as opposed to haptic feedback exoskeleton arms - which while cool, is unlikely to see much support) helping to provide some sort of touch based sensory feedback. I imagine it could be quite a compelling use case when paired with things like virtual keyboards - allowing the finger tips to feel each virtual button and react accordingly.

I was thinking that the gloves could be lined with some of the advanced haptic technology that is currently been shopped around - there are even patents for some sort of system that can actually physically raise a membrane creating a physically tactile button.

This should work pretty well where your finger brushing past the surface of objects go. Should also be pretty awesome for soft materials like hair or fur.

If the hand penetrates/clips through the object, then maybe the glove can contract and or create a sensation that doesn't occur in reality - such as a pulsating static/prickling sensation.

[Mark2036]

If the hand penetrates/clips through the object, then maybe the glove can contract and or create a sensation that doesn't occur in reality - such as a pulsating static/prickling sensation.

Adapt the glove for another body part and then the sex industry applications are endless....

[JMX]

Also I was using the kinect with a projector and using a wiimote for aiming. The wiimote had to constantly be manually recentered to keep orientation with the screen because we didn't have an absolute positon for the wiimote to do edge turning.

It should give a better experience with an HMD since both the kinect and wiimote should work much better when able to turn freely.

The kinect will probably still have some issues with occlusion though. Maybe optical tracking like with the kinect really needs at least one other camera to fix that.

[bobv5]

If the hand penetrates/clips through the object, then maybe the glove can contract and or create a sensation that doesn't occur in reality - such as a pulsating static/prickling sensation.

Adapt the glove for another body part and then the sex industry applications are endless....

Already been done. No, I'm not posting a link!

[pulsarinformatique]

Hi

I’ve been working in VR software (Web3D , stereo images ) and hardware ( force feeback design, designed and produced glasses, augmented reality ) from 1993 to 2005.

I left the area for a while but the new Occulus Rift headset opened new possibilities I would like to explore

I droped the treadmills options since it would require omnidirectonial treadmills to work with headsets since the user can turns in any orientation.

The kinect may be a solution: leaning forwards or backwards is intuitive enough for walking/running. leaning left and right would be stafing. Crouching is easy with the kinect too. Combined with the headset tracker, we would have all the 6 DOF required.

the main issu for me now is: can a kinect still track the body motion when the body is NOT facing the kinect?

since the user would wear a headset he rapidely can be in any orientation from the kinect. the kinect would map body gestures/positions to keyboard buttons

Can a Kinect track the motion of my arm when my body is 90° oriented from the Kinect? In this case one arm is partly hidden by the body.
What if I ‘m at 180° : will it understand all the motions are inverted ?

May be there is some piece of software on the net that already do the job?
thanks

cyril