Walking on the spot trackers

[3dvison]

No apologies needed JanVR,
My sentance structure is horrible.
Somtimes when I read one of my own post, even I wonder, what the helz I was trying to say ???

[3dvison]

Would there be a way to add run/sprint to the app ? I think most games have a walk and a run button and you have to press both to run/sprint..
Could you do somthing like have the Kinect detect closing your hand into a fist as a (RUN) button press, which then would be combind with the (W) Walking, as the combo key press needed to run/sprint ?

[JanVR]

That is a good suggestion. I definitely plan to add some hand/arm gestures (for example, to re-load or throw a grenade). I'll look into the sprint feature as well.

What other hand/arm gestures or other movements would be useful?

[brantlew]

I think sometimes "gesture" commands can be more problematic than useful - especially if the gesture is not analogous to the real world action. In this instance the "closed fist" indicates a run mode. The gesture is non-intuitive and doesn't really match the real world action so its really no better than a button. But a button is vastly simpler to reliably detect so in this instance I think the button would be the better choice.

[Flassan]

The simplest way I can think of is to move your legs faster. The WizDish Kinect demo program uploaded by Jan samples the x and z values of the ankles every 10ms, checks that they've changed enough not to be noise, then sends a 'w' keydown message to make you move forward. If that value was higher than a second threshold value it could send a 'run' command, assuming the game has that feature.
Another feature that would be great to include is to derive directional information from the same x and z values to apply a corresponding amount of turn, either for strafing or for when there is no head-tracking (e.g. for use in a CAVE). I've only just found out that version 1.5 of the Kinect SDK includes some face recognition. Might that be enough to tell if you are facing towards or away from the Kinect, which cannot be inferred from the x and z? Then it would work through the full 360 degrees.

[brantlew]

The WizDish Kinect demo program uploaded by Jan samples the x and z values of the ankles every 10ms, checks that they've changed enough not to be noise, then sends a 'w' keydown message to make you move forward.

There is a lot of information available in the returned skeletal data. More sophisticated algorithms should be able to integrate the skeletal data over time and generate a huge variety of motion commands. It would not be too difficult to convert leg motions into accurate analog motion and speed commands - even going so far as to model the non-constant gait of the person. Of course, the game would need a flexible interface to support this. Some games with analog gamepad controls could be interfaced this way.

[3dvison]

I would like to go button-less, so anything other than a button press would be fine with me.
The fist idea was just off the top of my head. I play alot of sports and when I take off in a sprint, for just a split second, my hand does clinch before going into a more relaxed open handed hand shake type position.
Even if it could detect a heel/toe type movment as (WALK) and then when you move your weight forward, onto the ball of your foot and toe area have Kinect detect it as a (RUN).

But the best would be what Flassan talked about, and just have the Kinect detect that your legs are moving faster and register it as a run button press.

[shent1080]

it seems like kinect is doing a great job with skeleton tracking but what is everyone using for head tracking, can imu's/ir tracking be used alongside the kinect sdk, or better yet be implemented into the software?

[Okta]

Trying to do away with buttons at this stage isn't going to make for smooth sailing. As Brentlew said, use the Kinect where it shines for gross intuitive moves, and hold a wireless controller or 2 for all the other multiple inputs required.

[3dvison]

I ment no buttons for walking & running.
To me running is a gross intuitive move.
And there is such a gross difference between walking and running throughout the body, that I would think one of those difference could be keyed on for the Kinect to detect as a run.

[Okta]

That cool. As mentioned before maybe the increased foot speed can be detected into a run but i wonder at the practicality for long sessions and increased head bob/equipment damage. Also a difference between walking and running is the length of the stride which we cant do walking on the spot, only raising our knees higher perhaps. I imagine a wizdish could do it maybe.
Never the less, I too would like to try it out at least for a short period to experience maximum immersion.

[JanVR]

For now, I'd like to have the following moves recognized by the Kinect instead of having to use a button: (note: imagine holding a Wiimote/Nunchuk gun in your hands that serves as a gamepad for the remaining actions)

- Walking / running (will be just same walking movement for now)
- Crouching / jumping
- Strafing
- Gun reload (maybe move left fist forward and backward)
- Open / activate / take (maybe move right fist forward)
- Weapon change?

What do you think? Any others? Let me know, because I will get this implementend in the code in the coming weeks.

[Lookforyourhands]

Wow JanVR that sounds totally amazing. I can't thank you enough for all
your hard work on this. I like all the ideas you want to implement,
especially the use ability. It would be really cool if you could map
all the most commonly used keys for fps games into gestures so it
will work with a wide variety. I found by doing that with my ps move
and navigator I can play almost any game without having to remap
keys all the time, maybe just one or two. Oh also something akin to
pressing the Esc key or pause like the way you do on Xbox by holding your
arm out on an angle. This is so much fun, and as fun as using
gun controllers is it would be super cool to be able to just use your body

[shent1080]

that would be ace, i'm watching kinect's on ebay right now

The only other command i can think of is THROW, if you could throw a grenade, in game, by throwing thin air...... that would be pretty cool.

Nice work JanVR

[JanVR]

All kudos though to Flassan of Wizdish, as I am using his original demo software app; just trying to implement some more options. The grenade throwing sounds great (perhaps move right hand above/behind your head); the Escape key is crucial too, since the keyboard is typically somewhat remote when you are standing up to play the game.

Ideally, long term, there should be a user interface that allows you to map certain key strokes to certain gestures, so that you can adjust settings for different games. There used to be a program called FAAST (out of USC, I believe the same lab where Palmer used to work) which allows you to do this (you can still download it); however, I don't think they use scan codes, so the key strokes don't work in most games.

[Flassan]

You're a true gentleman Jan. Actually I want to thank you guys for getting me interested in Kinect again and asking the right questions.
Now that Jan has ported the code to the much improved runtime and 1.5 SDK I thought about the suggestion to add running and realised it would only take a couple of easy lines of code. The results are remarkable. Just the ability to change speed lets you feel far more in control and thus immersed.
I'm starting to get a better appreciation of what's important and found out a lot from the two videos I've just shot below. It never occurred to me that full body immersion would actually give you an advantage when playing FPS games. I had hoped and now may have proved that standing and turning 360 helps overcome nausea induced through a disparity between visual and vestibular senses, but I didn't think you would be able to duck through doorways and shoot faster! I left the camera rolling just to see what would happen.

[youtube]http://www.youtube.com/watch?v=HE57GHM_UNk[/youtube]

I have this knack of turning off the video just before people make a gushing response as they remove the HMD. It's never the same second time but I thought I'd try to capture it:

[youtube]http://www.youtube.com/watch?v=zj2zZbLsWfE[/youtube]

[brantlew]

@Flassan. I like your rig for holding the cables above the head. That's going to be so important for these 360 situations. The Vuzix HMD is not known for causing nausea because it has such a low FOV - although I have heard of a couple people with extreme sensitivity that felt nausea. It would be better to find one of these sensitive people and have them try the game with the Vuzix first while standing still and then again on the Wizdish. That would back up your claim a bit more about nausea reduction. And of course, the real test is going to come with the Rift which has a completely different effect than the Vuzix.

[Flassan]

It would be better to find one of these sensitive people and have them try the game with the Vuzix first while standing still and then again on the Wizdish.

That's an excellent idea. I said 'may' because I didn't want it to sound like a claim but I ought to be more careful.
I have to admit I wasn't aware that FOV can affect nausea to that extent. Is it possible that less peripheral view of the real world might be the cause? The Vuzix FOV seems to work fairly well but is that because it matches the currently available games?
I was wondering how to suspend the cable and then someone on this forum mentioned Amazon were selling Mic booms for about $20. It works a treat!

[cybereality]

I will admit I did get a little sick with the Vuzix VR920 when I first got it, specifically when using the head-tracking in certain games. Using the headset just in 2D or 3D was fine when the tracking was off, but with the tracking it was making me sick at first. I think because the tracker was very jittery and not very accurate was why this happened.

[3dvison]

So is the running code now in Jan's app. ?

[JanVR]

Not yet. Flassan sent me his updated file, so with his permission, I will implement his code together with some of the other gesture recognitions we discussed.

[Flassan]

Of course. You don't need to ask.
I've attached the new file but the only change was:

Code: Select all

				if((vector > iSprint || vectorR > iSprint) && (vector < 400 || vectorR < 400))
				{
					iSprinting = iStopSprintingCount;
					GenerateKey(0x32);	// M key - configured in game as sprint
				}
				else 
				{
					if(iSprinting < 2)	// slug keyup event to prevent jerkiness
					{					
						if(iSprinting > 0)
						{
							iSprinting = 0;
							GenerateKeyUP(0x32);
						}
					}
					else
					{
						iSprinting--;
					}
				}

Sorry to burst into code but if people know how easy it is they may have a go themselves. Editing existing code can be done with some simple C knowledge so you don't have to be a full blown code monkey. Visual Studio Express and the Kinect SDk are free and you if there is one thing the Internet does well it's provide help with programming. On that point thanks to brantlew for his help with scan codes.

[Lookforyourhands]

Thank you for the new code and all your hard work Flassan!
You are really kind.

Edit: I was playing Mirrors Edge in VR tonight with my HMZ-T1 and the kinect
to track body with the updated code.. (Also moved my kinect for a better
view) and OMG I literally got motion sickness for the first time in my life
while gaming. It was pretty bad too I wanted to hurl. The combination of
running in game and head tracking actually made me feel like I was
moving forward and I had to stop playing. While interesting simply
because nothing like that has ever happened to me, I wonder if it's just
a case of the brain getting used to it or what. I'm not overly sensitive to
these types of things, so Im really surprised.

[Flassan]

Hi Lookforyourhands
I've only just seen your edit. I guess my first question is how are you currently adding head tracking to the HMZ-T1 as latency may be an issue?
If you are running on the spot, which your subconscious mind knows won't move you forward, I wonder if that is contributing to the motion sickness?
Also, are you sure the Kinect is best placed? I'm starting to suspect that inconsistent forward motion due to intermittent motion detection is an even bigger problem than latency. Certainly the arrangement in the video above genuinely appears to work.
There are a great many papers about HMDs causing motion sickness but I know from tests like the one in the video that it can be avoided. The curious thing is that the Vuzix 1200VR glasses aren't known for low latency or image persistence. There is a desperate need to understand this aspect of HMDs better. The contributory factors would appear to be latency, peripheral vision, 1:1 mapping, FOV, proximity of virtual objects, speed and frequency of body/head movements, familiarity with 3D gaming. Please add more if you can think of any!

[Lookforyourhands]

Hi Flassan.

I am using a Gyration Air Mouse Elite attached to my modded HMZ with velcro for head tracking.
It works very well and there's really hardly any perceptible latency. In fact it might be TOO good lol.
I'll try and make a video of what im seeing through the HMD if i can. I have been gaming for 30
years and have never experienced motion sickness like I did the other day. I think
it has to do with peripheral vision because when I play in total darkness (and with the absence of kinect)
I can play for hours without any dizziness or queasiness. Fire up mirrors edge with the kinect (obviously
the light in the room on to help the camera see my body) I can definitely see the floor
in my peripheral vision and it's quite strange. I have yet to try blocking out all light in this scenario to see
if that elimates motion sickness but I'm thinking it might. I know for a fact I'm not sensitive to these
types of things, but playing Mirrors Edge makes my palms sweat ! It's so freaky running towards
a ledge and jumping.. Maybe I'm just too 'into' it but I don't think this is a psychological reaction but
physiological. I won't lie I almost enjoy it because it makes it seem so much more 'real' but at the same time
I don't want to hurl in the middle of gaming lol. If VR goes mainstream there has gotta be a
standard that will keep the majority of people from experiencing motion sickness because no doubt
it will happen. Maybe it's a matter of the brain getting used to operating in that manner.. Give me another month
and we'll see if I still get motion sickness.. I think you're right on about the contributing factors. I think you have
mentioned most of them if I can think of any ill add. I wonder if because you are using an
on the spot treadmill it's having a different affect than walking in place??

[Flassan]

Content is another factor and Mirror's Edge could well be an ideal game to test with (I've just ordered it). Actually it makes me dizzy just thinking about it.
I would have thought some peripheral sight of the real world would help but you never know, it could do the reverse, and of course not everyone will respond the same way. Apparently an even more extreme game is Alien vs Predator, where you can run upside down without even knowing it.
We probably can't or shouldn't eliminate a sense of vertigo in VR that you would experience anyway in the real world. I suppose it's similar to the arcade vs simulation debate in mainstream gaming. Maybe a phrase like 'Virtual Fantasy' is needed? What would be nice though is to avoid nausea caused by the display technology so let's hope there are some Universities looking into this who are willing to share their research with the wider community.

[Okta]

Only guessing here but i think most of the nausea will come from the lack of absolute head tracking. While you are jogging on the spot, ducking, jumping and doing slight side to side movements during all this, the display will not register these small natural moves and it will build up to motion sickness. I was wondering if using the Kinect as described here could not also incorporate some small instance of absolute tracking to be translated to tiny in game digital/analogue moves?

Surely the Kinect can do absolute tracking in a small area? Could the Kinect track the x,y,z position of the head to then be somehow correlated to the facing and sent out as movements?

[JanVR]

Okta, theoretically that would be possible. However, not sure if it would help much, since for such small, subtle movements, the latency of the Kinect might be an issue.

[coresnake]

If the rumors are true about new Xbox coming out this year that means a new update for Kinect too presumably... might fix some of those latency issues.

[Flassan]

I agree. It's far too early to write-off Kinect and it's subsequent versions.
I was wondering if anyone has tried this? http://social.msdn.microsoft.com/Forums ... 39ad9ea021
It's normally best be wary of 'no programming experience required' claims but it might work for some solutions. I've not looked into it yet but would be interested to know if it has to work alone or could be used in conjunction with more standard code. For instance I may want to keep my 'if x is greater than n crouch otherwise stand' rule , but want it to learn a mimed gesture for reload.

[cybereality]

That link doesn't work. Do you need to have an account or something?

[Flassan]

Their site is here http://gesturestudio.ca/dev
I think the forum post truncated the link. Try:
http://social.msdn.microsoft.com/
Forums/en-US/kinectsdk/thread/
7babc355-76fb-43ba-9aac-4439ad9ea021

[Okta]

Okta, theoretically that would be possible. However, not sure if it would help much, since for such small, subtle movements, the latency of the Kinect might be an issue.

Ah yes of course, it would most likely make sickness worse if it isn't very accurate But it might be cool to try out for looking around corners type movement.

[STRZ]

- Weapon change?

Don't know if it's possible.. one handed = pistol, tight handling with both hands = machine gun, wide handling with both hands (one hand as supporting hand in front) = Rifle

If its doable, then it could be a fast way to select weapons, choosing them by performing the handling accordingly, tied to that weapon and how it's used IRL, avoiding cycling through the inventory.

[JanVR]

STRZ, that is a great thought. It should be possible, but probably requires some safety mechanisms/checks that prevent erroneous and often weapon changes (the user would have to do a pretty good job at keeping his hands in the appropriate position). Holding varying actual guns might make things easier. The Kinect can recognize each gun by the user's relative hand positions.

[STRZ]

I thought that it maybe could work due to muscle memory, the variation between weapons would suffer though, like identifying different weapons of the same family. Like 2 styles of pistols, rifles..

The selection itself would happen as you aim in my theory, so that you would have to perform the accurate gesture only while you're aiming. If you're not aiming, it wouldn't matter if you have the wrong weapon selected. The selecting process needs to happen very fast though for it to work. Could be a game design thingie wich you have no influence on if it's not hackable.

Maybe it would make sense to cycle through the weapon inventory if you're changing the gesture/handling on the fly. Could be a skill relative to VR to memorize the correct handling.

Edit:

The walking on a spot solutions are very interesting as well, i had that thought without when thinkink about a VR football (soccer) and tennis game, without knowing of you guys already have a nice solution

Another thought was a pair of sneakers, with some high precision loadcells build in, for the lazy non fitness people, you know.

The stuff wich is used in high-end simracing pedals. The idea is to control your character with weight balancing/shifting, and jumping by lifting your heel. Could be used for other stuff like virtual airboarding, skateboarding, snowboarding as well. I only haven't figured out how to include a change of the direction of the character, because my thought only focuses on forward, backwards and sideways by now, and jumping.

[bobv5]

For weapons a posibility is to only have one type, for example you could start a game with a single shot civilian version, then gradually upgrade it to a full auto military version with grenade launcher. Would allow for different/ more powerful shooting without having to worry about weapon selection.

[Okta]

For weapon selection i would prefer some simple hand/arm moves assigned to the mouse wheel up/down. Or just leave it up the players hand controller of choice and keep more gestures free for other key binding and maybe even some menu navigation.

[Mystify]

You could try a fusion of several approaches. Your hand position could select what type of weapon you want, and assuming that position would reselect the last weapon you had of that type. Another motion(drawing a weapon? Something subtler?) would switch weapons within that category.

[Okta]

New power supply arrived today. Got all excited. Went to run the Kinect 1.6sdk and quickly found out this is windows 7 only
Anyone had Kinect working with XP? Really pee's me off that its something like this that would finally force a change of OS.

edit- just got it working using faast http://projects.ict.usc.edu/mxr/faast/
haven't played with gestures yet but this skeleton tracking is ridiculously cool And occlusion isn't a problem with turning around and such so far.