Walking on the spot trackers

[Okta]

That cool. As mentioned before maybe the increased foot speed can be detected into a run but i wonder at the practicality for long sessions and increased head bob/equipment damage. Also a difference between walking and running is the length of the stride which we cant do walking on the spot, only raising our knees higher perhaps. I imagine a wizdish could do it maybe.
Never the less, I too would like to try it out at least for a short period to experience maximum immersion.

[JanVR]

For now, I'd like to have the following moves recognized by the Kinect instead of having to use a button: (note: imagine holding a Wiimote/Nunchuk gun in your hands that serves as a gamepad for the remaining actions)

- Walking / running (will be just same walking movement for now)
- Crouching / jumping
- Strafing
- Gun reload (maybe move left fist forward and backward)
- Open / activate / take (maybe move right fist forward)
- Weapon change?

What do you think? Any others? Let me know, because I will get this implementend in the code in the coming weeks.

[Lookforyourhands]

Wow JanVR that sounds totally amazing. I can't thank you enough for all
your hard work on this. I like all the ideas you want to implement,
especially the use ability. It would be really cool if you could map
all the most commonly used keys for fps games into gestures so it
will work with a wide variety. I found by doing that with my ps move
and navigator I can play almost any game without having to remap
keys all the time, maybe just one or two. Oh also something akin to
pressing the Esc key or pause like the way you do on Xbox by holding your
arm out on an angle. This is so much fun, and as fun as using
gun controllers is it would be super cool to be able to just use your body

[shent1080]

that would be ace, i'm watching kinect's on ebay right now

The only other command i can think of is THROW, if you could throw a grenade, in game, by throwing thin air...... that would be pretty cool.

Nice work JanVR

[JanVR]

All kudos though to Flassan of Wizdish, as I am using his original demo software app; just trying to implement some more options. The grenade throwing sounds great (perhaps move right hand above/behind your head); the Escape key is crucial too, since the keyboard is typically somewhat remote when you are standing up to play the game.

Ideally, long term, there should be a user interface that allows you to map certain key strokes to certain gestures, so that you can adjust settings for different games. There used to be a program called FAAST (out of USC, I believe the same lab where Palmer used to work) which allows you to do this (you can still download it); however, I don't think they use scan codes, so the key strokes don't work in most games.

[Flassan]

You're a true gentleman Jan. Actually I want to thank you guys for getting me interested in Kinect again and asking the right questions.
Now that Jan has ported the code to the much improved runtime and 1.5 SDK I thought about the suggestion to add running and realised it would only take a couple of easy lines of code. The results are remarkable. Just the ability to change speed lets you feel far more in control and thus immersed.
I'm starting to get a better appreciation of what's important and found out a lot from the two videos I've just shot below. It never occurred to me that full body immersion would actually give you an advantage when playing FPS games. I had hoped and now may have proved that standing and turning 360 helps overcome nausea induced through a disparity between visual and vestibular senses, but I didn't think you would be able to duck through doorways and shoot faster! I left the camera rolling just to see what would happen.

[youtube]http://www.youtube.com/watch?v=HE57GHM_UNk[/youtube]

I have this knack of turning off the video just before people make a gushing response as they remove the HMD. It's never the same second time but I thought I'd try to capture it:

[youtube]http://www.youtube.com/watch?v=zj2zZbLsWfE[/youtube]

[brantlew]

@Flassan. I like your rig for holding the cables above the head. That's going to be so important for these 360 situations. The Vuzix HMD is not known for causing nausea because it has such a low FOV - although I have heard of a couple people with extreme sensitivity that felt nausea. It would be better to find one of these sensitive people and have them try the game with the Vuzix first while standing still and then again on the Wizdish. That would back up your claim a bit more about nausea reduction. And of course, the real test is going to come with the Rift which has a completely different effect than the Vuzix.

[Flassan]

It would be better to find one of these sensitive people and have them try the game with the Vuzix first while standing still and then again on the Wizdish.

That's an excellent idea. I said 'may' because I didn't want it to sound like a claim but I ought to be more careful.
I have to admit I wasn't aware that FOV can affect nausea to that extent. Is it possible that less peripheral view of the real world might be the cause? The Vuzix FOV seems to work fairly well but is that because it matches the currently available games?
I was wondering how to suspend the cable and then someone on this forum mentioned Amazon were selling Mic booms for about $20. It works a treat!

[cybereality]

I will admit I did get a little sick with the Vuzix VR920 when I first got it, specifically when using the head-tracking in certain games. Using the headset just in 2D or 3D was fine when the tracking was off, but with the tracking it was making me sick at first. I think because the tracker was very jittery and not very accurate was why this happened.

[3dvison]

So is the running code now in Jan's app. ?

[JanVR]

Not yet. Flassan sent me his updated file, so with his permission, I will implement his code together with some of the other gesture recognitions we discussed.

[Flassan]

Of course. You don't need to ask.
I've attached the new file but the only change was:

Code: Select all

				if((vector > iSprint || vectorR > iSprint) && (vector < 400 || vectorR < 400))
				{
					iSprinting = iStopSprintingCount;
					GenerateKey(0x32);	// M key - configured in game as sprint
				}
				else 
				{
					if(iSprinting < 2)	// slug keyup event to prevent jerkiness
					{					
						if(iSprinting > 0)
						{
							iSprinting = 0;
							GenerateKeyUP(0x32);
						}
					}
					else
					{
						iSprinting--;
					}
				}

Sorry to burst into code but if people know how easy it is they may have a go themselves. Editing existing code can be done with some simple C knowledge so you don't have to be a full blown code monkey. Visual Studio Express and the Kinect SDk are free and you if there is one thing the Internet does well it's provide help with programming. On that point thanks to brantlew for his help with scan codes.

[Lookforyourhands]

Thank you for the new code and all your hard work Flassan!
You are really kind.

Edit: I was playing Mirrors Edge in VR tonight with my HMZ-T1 and the kinect
to track body with the updated code.. (Also moved my kinect for a better
view) and OMG I literally got motion sickness for the first time in my life
while gaming. It was pretty bad too I wanted to hurl. The combination of
running in game and head tracking actually made me feel like I was
moving forward and I had to stop playing. While interesting simply
because nothing like that has ever happened to me, I wonder if it's just
a case of the brain getting used to it or what. I'm not overly sensitive to
these types of things, so Im really surprised.

[Flassan]

Hi Lookforyourhands
I've only just seen your edit. I guess my first question is how are you currently adding head tracking to the HMZ-T1 as latency may be an issue?
If you are running on the spot, which your subconscious mind knows won't move you forward, I wonder if that is contributing to the motion sickness?
Also, are you sure the Kinect is best placed? I'm starting to suspect that inconsistent forward motion due to intermittent motion detection is an even bigger problem than latency. Certainly the arrangement in the video above genuinely appears to work.
There are a great many papers about HMDs causing motion sickness but I know from tests like the one in the video that it can be avoided. The curious thing is that the Vuzix 1200VR glasses aren't known for low latency or image persistence. There is a desperate need to understand this aspect of HMDs better. The contributory factors would appear to be latency, peripheral vision, 1:1 mapping, FOV, proximity of virtual objects, speed and frequency of body/head movements, familiarity with 3D gaming. Please add more if you can think of any!

[Lookforyourhands]

Hi Flassan.

I am using a Gyration Air Mouse Elite attached to my modded HMZ with velcro for head tracking.
It works very well and there's really hardly any perceptible latency. In fact it might be TOO good lol.
I'll try and make a video of what im seeing through the HMD if i can. I have been gaming for 30
years and have never experienced motion sickness like I did the other day. I think
it has to do with peripheral vision because when I play in total darkness (and with the absence of kinect)
I can play for hours without any dizziness or queasiness. Fire up mirrors edge with the kinect (obviously
the light in the room on to help the camera see my body) I can definitely see the floor
in my peripheral vision and it's quite strange. I have yet to try blocking out all light in this scenario to see
if that elimates motion sickness but I'm thinking it might. I know for a fact I'm not sensitive to these
types of things, but playing Mirrors Edge makes my palms sweat ! It's so freaky running towards
a ledge and jumping.. Maybe I'm just too 'into' it but I don't think this is a psychological reaction but
physiological. I won't lie I almost enjoy it because it makes it seem so much more 'real' but at the same time
I don't want to hurl in the middle of gaming lol. If VR goes mainstream there has gotta be a
standard that will keep the majority of people from experiencing motion sickness because no doubt
it will happen. Maybe it's a matter of the brain getting used to operating in that manner.. Give me another month
and we'll see if I still get motion sickness.. I think you're right on about the contributing factors. I think you have
mentioned most of them if I can think of any ill add. I wonder if because you are using an
on the spot treadmill it's having a different affect than walking in place??

[Flassan]

Content is another factor and Mirror's Edge could well be an ideal game to test with (I've just ordered it). Actually it makes me dizzy just thinking about it.
I would have thought some peripheral sight of the real world would help but you never know, it could do the reverse, and of course not everyone will respond the same way. Apparently an even more extreme game is Alien vs Predator, where you can run upside down without even knowing it.
We probably can't or shouldn't eliminate a sense of vertigo in VR that you would experience anyway in the real world. I suppose it's similar to the arcade vs simulation debate in mainstream gaming. Maybe a phrase like 'Virtual Fantasy' is needed? What would be nice though is to avoid nausea caused by the display technology so let's hope there are some Universities looking into this who are willing to share their research with the wider community.

[Okta]

Only guessing here but i think most of the nausea will come from the lack of absolute head tracking. While you are jogging on the spot, ducking, jumping and doing slight side to side movements during all this, the display will not register these small natural moves and it will build up to motion sickness. I was wondering if using the Kinect as described here could not also incorporate some small instance of absolute tracking to be translated to tiny in game digital/analogue moves?

Surely the Kinect can do absolute tracking in a small area? Could the Kinect track the x,y,z position of the head to then be somehow correlated to the facing and sent out as movements?

[JanVR]

Okta, theoretically that would be possible. However, not sure if it would help much, since for such small, subtle movements, the latency of the Kinect might be an issue.

[coresnake]

If the rumors are true about new Xbox coming out this year that means a new update for Kinect too presumably... might fix some of those latency issues.

[Flassan]

I agree. It's far too early to write-off Kinect and it's subsequent versions.
I was wondering if anyone has tried this? http://social.msdn.microsoft.com/Forums ... 39ad9ea021
It's normally best be wary of 'no programming experience required' claims but it might work for some solutions. I've not looked into it yet but would be interested to know if it has to work alone or could be used in conjunction with more standard code. For instance I may want to keep my 'if x is greater than n crouch otherwise stand' rule , but want it to learn a mimed gesture for reload.

[cybereality]

That link doesn't work. Do you need to have an account or something?

[Flassan]

Their site is here http://gesturestudio.ca/dev
I think the forum post truncated the link. Try:
http://social.msdn.microsoft.com/
Forums/en-US/kinectsdk/thread/
7babc355-76fb-43ba-9aac-4439ad9ea021

[Okta]

Okta, theoretically that would be possible. However, not sure if it would help much, since for such small, subtle movements, the latency of the Kinect might be an issue.

Ah yes of course, it would most likely make sickness worse if it isn't very accurate But it might be cool to try out for looking around corners type movement.

[STRZ]

- Weapon change?

Don't know if it's possible.. one handed = pistol, tight handling with both hands = machine gun, wide handling with both hands (one hand as supporting hand in front) = Rifle

If its doable, then it could be a fast way to select weapons, choosing them by performing the handling accordingly, tied to that weapon and how it's used IRL, avoiding cycling through the inventory.

[JanVR]

STRZ, that is a great thought. It should be possible, but probably requires some safety mechanisms/checks that prevent erroneous and often weapon changes (the user would have to do a pretty good job at keeping his hands in the appropriate position). Holding varying actual guns might make things easier. The Kinect can recognize each gun by the user's relative hand positions.

[STRZ]

I thought that it maybe could work due to muscle memory, the variation between weapons would suffer though, like identifying different weapons of the same family. Like 2 styles of pistols, rifles..

The selection itself would happen as you aim in my theory, so that you would have to perform the accurate gesture only while you're aiming. If you're not aiming, it wouldn't matter if you have the wrong weapon selected. The selecting process needs to happen very fast though for it to work. Could be a game design thingie wich you have no influence on if it's not hackable.

Maybe it would make sense to cycle through the weapon inventory if you're changing the gesture/handling on the fly. Could be a skill relative to VR to memorize the correct handling.

Edit:

The walking on a spot solutions are very interesting as well, i had that thought without when thinkink about a VR football (soccer) and tennis game, without knowing of you guys already have a nice solution

Another thought was a pair of sneakers, with some high precision loadcells build in, for the lazy non fitness people, you know.

The stuff wich is used in high-end simracing pedals. The idea is to control your character with weight balancing/shifting, and jumping by lifting your heel. Could be used for other stuff like virtual airboarding, skateboarding, snowboarding as well. I only haven't figured out how to include a change of the direction of the character, because my thought only focuses on forward, backwards and sideways by now, and jumping.

[bobv5]

For weapons a posibility is to only have one type, for example you could start a game with a single shot civilian version, then gradually upgrade it to a full auto military version with grenade launcher. Would allow for different/ more powerful shooting without having to worry about weapon selection.

[Okta]

For weapon selection i would prefer some simple hand/arm moves assigned to the mouse wheel up/down. Or just leave it up the players hand controller of choice and keep more gestures free for other key binding and maybe even some menu navigation.

[Mystify]

You could try a fusion of several approaches. Your hand position could select what type of weapon you want, and assuming that position would reselect the last weapon you had of that type. Another motion(drawing a weapon? Something subtler?) would switch weapons within that category.

[Okta]

New power supply arrived today. Got all excited. Went to run the Kinect 1.6sdk and quickly found out this is windows 7 only
Anyone had Kinect working with XP? Really pee's me off that its something like this that would finally force a change of OS.

edit- just got it working using faast http://projects.ict.usc.edu/mxr/faast/
haven't played with gestures yet but this skeleton tracking is ridiculously cool And occlusion isn't a problem with turning around and such so far.

[JanVR]

Okta, that's great. FAAST is a powerful program. I wrote that lab to get the source code, but never heard back... Unfortunately FAAST does not seem to use scan codes, so the key strokes don't work in games. Is your Kinect SDK working now?

[Okta]

Okta, that's great. FAAST is a powerful program. I wrote that lab to get the source code, but never heard back... Unfortunately FAAST does not seem to use scan codes, so the key strokes don't work in games. Is your Kinect SDK working now?

No SDK yet just primesense and faast. How do you mean the key strokes wont work in games? Ive tried some Quake 2 in window and full screen and the keys work fine. How the big battle is making a useful gesture set. I played with a 'lean to move' set and found out it didn't work so well when you face away from the K. Just now i tried a 'one knee in front of the other' for 'W' and that walks fine.

I cant see this being a practical solution rather than something to try out for a VR walk about.

[JanVR]

I am implementing some more gestures, but one big issue that I am running into is the free rotation of the user when wearing a HMD. With a VR HMD, you are not always facing the Kinect. So, detecting "moving your hand forward" becomes difficult, since the X-Y coordinates associated with "forward" are different if you are facing the Kinect or turned 90 degrees away from the Kinect. "Moving your hand above your head" or "Moving your hand away from your torso" are no problem, since they are independent of rotation. But anything specifically "forward" or "sideways" becomes complex, since forward become sideways when turning away from the Kinect. Not sure if I am being clear? Anyone have any suggestions?

[Okta]

I ran into the same problem with my leaning gestures. when you turn side on, leaning forwards became a strafe etc. I would have thought it kept absolute rotation because the skeleton itself seems to keep decent track of facing but... it was designed for facing a TV and most the software is aimed at that.

I am having all sorts of problems with jump/walk/crouch. MOst of the gestures options for each seem to blend into the others- leg angle, foot to hip distance, "jump" hight triggering to easily or not at all etc.

The demo videos of yours are far above what i achieved so far. I will continue to play with it but I at least have walking (on foot above the other) working half decent. I will soon start on my left hand controller mod hopefully.

Another annoying thing, i need to stand about 12 feet away from the Kinect just to get body height in frame, not including raising an arm, What were they thinking....

[3dvison]

Just a Quick, What would you do question ???

Right now I can only buy one of the two...
Should I get :
Wireless Xbox 360 controlller ?
OR
Kinect ?

For showing off a Diy Rift which would you go with ?
Which one is more fun, even if Kinect is more risky for new Rift users, like tripping over ones virtual foot onto ones non-virtual REAL Face..I want which ever one is more fun, not more safe..LOL

[Okta]

Just a Quick, What would you do question ???

Right now I can only buy one of the two...
Should I get :
Wireless Xbox 360 controlller ?
OR
Kinect ?

For showing off a Diy Rift which would you go with ?
Which one is more fun, even if Kinect is more risky for new Rift users, like tripping over ones virtual foot onto ones non-virtual REAL Face..I want which ever one is more fun, not more safe..LOL

For ease of use and simplicity i would get the wireless controller.

[3dvison]

Another annoying thing, i need to stand about 12 feet away from the Kinect just to get body height in frame, not including raising an arm, What were they thinking....

Okta, have you seen this ?

http://www.amazon.com/Zoom-Kinect-Xbox- ... B0050SYS5A

[geekmaster]

... Another annoying thing, i need to stand about 12 feet away from the Kinect just to get body height in frame, not including raising an arm, What were they thinking....

you can buy a snap-on wide angle lens set that lets you play 40-percent closer to the Kinect. One of them is the Nyko Zoom:



There are reports that one of the posted videos showed Palmer Luckey with a Kinect on his desk, which had a Nyko Zoom attachment.

[Flassan]

I bought the Nyko Zoom quite a while ago to test it. It didn't seem to be as reliable but come to think of it that was with the original runtime. I'll try it again and let you know.

[Luucid]

For the movement speed of our physical legs reciprocating into walk/run, I think it would be cool if the rl movement speed directly correlated into the game instead of just "walk" or "run". Obviously harder to do at this point but with the new VR tech coming out I think things like this need to happen.

Even if it could detect a heel/toe type movement as (WALK) and then when you move your weight forward, onto the ball of your foot and toe area have Kinect detect it as a (RUN).

'one knee in front of the other' for 'W' and that walks fine.

For a while I've been thinking about the integration of gestures that aren't as realistic as many that have been proposed. Like putting one foot in front of the other to walk, raising the heel to run, diagonal movement by pointing toes in the direction you want to travel, stand on toes to jump, etc... there are so many combinations that our VR maneuvering could be vastly improved compared to games out now and that wouldn't be possible doing realistic movements in your home. Also, a really low tech solution to possibly running into your wall with a Rift would be to stand on a circular mat with varying degrees of hardness so you know where you are. Like in the center its hard and gets squishy the farther from the center or vise versa.

One thing I really want is the ability to see my hands when I raise them and to manipulate things with my fingers. Ideally I would want something like this http://www.finger-motion-capture.com/ but I'm pretty sure that's too expensive (SAO style menu interaction would be sweet). Besides that I think a good alternative to buttons would be to bring a finger to your thumb which would be like 20 combinations using both hands if I'm not mistaken (maybe a Fable style HUD with these combos?). I think this could be done relatively cheaply if you don't use some mocap system and instead use a button-like system.
^ I guess I'm basically saying I'd rather use the body as a controller with tons of buttons instead of going for maximum immersion. The Rift is already going to be immersive enough for me. Besides the generic movements I mentioned two paragraphs up, I think there should be tons of other gestures in future games for maneuvers that haven't been utilized in gaming yet.

For inventory items, objects could be worn and held to equip.

I'd love to get some exercise while gaming but I haven't seen any low-cost solutions that I feel drawn to-then again I haven't tried any.

Also, along with default control schemes, I think there should be an easy way for a player to customize their controls-like an interface with a 3d dude or just entering them physically.

Can you move a character diagonally on the WizDish?