Walking on the spot trackers

Post by **Okta** » Mon Nov 19, 2012 10:09 pm

Considering we cant expect a home ODT any time soon (aside from possibly the wizdish) i am hoping when the Rift arrives to use some sort of walking on the spot,ducking,jumping tracker. Although systems like Brentlews outdoor system is the next best option to a perfectly working ODT, for myself i cant see a lot of times available when i can use that system in a large area and would get more use time out of the on his on the spot portion of his system for home use so i list if below as an option.
I think it is desirable that our solution also supports strafing, diagonal and backwards movement although that ups the ante for complication considerably. Ideally it would be cheap and easy to set up/strap on system made from available parts.
I have been thinking of a few designs but am not at all happy with my ideas (or ability to implement them) so far.
I have played around with wii controllers and have found them very hit and miss using accelerometer detection. Hopefully a better scripter than I has a workable system already but i haven't found any searching the net. The first solution would be to use something like a wiimote to activate 'w' whenever its sensors tip a set value but that is a touch limited. Next option is the use that wii sensor combined with the nunchuck directional stick to go WASD when it is pushed AND movement from the legs is detected. From there we can script so that whenever the accelerometers detect movement over a certain level higher than the walk level in a given period 'Shift' (if mapped in game as your run key) is also held down to turn the WASD into running in game.

On the forums and elsewhere are already some walking (on the spot) solutions so I will list some here to see what is/soon will be available and details hopefully about how they work and what we can expect.

1. Brantlew's Red Rovr http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=14069
Brant uses a large area positional system but also on board walking tracking for on the spot walking detection.

2. The Wizdish http://www.mtbs3d.com/phpBB/viewtopic.p ... ish#p85530
Creator (Flassan on the forum) says in his demo he used a Kinect? to read his walking gesture but this might only work walking forwards (W)?

Please add anything you have yourself or links to pages, i really want and hope others want a semi decent solution come rift time

Post by **brantlew** » Tue Nov 20, 2012 12:56 am

Personally, I think for stationary indoor areas the best performance can be gotten from optical IR trackers. Something like MemeBox's optical system. (Although personally I would like to see a Wiimote branch of that source code) However with performance comes complexity. Setting up these systems and hooking them into the game is no picnic.

Probably the next best solution is something like the Kinect. Latency is an issue, but it's a very general solution (allowing all types of motion) and a lot easier to setup than sticking a bunch of IR's all over your body.

The simplest and easiest to setup but the least flexible are inertial systems. You can hook into these devices (ie. Wiimotes) with relatively simple GlovePIE scripts. Getting dependable multi-directional movement is hard (damn near impossible), but if you just want unidirectional motion detection it's not really hard at all. In fact you don't even need to use a separate device. You could just use the same sensor that is being used for head-tracking for an all-in-one head-tracking/motion detection system. I'm hoping the Rift SDK allows hooking into the tracker feed because this would be a cool little project.

Post by **Okta** » Tue Nov 20, 2012 6:15 am

Going on your post i have looked at some Kinect videos and it looks to be very appropriate with some caveats. The Kinect is designed so you must face the device for skeletal tracking, so in theory you might need to use 2 Kinects at 180 degrees or 3 Kinects at 120 degrees or more. From some brief browsing there are few problems with this.

1. Kinects must be plugged into separte usb host controllers on 1 PC to be recognised.

Easy to achieve with most PC's having more than one host or just use a USB hub for 1 and direct to PC for the other.

2. Multiple Kinects on one PC will only read one skeletal model at a time.
Without first hand experience i dont know how much trouble it is to set the software to read the best facing device or perhaps rig a magnetometer that will switch between devices as you face each.

Ok its possible http://www.youtube.com/watch?v=msRtIZX529Q unfortunately their basic software is $595.

3. Each Kinect shines and reads its own matrix of IR dots that could interfere with each other.
Use polarising filters at different orientation for each device?

http://openkinect.org/wiki/FAQ

Can optical polarization (or other tricks) be used to combine two Kinect sensors viewing the same scene from different angles?

Unfortunately, polarization is not generally preserved when reflecting off an object. Special surfaces can be used that preserve polarization, but the concept of tagging each Kinect's IR field by polarizing the light does not appear to be feasible. Initial tests using two simultaneous Kinect sensors suggest that they may not severely interfere with one another.

So not a big problem.

4. Cost is getting up there for multiple Kinects.

Post by **MemeBox** » Tue Nov 20, 2012 7:32 am

It seems that multiple kinects do not catastrophically interfere. But I came across this a while ago, a simple solution for enabling lots of kinects to operate in the same area. It also improves the noise in the signal.

http://hackaday.com/2012/08/13/building ... ger-motor/

Getting the skeleton from multiple kinects looks difficult. There is an open source tracker, which presumably would do the trick: http://vimeo.com/44739007, but it's made to run on linux. I'd say read the academic paper and replicate the algorithm, but it looks very involved and getting something like this to run smoothly is a lot about tuning...
There looks to be something in the pipeline over at pointclouds.org, possibly available in the main code branch judging from this convo:
http://www.pcl-users.org/Skeleton-Track ... 39370.html
This looks good since it runs on the GPU (and therefore liable to be smooth as silk) and makes use of a generic point cloud rather than a depth map. So this means you could generate your pointcloud with a million kinects, widgets, bats, whatever you fancy. I also came across this, which uses PCL (Pointclouds.org) https://github.com/jeffbryner/pclsceleton.

You could buy another PC? lol.

I'm intending to use multiple kinects / kinect2 s with the vibration technique in conjunction with an optical tracker, to allow you to view your body in the simulation, or augment/interact with an existing space. Lets see how obsessed I get!

Good luck, there definitely needs to be something like this for games where most people don't want to be wandering around their homes!

Post by **JanVR** » Tue Nov 20, 2012 8:13 am

I have done a lot of testing with motion recognition via Kinect and it works really well. I don't believe you would need more than 1 Kinect. The Kinect tracks the body accurately, regardless of your angle (you do not need to face the Kinect; you could be turned away 180 degrees without issues). Latency is not a problem either (the 0.5 sec or so is not noticable when you are walking / running).

Flassan (from Wizdish) sent me his adapted software for walking recognition, and I made a few changes to it to expand the functionality (it works for walking-in-place as well; no need for a Wizdish). It is basically the SkeletalViewer app that comes for free with the free SDK, with built-in additions for walking recognition, jumping, crouching, etc. I am still working on building in strafing, walking backwards, and even arm gestures for reloading, changing weapons, etc. The possibilities are limitless.

If this would be useful, I can upload the software so that everyone can use, improve and expand the functionality?

Post by **brantlew** » Tue Nov 20, 2012 8:19 am

The jitter-Kinect is the way that most people have solved the Kinect interference problem and seems to work surprisingly well.

As far as Kinect skeletal tracking, the implementations that seem to perform best are the OpenNI ones. I don't know how stable this skeleton remains if this guy faced away from the camera for long, but he is able to perform 360 turns seemingly without any loss of tracking.

[youtube]http://www.youtube.com/watch?v=nr8vgCnb9_0[/youtube]

Post by **Okta** » Tue Nov 20, 2012 9:01 am

JanVR wrote:I have done a lot of testing with motion recognition via Kinect and it works really well. I don't believe you would need more than 1 Kinect. The Kinect tracks the body accurately, regardless of your angle (you do not need to face the Kinect; you could be turned away 180 degrees without issues). Latency is not a problem either (the 0.5 sec or so is not noticable when you are walking / running).

Flassan (from Wizdish) sent me his adapted software for walking recognition, and I made a few changes to it to expand the functionality (it works for walking-in-place as well; no need for a Wizdish). It is basically the SkeletalViewer app that comes for free with the free SDK, with built-in additions for walking recognition, jumping, crouching, etc. I am still working on building in strafing, walking backwards, and even arm gestures for reloading, changing weapons, etc. The possibilities are limitless.

If this would be useful, I can upload the software so that everyone can use, improve and expand the functionality?

If you can share this would be great. I was just a little concerned about occlusion, from the videos i have seen of the skeletal viewer it seems to freak out as soon as you occlude anything instantly. If your testing isnt doing this then i need to get myself a Kinect.. was there a pc version that came out a while ago or id the 360 version still ok?

Post by **Okta** » Tue Nov 20, 2012 9:02 am

brantlew wrote:The jitter-Kinect is the way that most people have solved the Kinect interference problem and seems to work surprisingly well.

As far as Kinect skeletal tracking, the implementations that seem to perform best are the OpenNI ones. I don't know how stable this skeleton remains if this guy faced away from the camera for long, but he is able to perform 360 turns seemingly without any loss of tracking.

[youtube]http://www.youtube.com/watch?v=nr8vgCnb9_0[/youtube]

Nice find that is very impressive. Will have to try this if/when i can score a Kinect.

Post by **JanVR** » Tue Nov 20, 2012 9:29 am

You can still use the Xbox version - it works equally well on a pc (just need to have the USB cable, which comes with the standalone version). The SDK is free and works with both PC and Xbox versions; there is simply a warning that, if you were to sell your product commercially, you would need to update to the PC version to make it work, and get a license.

The latest SDK 1.6 works surprisingly well; I have not had many issues with occlusion. Even when my feet are not visible on the tracking screen in the skeletal viewer application, they are still tracked by the Kinect (has quite a large viewing field, even from close).

I will upload the software tonight or tomorrow. Glad to have some more folks interested in Kinect development. I believe it is a powerful tool that is easy to use.

Post by **brantlew** » Tue Nov 20, 2012 10:15 am

Thanks for the info JanVR. It's good to know that the Kinect SDK can adequately handle this problem. This is probably going to be the simplest and best solution for stationary 360 tracking.

Edit: Oh and for those that may be worried about Kinect latency - latency is a huge issue for head tracking but not as much for motion tracking. Arm motions are still a bit sensitive to latency but walk detection is far more forgiving. Even a 1/4 second latency feels "pretty good" and 100ms latency on walk detection feels damn near instantaneous.

Post by **JanVR** » Wed Nov 21, 2012 5:56 pm

I have uploaded the project to DropBox:

https://www.dropbox.com/s/mvpaesp7awama ... VS2012.zip

Note that this version was built using the latest Kinect SDK 1.6 and Visual Studio 2012 (although this should work with previous versions as well). Let me know if this works.

The program currently recognizes forward walking, jumping, and crouching. I use this in conjunction with Vuzix 920 glasses (as such, you walk in the direction that you look).

I've been testing mainly with Crysis Warhead, since that game can be run in non-full screen mode (the Kinect application runs in the background while running the game). However, the gesture recognition application shuts off when a game is switched to full screen mode (as many games do). There is probably a way to keep the Kinect app from shutting down, just haven't figured it out yet (I am not a great programmer, by any means).

I am working on incorporating sideways stepping, backwards stepping, and even hand and arm gestures, which would be great to control a game while having a Rift on. I should probably post a video sometime in the future. The level of immersion is pretty neat.

Let me know if this is useful. I'll keep everyone posted as well.

Post by **Flassan** » Thu Nov 22, 2012 2:26 am

Thanks JanVR, you have renewed my interest in the Kinect and I will definitely try your software.
If you have a tripod I found this adapter useful:
http://www.amazon.co.uk/gp/product/B004 ... 05_s00_i00

Post by **3dvison** » Mon Nov 26, 2012 4:47 pm

Flassan wrote:Thanks JanVR, you have renewed my interest in the Kinect and I will definitely try your software.

Which software is JanVR using ? Is it the iPi Basic software package ?

Post by **JanVR** » Mon Nov 26, 2012 6:13 pm

This is based on the free Kinect SDK 1.6 (free on Windows website). Flassan got it to work on his system - let me know if this works for you too.

Post by **Lookforyourhands** » Thu Dec 27, 2012 5:03 am

JanVR, would you please be so kind as to post instructions on how to setup and use
the Kinect for crouching and jumping as you described ??

I can get the Skeletal Viewer to work but how to I translate my body movement to a keyboard command ?

Post by **JanVR** » Fri Dec 28, 2012 10:47 am

If you use the SkeletalViewer app that I posted, it should automatically translate your body movements to key commands. The code is embedded in SkeletalViewer.cpp, the Nui_DrawSkeleton method of NuiImpl.cpp and SkeletalViewer.h. As of now, walking induces a "W" command, crouching a "C" command, and jumping a "space".

Try to open Wordpad, and see if these keys appear in the text window when you are using the SkeletonViewer app (the app should be active, showing your skeleton in action).

Let me know if this works? Otherwise, let me know what you are seeing (or not seeing) - I'll try to provide all guidance needed.

Post by **Lookforyourhands** » Tue Jan 01, 2013 7:03 am

Thanks. I'm at work now but I'll give this a shot later today and post
results. All I want is to be able to crouch and jump in games.. Thanks again
JanVR!

Edit: COOL ! I followed your instructions and it worked perfectly. Got the
characters to show up in notepad. Fired up Crysis 2 and it works!! Mounted
my kinect on the wall above my setup.. Tracks surprisingly better than I thought
I can even turn 360 degrees from the camera and it still tracks
my movements. SO fun! I almost fell over when I crouched for the first time
because I was still 'walking' in the game and it was disorienting. Lol
The only thing I noticed after a few minutes of testing was that I have to dip my
knees down pretty far and come back up quickly for it to recognize jump but it works
however due to wearing an HMD it's somewhat awkward. Is there a way to
tweak the sensitivity or something ? Thanks again hey, this is really something.

Post by **3dvison** » Tue Jan 01, 2013 12:16 pm

So on a Diy RIFT, will a game like Crysis fill the whole screen when playing, I see talk about only being able to run a game in Non-Full screen mode when doing body tracking ?
Is this true for all games, if you are going to use the Kinect for body tracking ?

Lookforyourhands, does Crysis have a native side by side mode, so no third party 3D driver is needed for side by side 3D with a Diy RIFT ?

Post by **JanVR** » Tue Jan 01, 2013 1:06 pm

Lookforyouhands, thanks for the feedback - very useful. I need to optimize the code some more to adjust the sensitivity and implement strafing, arm movements, etc. I think the Kinect is a powerful tool to enhance immersion. Unfortunately, I sense that many developers are averse to anything Microsoft, and even on this forum I see some users go through loops and bounds to attach Hydras and other sensors to their legs etc., while you have an optical tool like the Kinect with a user-friendly SDK to do all the work for you.

3dvision, regarding the full screen, one issue with the current app is that it stops tracking the skeleton when another program is switched to full screen. With Crysis, you can run the game in non-full screen on your monitor while you are still seeing full screen in the HMD. As such, the Kinect app keeps tracking the skeleton while you are playing. This is not the case, however, with other games (Skyrim, for example, switches to full screen on the monitor and switches off the tracking). I think this can be easily fixed by removing a couple of lines of code in the program, but I have not yet figured out how (I am not an experienced C++ programmer). Perhaps an experienced programmer on this forum can be of any help?

Post by **Fredz** » Tue Jan 01, 2013 2:34 pm

JanVR wrote:Unfortunately, I sense that many developers are averse to anything Microsoft, and even on this forum I see some users go through loops and bounds to attach Hydras and other sensors to their legs etc., while you have an optical tool like the Kinect with a user-friendly SDK to do all the work for you.

I don't think it has anything to do with a supposed hatred against Microsoft. The Kinect simply has a very high latency (around 150ms) and a slow frequency (30Hz), so it's less than ideal as a skeletal tracking solution for HMDs.

Post by **Lookforyourhands** » Tue Jan 01, 2013 4:02 pm

Oh I forgot to post that this works in full screen for me. I was hesitant to try
because I didn't want to be disappointed but to my surprise it works !! I know
there was mention of it not working in full screen but yah no problems
here. Maybe cause I'm using 3d TV play? Idk.. But it worked full screen
on both my LED monitor and my HMD..

I'll add a video later..
And even though there is supposed latency it seems very smooth and responsive!

Post by **JanVR** » Tue Jan 01, 2013 7:49 pm

Ok, great. Not sure why I am having issues with full screen.

Fredz, I agree regarding the latency, but it does not matter as much for walking recognition, which is more forgiving. It works really smoothly.

Post by **Lookforyourhands** » Tue Jan 01, 2013 9:31 pm

Quick demo of Kinect with Sony HMZ-T1

(My wife was kind enough to try it out for me)

[youtube]http://www.youtube.com/watch?v=SFu4YjcUBBU[/youtube]

It does work very smooth as JanVR said. When playing you really can't notice much
of a delay if any.. Although I imagine having a relatively decent computer helps..

And another..

[youtube]http://www.youtube.com/watch?v=i3R4e405oaY[/youtube]

Post by **Okta** » Wed Jan 02, 2013 2:50 am

Awesome vids thanks. I bought a kinect over a month ago and have been waiting since 21/11/12 for a stupid psu from ebay for it. Those vids confirm how workable this solution is. The head bobbing is pretty extreme though, does it bother you much in game?

Post by **Lookforyourhands** » Wed Jan 02, 2013 6:15 am

Annoying I hate waiting for things! Lol actually yes the head bobbing is kind
of annoying but at the same time it also adds a level of realism that wasn't
there before. Like when you run/walk your head bobs in real life anyway, but
yes with an HMD obviously the more it bounces around the worse it is. I'm thinking
maybe finding a better angle for my kinect to see if it will result in being able
to track me better without having to move my legs so much thus reducing head bobbing lol.
But yeah for just firing it up and jumping into gaming it works exceptionally well!
I'm stuck at work again but later I'm going to see how long I can game
for like this before getting tired as I have only had time to set everything up
and get it working but haven't actually gamed much yet. It's pretty cool though
imagine like an FPS, being chased by guys.. U r running all over the level.. Health
is low.. (And you're physically tired) so you find a place to hide in game to rest
your physical body as well as your avatar. It just adds an element of realism that
takes gaming to a whole new place.. The wii and ps move are just tip of the iceberg

Post by **brantlew** » Wed Jan 02, 2013 9:39 am

Okta wrote:Awesome vids thanks. I bought a kinect over a month ago and have been waiting since 21/11/12 for a stupid psu from ebay for it. Those vids confirm how workable this solution is. The head bobbing is pretty extreme though, does it bother you much in game?

You can achieve "similar" results using an inertial sensor (IMU) on your belt. The Kinect is much more flexible and has higher specificity with complex movements. But for simple motion signals (ie. move/stop), inertial sensors have a higher sensitivity so you don't have to exaggerate your movements quite as much.

Post by **3dvison** » Wed Jan 02, 2013 11:05 am

Hey Lookforyourhands, and JanVR,
When the Kinect is being used like this, along with a computer. Is the kinect seen by the computer as just a generic input device, like a mouse/keyboard combo or a joystick/gamepad ?

I was wondering if it is plug and play, with any computer game that uses a mouse/keyboard or joystick/gamepad.

Post by **JanVR** » Wed Jan 02, 2013 10:10 pm

3dvision, Microsoft recognizes the Kinect as soon as you plug it in, but you need to download the Kinect For Windows SDK/Drivers to make it work on your computer.

It does not function as a gamepad, though; it just recognizes your body skeleton and joints. To use it for gaming, you can use my skeleton viewer app that I posted earlier that translates your body movements into key strokes. The key strokes will drive the game.

Post by **3dvison** » Wed Jan 02, 2013 10:40 pm

Thanks JanVR,
I think that is sort of what I ment.
If your software app, is seen by the computer as keystrokes, does that mean that a Kinect + your app should work with any computer game that accepts keyboard input ?

Post by **JanVR** » Wed Jan 02, 2013 11:40 pm

Yes, apologies, I misunderstood your question. The app should work with any game or application that accepts key strokes. Only caveat I mentioned before is that I have some issues with certain games that switch to full screen mode and shut off the Kinect app (Skyrim, for example). It should be an easy fix though; I should be able to figure it out in the coming weeks, as soon as I find some time.

Post by **3dvison** » Thu Jan 03, 2013 12:03 am

No apologies needed JanVR,
My sentance structure is horrible.
Somtimes when I read one of my own post, even I wonder, what the helz I was trying to say ???

Post by **3dvison** » Thu Jan 03, 2013 4:24 pm

Would there be a way to add run/sprint to the app ? I think most games have a walk and a run button and you have to press both to run/sprint..
Could you do somthing like have the Kinect detect closing your hand into a fist as a (RUN) button press, which then would be combind with the (W) Walking, as the combo key press needed to run/sprint ?

Post by **JanVR** » Fri Jan 04, 2013 10:55 am

That is a good suggestion. I definitely plan to add some hand/arm gestures (for example, to re-load or throw a grenade). I'll look into the sprint feature as well.

What other hand/arm gestures or other movements would be useful?

Post by **brantlew** » Fri Jan 04, 2013 11:17 am

I think sometimes "gesture" commands can be more problematic than useful - especially if the gesture is not analogous to the real world action. In this instance the "closed fist" indicates a run mode. The gesture is non-intuitive and doesn't really match the real world action so its really no better than a button. But a button is vastly simpler to reliably detect so in this instance I think the button would be the better choice.

Post by **Flassan** » Fri Jan 04, 2013 12:55 pm

The simplest way I can think of is to move your legs faster. The WizDish Kinect demo program uploaded by Jan samples the x and z values of the ankles every 10ms, checks that they've changed enough not to be noise, then sends a 'w' keydown message to make you move forward. If that value was higher than a second threshold value it could send a 'run' command, assuming the game has that feature.
Another feature that would be great to include is to derive directional information from the same x and z values to apply a corresponding amount of turn, either for strafing or for when there is no head-tracking (e.g. for use in a CAVE). I've only just found out that version 1.5 of the Kinect SDK includes some face recognition. Might that be enough to tell if you are facing towards or away from the Kinect, which cannot be inferred from the x and z? Then it would work through the full 360 degrees.

Post by **brantlew** » Fri Jan 04, 2013 1:32 pm

Flassan wrote:The WizDish Kinect demo program uploaded by Jan samples the x and z values of the ankles every 10ms, checks that they've changed enough not to be noise, then sends a 'w' keydown message to make you move forward.

There is a lot of information available in the returned skeletal data. More sophisticated algorithms should be able to integrate the skeletal data over time and generate a huge variety of motion commands. It would not be too difficult to convert leg motions into accurate analog motion and speed commands - even going so far as to model the non-constant gait of the person. Of course, the game would need a flexible interface to support this. Some games with analog gamepad controls could be interfaced this way.

Post by **3dvison** » Fri Jan 04, 2013 2:05 pm

I would like to go button-less, so anything other than a button press would be fine with me.
The fist idea was just off the top of my head. I play alot of sports and when I take off in a sprint, for just a split second, my hand does clinch before going into a more relaxed open handed hand shake type position.
Even if it could detect a heel/toe type movment as (WALK) and then when you move your weight forward, onto the ball of your foot and toe area have Kinect detect it as a (RUN).

But the best would be what Flassan talked about, and just have the Kinect detect that your legs are moving faster and register it as a run button press.

Post by **shent1080** » Fri Jan 04, 2013 6:13 pm

it seems like kinect is doing a great job with skeleton tracking but what is everyone using for head tracking, can imu's/ir tracking be used alongside the kinect sdk, or better yet be implemented into the software?

Post by **Okta** » Fri Jan 04, 2013 10:02 pm

Trying to do away with buttons at this stage isn't going to make for smooth sailing. As Brentlew said, use the Kinect where it shines for gross intuitive moves, and hold a wireless controller or 2 for all the other multiple inputs required.

Post by **3dvison** » Fri Jan 04, 2013 11:49 pm

I ment no buttons for walking & running.
To me running is a gross intuitive move.
And there is such a gross difference between walking and running throughout the body, that I would think one of those difference could be keyed on for the Kinect to detect as a run.