full body tracking in UDK

[blasphemer]

http://www.youtube.com/watch?feature=pl ... STge5IDxF4

http://forums.epicgames.com/threads/928 ... EI-3-Space

Looks awesome, doesnt it?

[Chriky]

Yep that looks like it would be pretty much perfect for an ODT. The looking down at your hands and body stuff looks really good.

[Namielus]

Seriously impressed.

[cybereality]

Wow, looks great. Really good actually. I do wonder, though, where he got an extension cord for the HMZ-T1.

[Namielus]

His engineering website has lots of goodies that seem to be his own creations so he might have just made the extension
cord himself.
The 17x "space sensor" kit he is using seems to be around 4000 usd, I wonder how far down the price can go.

[rhinosix]

Wow! Looks really cool to see crouching. I guess he could probably jump too?

[Namielus]

I cant see why not.

[BOLL]

I cant see why not.

I'm just guessing here, but isn't this skeleton tracking based on relative angles? Meaning no absolute position data coming off the sensors but the joints are rotated the same as the joints in your actual skeleton. This would mean that one part of the body would always be at ground level, no? I guess a jump could be calculated from accelerometer data, not sure how accurate that would be but possible I guess... and as don't know what I'm talking about I'll stop, haha.

[Namielus]

If it can detect one leg being up it can detect two. And that's enough to trigger a jump move at the very least.

The whole body does not move upward like a stiff entity, so there are lots of clues the software can get that you are
performing a jump.
You know you kinda crouch your legs under your body?

As far as that being absolute positioning or not, I am unsure so I cant comment that.

[brantlew]

Very nice demo. I've been curious how well you could implement body tracking using inertial sensors and a skeletal constraint model. I guess this answers it. Not quite as good as absolute tracking - but good enough for gaming. $4000 is out of range for the average guy. You could reproduce the sensor package with other hardware, but the software for driving the whole constrained skeletal model would take a bit of work.


Edit: More interesting than the skeletal tracking is the walking. You can see the a couple of glitches as he shifts from foot to foot. The whole thing probably hinges on having that one stable point on the floor. Still - it works surprisingly well. I'm curious how much your true walking position would drift.

[FingerFlinger]

Was it Chriky who was working on a skeletal-tracked walking thing with the Hydra? I can't find the thread. From this demo, it seems like that is definitely an idea worth exploring.

[brantlew]

Was it Chriky who was working on a skeletal-tracked walking thing with the Hydra? I can't find the thread. From this demo, it seems like that is definitely an idea worth exploring.

Well it's a similar idea but the solution is fundamentally different. A magnetic skeletal tracker would be comparatively trivial because you could basically just read the sensor coordinates as-is. This inertial solution however has to run all the drifty sensors through a skeletal constraint model - probably in the form of a very complex Kalman filter to get stable coordinates out of the model. Anyway, I can see Chiky's idea working in conjunction with this system to improve performance. Right now I assume this inertial system requires that one foot be on the ground to act as a stable origin point. But if you added magnetic sensors into the mix you could have more than one point acting as stabilizers.

PS. I could be wrong about this. Maybe you don't need a stable point. Does anybody have any insight as to whether you can cancel out the drift from a web of sensors simply using a constraint model? Surely having stable points would only help the system though?

[FingerFlinger]

Right, that's what I mean. Having seen the demo with IMUs, I'm pretty optimistic about what we could do with a magnetic setup.

EDIT: I should clarify further; I think that using a magnetic skeletal tracking system that references from the user's torso might be a good solution for ambulation tracking.

[brantlew]

Too bad all we have to work with is the Hydra.

I wonder what would take more effort - creating a magnetic tracker from scratch and the accompanying algorithms or replicating this IMU sensor mesh and the accompanying skeletal model?

[FingerFlinger]

Well, my other project fell through due to my lack of antenna design ability, so I am looking for something new... I would love to get a magnetic tracker project off the ground, but I've really been thinking that a dead-reckoning system integrating optical flow and IMUs could be the way to go at this point.

EDIT: Sorry, getting a bit off topic here.

[Krenzo]

Well, my other project fell through due to my lack of antenna design ability, so I am looking for something new...

You gave up already?

[FingerFlinger]

Not ultimately, but I really decided that I want to do a phased-array, and there is a lot of theory for me to catch up on. Either that or use a big-ass dish to narrow the beamwidth. Also, just using some basic trig, I don't think we'll be able to get the kind of performance I was hoping for. It's still a cool solution for general-purpose RTLS, but less so for VR...

[brantlew]

Well, my other project fell through due to my lack of antenna design ability, so I am looking for something new... I would love to get a magnetic tracker project off the ground, but I've really been thinking that a dead-reckoning system integrating optical flow and IMUs could be the way to go at this point.

EDIT: Sorry, getting a bit off topic here.

I agree, but my gut tells me that optical flow is going to be difficult to integrate and customize. Optical flow is a pretty beefy topic that assumes an extensive background in EE, mathematics, and computer vision. I'm not convinced that a "canned" open source implementation can just be used out of the box like an API.

Sensor networks on the other hand seems like a more straight forward problem. Not something you could knock-out over the weekend, but maybe something that could be tackled in months rather than years.

Just my 2 cents.

[Chriky]

I'm not working on much of anything at the moment unfortunately but I did make a quick proof of concept with the Hydra for a leg tracking system a month or two ago. I'm reading a book on Arduino at the moment and I'm hoping to take a few weeks holiday before the Rift arrives to work on systems for optical and magnetic tracking. I'm ok at computer science/maths type stuff but not that good at electrical engineering. I found out there is a Hackspace quite near though me that I am going to go check out, should have some helpful/clever people there.

[FingerFlinger]

I actually want to talk to Jon Olick about the optical flow side of it, because he put together a damn impressive AR demo in a (seemingly) short amount of time. Generally speaking, I agree with you. If you can afford all of the sensors, then it's a more straight-forward problem to do the sensor network.

The other reason I want to dive into optical flow is that I am changing jobs and joining a team that does computer vision research, so I hope that I'll be able to gain some knowledge over the coming months/years.

[RoadKillGrill]

Hi guys!! I'm the one who worked on the demo, I've been looking around where the video has ended up and came across this site.

Looking over some of the discussion some people here already have some knowledge on how this system is working so I can confirm some of the assumptions.

For the movement demo uses the skeletons lowest point to ground it to the floor and as an anchor. The root is translated to maintain the bone in the anchored position. Once another bone is the lowest is switch the anchor to the other bone and adjusts accordingly. This method was really easy to add to the demo at the last minute and needs some work. There isn't any interpolation or smoothing between transitions so it adds a bit of shake to the overall model on walking. This should be easy to fix. There is enough data to calculate some jumping and other things in the future.

The in-game skeleton currently has no constraints, the sensor's on board orientation calculation is being directly applied to the bones. There is no client side filtering being done to the data. Having the sensors placed on a body already gives it some constraints on range of motion.

The unrealscript source is planned to be released, once its cleaned up a bit and a few more things added.

If you have any other questions ask and I can try to answer them.

[brantlew]

@RoadKillGrill: Welcome to the forum. You're in the right place for discussion of tracking solutions.

I have a lot of questions about the system. You are using some sort of 3-Space API to pull the sensor values. What is available to you through the API? Do they just supply individual sensor orientations or do they provide sensor positions as well? Do they provide a skeletal model, or are you feeding data into a skeletal model within UDK? I'm not familiar with the way UDK works, so does it provide a skeletal model with physical constraints - so that if you rotate the thigh upwards, the shin and foot come with it automatically? What custom code are you supplying - are you just gluing the sensor readings to a UDK skeleton, or are you computing a model of your own?

Sorry for the tornado of questions, but the demo is impressive and I want to understand how it works. Just a couple more...

How many sensors are you using, and what is the data rate for that sensor mesh as in how many samples per second per sensor can you read?

[Fredz]

First things first, congrats for what you've done ! It's quite an impressive achievement.

Now for questions, not directly related to your project, but since you seem to be quite familiar with the YEI 3-Space Sensor :
- did you happen to measure the latency (not update rate) for wired embedded and wireless ?
- is the frequency with the Kalman Filter AHRS functionality a solid 260Hz or is it a best case situation ?
- does this last mode correctly account for yaw drift ?

Thank you.

[cybereality]

Hey RoadKillGrill, welcome. Great demo you have done here. Don't have any questions other than those that were asked, but great job dude!

Actually, one question: how is this technology going to be used? Will you be releasing a game with controls like this, or is this just for demo purposes?

[RoadKillGrill]

You are using some sort of 3-Space API to pull the sensor values. What is available to you through the API? Do they just supply individual sensor orientations or do they provide sensor positions as well?

The API is a wrapper of the serial port to simplify communication with the sensor. It also allows it to be bound into programs like UDK using its DLL_Bind. The sensors can output currently the orientation in a verity of formats or the raw IMU data and a few other things like the button states or temperature. With a single sensor the position can be estimated with orientation and acceleration data.

I'm not familiar with the way UDK works, so does it provide a skeletal model with physical constraints - so that if you rotate the thigh upwards, the shin and foot come with it automatically? What custom code are you supplying - are you just gluing the sensor readings to a UDK skeleton, or are you computing a model of your own?

UDK's animation system provides a lot of things but all I did was make an animation tree that allowed me to access the bones and set every orientation manually. There is a sensor on the thigh, shin, and foot so that each has an orientation. The neck and head are a bit different in that they are connected but have a limited range of motion. I applied some of the rotation from the head to the neck to make it move more correct. The spine should have an interpolation between the chest and the hip sensors but I didn't get it working as I wanted in time.

Finally the fingers, they are animated with an existing idle animation in UDK. It looks better than plank hands or invisible gun clenching hands.

How many sensors are you using, and what is the data rate for that sensor mesh as in how many samples per second per sensor can you read?

There are 17 sensors and they communicate at an average of 60 samples. If a packet is dropped its not resent as its already stale data. The current throughput can be improved.

- did you happen to measure the latency (not update rate) for wired embedded and wireless ?
- is the frequency with the Kalman Filter AHRS functionality a solid 260Hz or is it a best case situation ?
- does this last mode correctly account for yaw drift ?

The latency depends upon the filter mode selected and the communication method used.
For wired embedded sensors average latency (using the complementary filter) is around 2.4ms Worst case latency is <3.2ms.
For a wireless ( with a single sensor using the complementary filter) the average latency would be about 6.2ms worst case ( assuming packets aren't being dropped ) would be 8.2ms.

The rates specified are average rates. In practice there is little variance in these under normal operation.
Although with the embedded sensor being used of SPI it is possible to inadvertently cause a reduction in the filter rate by polling the sensor too frequently over the SPI interface.

All AHRS modes have sensor fusion that accounts for yaw drift.

how is this technology going to be used? Will you be releasing a game with controls like this, or is this just for demo purposes?

The technology is already in use, from doing spine analysis to sticking in a ball and getting rotation and acceleration data.
The VR and game applications personally interests me and I plan to make a few more interactive demonstrations in the next few months. Releasing it as a full game in the near future sounds doubtful since the price of the units is higher than most consumers are willing to spend. I also cannot do the art, I would need to contract work or get volunteers which in the past has has high turnover and low consistency.
The current demo doesn't show much in the way of controls, that is one of the goals of the next one.

BTW here was a fun little game using a single sensor I made about a year ago. http://www.youtube.com/watch?v=hbSO-H5HRlc this one is playable but requires a 3-Space Sensor.

[Fredz]

Thanks for the info RoadKillGrill, this sensor looks like a good candidate for a DIY Rift.

[Namielus]

RoadKillGrill, how low can you get the price if its the wired version? Have you considered renting out kits?
I am interesting in taking your technology on my little tour where I am going to showcase as much tech as I can get my hands on to different people.
Trying to create some enthusiasm but also getting funds to go forward with my own project.

If you can provide a simple demo that also works with the rift, I am interested in buying or renting depending on price.

[RoadKillGrill]

@Namielus
I don't have a Rift to test with however I think the UDK suit demo can work as is*. *The Rift HMD can be used and the YEI wireless 3-Space Sensor can be placed on the back of the head for the tracking, UDK has sidebyside built in. I backed the Rift kickstarter and will be in by December hopefully to confirm this. Once I have it in I'll fully integrate it.

For the rental, alternative sensor options and pricing contact the sales team at support@yostengineering.com

[ripcurl123]

four grand is just too much for most people, very impressive though