Foot controller with inherent position tracking capabilities

This is for discussion and development of non-commercial open source VR/AR projects (e.g. Kickstarter applicable, etc). Contact MTBS admins at customerservice@mtbs3d.com if you are unsure if your efforts qualify.
Post Reply
MSat
Golden Eyed Wiseman! (or woman!)
Posts: 1329
Joined: Fri Jun 08, 2012 8:18 pm

Foot controller with inherent position tracking capabilities

Post by MSat »

I've been quite fascinated with foot controls ever since crespo started the original thread on them. Giving your feet the duty to manage what they do best seems like not only an an intuitive and immersive scheme to control player movement, but also frees your hands for other tasks. I've given the subject a lot of thought based on both, what other people have suggested/done and pieced them together with some of my various and somewhat unrelated ideas. I've been keeping this close to my chest the past several weeks as I wanted to have a go at it myself, but I know I can't do it alone - particularly when it comes to software, or making a business of it. So now I'd rather share my thoughts in public and see where it can go from here. :)

One of the obsessions was concerning how to incorporate positional tracking in an inherently limited environment. Given that your feet are essentially glued to the foot pedals, some of the challenges related to tracking are greatly reduced. I'm guessing some of you might be wondering what good is position tracking if you're not actually free to physically walk around. It's true that with a foot controller, you're essentially tied to one spot, but your body can still sway a bit from side to side, and forwards and back. Those small motions are significant enough due to parallax effects, which can give you a greater sense of "being there".

My original thought was using a mechanism like the GameTrak in the pedal assembly, or perhaps mounted a bit higher perpendicular to the floor with the cables connected to a chest strap with "wings" mounted behind. The GameTrak's simplicity is quite ingenious but it has some limitations. More importantly, I just wanted to avoid patent issues, so I tried figuring out some variant to side-step it. While I was moving around on a make-believe pedal board trying to visualize the motions and how to sense them, I realized that I was shifting weight from one foot to another, or towards the front or rear of my feet - the very thing that wii balance board, or more generally, a platform with independent weight sensors in each corner is designed to detect.

Now on its own, it's only of limited use - it can't provide all the movement controls we need, or even sense some of the body motions for positional tracking (example: it can't differentiate between leaning your torso to the side and shifting your body to the side). The simplest solution to the latter problem seems to be a chest-mounted orientation sensor. There are additional benefits to the chest sensor as well - such as being a reference point for skeletal modelling in conjunction with arm mounted sensors.

Ok, so the balance board is essentially a 2 axis sensor. That means we can incorporate forwards/reverse, and either strafing or turning. If we want to enable both of those, the board needs an extra axis. My thought is a twistable pedal under each foot. Realistically, only one would probably be needed, but two adds a lot more flexibility in terms of control schemes which is what I want to get into next.

So if a weight-shift sensing board is used for position tracking, then how can it be used to control movements? Well, there's actually quite a few ways - none of them more correct than the other, but more a matter of preference - not unlike how there's several control schemes available with a single game pad. It should offer enough flexibility to find various methods that suit most people. Here are a few options:

1) Moving side to side and front to back evenly is only for slight position tracking such as looking around a corner or off the edge of a cliff without controlling player movement. In order to actually move your character forwards or backwards, weight is shifted towards the front or back of a specific foot. It would take some getting used to in order to prevent actual movements when all you want to do is look around, but I think it's workable. In order to turn and strafe, each twist pedal would be assigned to one of the tasks.

2) Enable or disable movement with the push of a button on the hand controls. For example, while not pressing a button on a hand controller, any swaying on the platform is used to detect body position, though twisting either of the pedals can still turn your character. When you press the button, and lean forward or backward it will move your character accordingly, while shifting your weight side to side causes you to strafe (or turn). The remaining motion can be controlled with the twist pedals.

Those essentially cover the premise. There are several variations or combinations to the two examples above which makes the system quite flexible. So while a weight shift sensing board can be effective for character control, it provides almost everything needed for basic position sensing too at no additional cost (other than the chest sensor, which as I mentioned before offers other potential benefits).

I think such a system including arm tracking and hand controllers (with just buttons - no analog sticks needed) could be sold for under $150 based on my estimates.
Aabel
Cross Eyed!
Posts: 190
Joined: Sat Jun 16, 2012 11:43 am
Location: Irvine, CA

Re: Foot controller with inherent position tracking capabili

Post by Aabel »

Some really good thoughts on the problem! however I would highly recommend you try developing with the wii board before you leap to any conclusions about what it can or can not do, it's a deceptively humble little thing and when you use it as a balance controller a surprising amount of nuanced control is possible. The realization that optical tracking is opened wide up once you plant your feet in a stationary spot, eliminating most occlusion issues, was one we had as well, and is something to investigate further. I wonder if this doesn't also solve some of the positional tracking issues for audio as well?

Are you in the Los Angeles/Orange county area by chance?
MSat
Golden Eyed Wiseman! (or woman!)
Posts: 1329
Joined: Fri Jun 08, 2012 8:18 pm

Re: Foot controller with inherent position tracking capabili

Post by MSat »

Aabel wrote:Some really good thoughts on the problem! however I would highly recommend you try developing with the wii board before you leap to any conclusions about what it can or can not do, it's a deceptively humble little thing and when you use it as a balance controller a surprising amount of nuanced control is possible. The realization that optical tracking is opened wide up once you plant your feet in a stationary spot, eliminating most occlusion issues, was one we had as well, and is something to investigate further. I wonder if this doesn't also solve some of the positional tracking issues for audio as well?

Are you in the Los Angeles/Orange county area by chance?

I have not had the opportunity to play with a balance board as I told myself I would not start any new projects until I moved - which was supposed to be in January, but it'll be the end of this month instead.

Being able to detect the body's center of balance seems like a very practical way of position sensing as well as input control. I certainly agree with you that standing in a fixed location makes other forms of tracking more feasible whether it's optical, magnetic etc., but those don't seem necessary to me as long as you can get orientation data from a few other parts of the body, which is why I mentioned the chest sensor (which I believe could be as simple as a single magnetometer + accelerometer chip). I could be making some incorrect assumptions, so I'd love to know what you think. As far as audio goes, I haven't put any thought into it, but I don't immediately see how it could simplify the problem given that you're still free to move your head and upper body.

I live in the midwest, but does it count if I wish I lived in Cali? :lol:
At any rate, I'd love to hear more about your work with the balance board. Do you have a site or blog about it?
Aabel
Cross Eyed!
Posts: 190
Joined: Sat Jun 16, 2012 11:43 am
Location: Irvine, CA

Re: Foot controller with inherent position tracking capabili

Post by Aabel »

MSat wrote:Being able to detect the body's center of balance seems like a very practical way of position sensing as well as input control.
It really is and you have some very interesting ideas that would build on some of the work my wife and I did.
MSat wrote: I could be making some incorrect assumptions, so I'd love to know what you think. As far as audio goes, I haven't put any thought into it, but I don't immediately see how it could simplify the problem given that you're still free to move your head and upper body.
Yeah I haven't thought it through too well either, but there is something nagging me in my head that tells me this could be useful for some part of 3d audio.
MSat wrote:At any rate, I'd love to hear more about your work with the balance board. Do you have a site or blog about it?
No, We tend to hold things close and not run off to the internet with what we have done, it's just the development culture we come from. We are trying to break that habit and get a bit more open. I don't really like knowing that I am putting my stuff out there for just anyone to see before it's in a state where I am comfortable with it being a finished commercial product. That's why I prefer to show things in person. We are really debating opening up a bit as we think we have somethings that might be valuable to others and some feedback would be nice.
User avatar
nanicoar
Cross Eyed!
Posts: 101
Joined: Mon Sep 10, 2012 10:09 am
Location: Finland

Re: Foot controller with inherent position tracking capabili

Post by nanicoar »

MSat wrote:Ok, so the balance board is essentially a 2 axis sensor. That means we can incorporate forwards/reverse, and either strafing or turning. If we want to enable both of those, the board needs an extra axis. My thought is a twistable pedal under each foot.
Hmm... Like this?

[youtube]http://www.youtube.com/watch?v=7oyr8rYy2nM[/youtube]
Aabel
Cross Eyed!
Posts: 190
Joined: Sat Jun 16, 2012 11:43 am
Location: Irvine, CA

Re: Foot controller with inherent position tracking capabili

Post by Aabel »

nanicoar wrote:
Hmm... Like this?

[youtube]http://www.youtube.com/watch?v=7oyr8rYy2nM[/youtube]

Yes, something like that would be amazing, you would be able to account for going up and down slopes, balancing to adjust to increase or decrease your speed up or down the slope. I just worry about the practicality of using such a device while wearing a HMD, but I personally would be willing to risk it. But yes after I implemented my wiiboard ideas and I was critiquing where it needed to go to be 'better', a device like the one you show is what I imagined, however I am no engineer.

There was another video of a similar concept on here somewhere that more clearly showed the device responding to changes in the virtual terrain, I'll see if I can dig that up.
MSat
Golden Eyed Wiseman! (or woman!)
Posts: 1329
Joined: Fri Jun 08, 2012 8:18 pm

Re: Foot controller with inherent position tracking capabili

Post by MSat »

nanicoar wrote:
MSat wrote:Ok, so the balance board is essentially a 2 axis sensor. That means we can incorporate forwards/reverse, and either strafing or turning. If we want to enable both of those, the board needs an extra axis. My thought is a twistable pedal under each foot.
Hmm... Like this?

[youtube]http://www.youtube.com/watch?v=7oyr8rYy2nM[/youtube]
No, not really like that at all. By "Balance board" I'm referring to the device for the Wii which you "balance" on in a very general sense. The only part that might (but not necessarily) perceptibly move is each foot pedal - yawing, or rotating along a horizontal axis. At least that's what I have in mind. Not saying it's the correct, or only way. :)
User avatar
nanicoar
Cross Eyed!
Posts: 101
Joined: Mon Sep 10, 2012 10:09 am
Location: Finland

Re: Foot controller with inherent position tracking capabili

Post by nanicoar »

Aabel wrote: Yes, something like that would be amazing, you would be able to account for going up and down slopes, balancing to adjust to increase or decrease your speed up or down the slope. I just worry about the practicality of using such a device while wearing a HMD, but I personally would be willing to risk it. But yes after I implemented my wiiboard ideas and I was critiquing where it needed to go to be 'better', a device like the one you show is what I imagined, however I am no engineer.

There was another video of a similar concept on here somewhere that more clearly showed the device responding to changes in the virtual terrain, I'll see if I can dig that up.
...
MSat wrote:No, not really like that at all. By "Balance board" I'm referring to the device for the Wii which you "balance" on in a very general sense. The only part that might (but not necessarily) perceptibly move is each foot pedal - yawing, or rotating along a horizontal axis. At least that's what I have in mind. Not saying it's the correct, or only way. :)
I was imagining you would be sitting down. Partially because my own balance is tied so closely to my vision that any game which even decoupled it a little from my head movement would send me flying across the room. For standing up... Maybe something with snowboard bindings and a very large platform for stability could keep me safe. - What games would you play on this?

For an actual simulation of natural walking, running and jumping I would have to demand two industrial robot arms which can track my feet from millisecond to millisecond. Of course I'd also be terrified that there's a bug in the software and that as soon as a sensor got confused the machine would tear off one of my legs, for its strength and speed.


Aabel, if you find the video I'd love to see it!
Aabel
Cross Eyed!
Posts: 190
Joined: Sat Jun 16, 2012 11:43 am
Location: Irvine, CA

Re: Foot controller with inherent position tracking capabili

Post by Aabel »

Turns out it was a video by you Nanicoar!

http://www.mtbs3d.com/phpBB/viewtopic.p ... 40#p104217

am I correct in assuming that the elongated pyramid with a sphere on top is an avatar moving across a virtual terrain and the board is rotating to match in the slope in the virtual world?

Yes, I am worried about people bieng able to keep balance while standing and having a Rift on. It's something I will be testing with my wiiboard solution when I get my Rift to see if it's a road to continue going down, or a dead end due to people losing balance. I am really hoping for a standing solution as I do not like sitting, I even work standing up.
User avatar
nanicoar
Cross Eyed!
Posts: 101
Joined: Mon Sep 10, 2012 10:09 am
Location: Finland

Re: Foot controller with inherent position tracking capabili

Post by nanicoar »

Yup, that's the idea of it. There's no manual animation in there, just logic. I wanted it that way to show where you would get data from in the case of force-feedback.
It's a bit overly sensitive though, so in the video in this thread I reduced the sampling interval so it's smoother.

My guess is that if you need to think about how to move you're going to fall, but if the motion feels natural you'll do fine by intuition even when unexpected things happen. Having said that, I hope VR environments will soon be so engaging that wearing a helmet is considered standard. =)
User avatar
brantlew
Petrif-Eyed
Posts: 2221
Joined: Sat Sep 17, 2011 9:23 pm
Location: Menlo Park, CA

Re: Foot controller with inherent position tracking capabili

Post by brantlew »

I posted this in another thread but thought it was relevant here as well.
I just had an interesting idea that combines the strength of a couple of approaches I've heard around here.

What if you used a foot controller as a toggle for head-controlled character turning. Something simple like - with your feet flat your head only turns your head, but if you lift your heel your head would turn your body. It would be more intuitive than a button or a head tilt since moving your feet is actually how you would initiate a body turn.
An important aspect of this approach is that all spinning motion will match your vestibular senses so nausea effects will be greatly reduced - as opposed to using controllers.
MSat
Golden Eyed Wiseman! (or woman!)
Posts: 1329
Joined: Fri Jun 08, 2012 8:18 pm

Re: Foot controller with inherent position tracking capabili

Post by MSat »

nanicoar wrote: I was imagining you would be sitting down. Partially because my own balance is tied so closely to my vision that any game which even decoupled it a little from my head movement would send me flying across the room. For standing up... Maybe something with snowboard bindings and a very large platform for stability could keep me safe. - What games would you play on this?
While it should probably work sitting down too, much of the benefit (such as positional tracking) stems from the fact that the system is able to detect the body's center of balance. You can still get a limited effect using an orientation sensor on the upper body as you can really only lean over while sitting, whereas with standing, you can detect movement due to both shifts at the hips and leans of the upper body.

It's really not about emulating natural walking, but incorporating positional tracking into a foot controller which some of use might prefer instead of using thumb sticks for player control. The point is that the system inherently allows for a certain level of positional tracking, which is a bonus.
brantlew wrote:I posted this in another thread but thought it was relevant here as well.
I just had an interesting idea that combines the strength of a couple of approaches I've heard around here.

What if you used a foot controller as a toggle for head-controlled character turning. Something simple like - with your feet flat your head only turns your head, but if you lift your heel your head would turn your body. It would be more intuitive than a button or a head tilt since moving your feet is actually how you would initiate a body turn.
An important aspect of this approach is that all spinning motion will match your vestibular senses so nausea effects will be greatly reduced - as opposed to using controllers.
This is an interesting thought, and certainly worth investigating. I had considered a similar effect that might occur if you shift your body in the direction you want to move. Since your body would feel an initial acceleration, it might quell some discomfort you might get from your avatar being stationary to all the sudden moving with 0 vestibular feedback. Of course, the actual motion would be slight, so it's hard to say if it would be beneficial at all, but I believe it's also something worth looking into.
antimatter
One Eyed Hopeful
Posts: 13
Joined: Mon Mar 25, 2013 5:07 pm

Re: Foot controller with inherent position tracking capabili

Post by antimatter »

I was thinking of something like a DDR mat when it could track your foot position. But the wii balance board sounds like something worth experimenting with since many people already have it.
Post Reply

Return to “VR/AR Research & Development”