Possible input device project. Suggestions/help wanted...

[MSat]

For the past year, I've been working on a project targeting a niche market of guitarists/bassists (more like a niche within a niche, but I digress). In the end, it made the economics of it very tough by requiring that the target price be raised for it to even be a viable financial proposition for me- pushing it even further into niche territory. Somewhat heartbreakingly, I realized this past week I had little choice but to abandon the project for any commercial purposes. (If you're a musician, and you want to know more, just ask and I'll tell/show ya all about it )

So anyway, I'm itching to move on and do something new. Thanks to the work of Palmer and John Carmack (if it wasn't for the latter, I may have never heard of the former), the dream of VR I had as a kid has been sparked again and more vivid than ever. It's something that I would very much like to be a part of. Given the current state of VR, I've been trying to consider what the next small step could be, utilizing most of my skills. What I've been thinking about is input devices. Of course, we all agree that there's no one-size-fits-all solution, but in the current state of things, perhaps there's a one-size-fits-most? I'd like to work on something that compliments the Rift (at least what we currently know about it), and offers a compelling experience for users.


What I'm proposing is a multi-node motion tracker and joystick pair. Here's what I had in mind (and my reasoning):

A device that consists of two joysticks containing IMUs, two strap-equipped IMUs for mounting to the user's upper arm, and a strap-equipped chest mounted IMU all daisy chained together. My initial thoughts were that each IMU could just consist of an accelerometer and magnetometer, and perhaps it is plausible, but perhaps adding gyros would help keep the magnetometer locked on to the earth's magnetic field despite local distortions, and other sources of magnetic noise. Either way, the goal is to provide absolute orientation and skeletal modelling of the upper body and arms - something those 5 nodes could provide. With this data, and the data from the head tracker, the orientation of the head relative to the body can also be determined.

As for the joysticks themselves, I have a few ideas that I have not seen used on other joysticks before. The first, instead of using an analog thumb stick like the wii nunchuck or razer hydra, it would use an "eraser head" commonly found on laptops (with replaceable hats of various sizes). I know a lot of people hate them at first, but once you get used to them, they can be incredibly quick and precise. Have you ever seen an analog stick used with any success for controlling a cursor? Even modern fighter aircraft use pressure sensing joysticks with imperceptible movement rather than the old displacement types.

Another idea for the joystick is to flank both sides of the thumb stick with vertical scroll wheels (+ button). Their importance on modern mice cannot be overstated, and I think it would be easy to find plenty of use for them. Below each one, angling down and in would sit a button that you can press by rolling over it with your thumb (similar to: http://www.pcdistrict.com/modules/produ ... ro-230.jpg) Besides those things, the common two trigger setup (one analog) could be the last of the buttons.

The last feature that I've been thinking about is programmable voice commands which could be bound to keys. So for example, instead of having to bind the 'Esc' key to a joystick button, you can assign a voice command to it.

I would like the firmware and drivers to be open source. I'm thinking that perhaps it shouldn't have any embedded firmware at all except for perhaps a bootloader. The application on the host machine could load any version of firmware (including customized, obviously) it wants.

Of course, I can't go at it alone, so I'm hoping to assemble a well rounded team to tackle this. Let me know if you're interested!

For everyone else, I'd like to hear your thoughts, suggestions, and questions.

Oh, and sorry for the long-winded post

[cybereality]

Interesting idea man! Good luck.

[RoadKillGrill]

Something similar to this thingy? http://www.mocapsuit.com/
Which is a stripped down Animazoo Gypsy from what I have seen. The Gypsy systems look like really badass exosuits, I kinda want to get one just to play with they but they are expensive.

This thing has been in dev for years, there are videos from 3 years ago of these things in action but I have no idea what the price of it is.
Their channel has a few videos showing it work http://www.youtube.com/user/emovemocap


I've been temped to try making a similar system with YEI 3-Space Sensors. It would be much lighter and have less moving parts but I have enough projects going at the moment that I cannot attempt this yet. I know its possible since I got a full body system working fairly well. http://www.youtube.com/watch?v=MSTge5IDxF4 Add a few joysticks, strip it down to fewer sensors and it should work.

[MSat]

Interesting find, RKG. I've seen IMU-based mocap suits before, but never that one. A neat thing about it (unlike what I had in mind) is that it enables sensing of wrist articulation (at least from what I can tell - can't view videos right now, unfortunately). However, I'm not sure how well it could track the arms without having additional sensing points on the upper arms. An upper body exoskeleton using potentiometers instead of mems-type sensors (though it would still likely need some) would certainly have a simpler software implementation, but would be pretty bulky.


I'm still leaning towards eliminating the gyros - at least for the upper arm and body sensors. That way, those parts could be greatly simplified by using a single ST mag/accel combo chip in each communicating over their native I2C bus. I know rotational data can be extrapolated from a magnetometer , though it's not as straightforward as using a gyro. Still, the benefit is simplified hardware, and while it may require a bit more computational effort to get rotation data out of them, without having to handle gyro data and the associated sensor fusion, the net results may just end up being the same.

[FingerFlinger]

I don't understand what you gain by not using a gyro? Strictly cost savings? There is a single chip 9DOF available. $17 for the chip by itself. Invensense and Sparkfun

Unless you are doing something exotic, the sensor fusion shouldn't be too bad. You can achieve great results with even a complementary filter, and a Kalman filter isn't terribly difficult to implement if necessary.

[MSat]

I don't understand what you gain by not using a gyro? Strictly cost savings? There is a single chip 9DOF available. $17 for the chip by itself. Invensense and Sparkfun

Unless you are doing something exotic, the sensor fusion shouldn't be too bad. You can achieve great results with even a complementary filter, and a Kalman filter isn't terribly difficult to implement if necessary.

Maybe trying to avoid gyros altogether is a bit silly. It's more of a thought experiment - as in why use them if they're not necessary? I really do believe they could at least be eliminated from some of the IMUs. My current thought is that the chest-mounted, IMU could strictly be an accel/mag device to extrapolate orientation. The upper arm tracker could potentially require only an accelerometer because it only needs to know its position relative to gravity, though it would likely be a good idea to incorporate a gyro to filter out g-loads due to movement. Similar to the chest-mounted IMU, the joysticks would need orientation along the vertical and horizontal axis, necessitating a magnetometer and and accelerometer - though like the sensor for the upper arm (and probably even more-so) it probably needs a gyro for the same reasons.

The idea of eliminating unnecessary sensors is for the sake of simplicity in terms of both hardware and software requirements (we're talking about 5 separate IMUs here), and of course cost, should this ever stand the chance of becoming a consumer product. Being able to reduce all the sensors to a single type would be great, and so that's why I'm curious if it's a possibility.

The 9DOF chip is pretty interesting, especially since it simplifies board layout, and it can reduce some of the workload from the host processor which is good for low power devices where IMU capability is not necessarily the primary function of the device. However, it doesn't seem to have the ideal specs for what I had in mind, not to mention it's extremely cost prohibitive for a consumer product.

[FingerFlinger]

I don't know if it is an important consideration for arm tracking, but one more thing to consider is that accelerometers will have some "bounce" due to inertia. It might not matter at the acceleration that an arm can generate during typical use, but it is something else to consider.

I'm excited to see what you come up with. Starting to think that skeletal tracking will be the fastest solution to fully self-contained walk-around VR. Combined with Brantlew's RedRoVR and a backtop... I think visual odometry can get there, but it's not close to being ready yet.

[MSat]

I've played around a bit holding a penny in the palm of my hand and then lowering my palm quicker than gravity can accelerate the penny. Muscles appear to allow for pretty rapid initial acceleration, but they don't seem to be able to sustain it for long. My guess is probably 3-4g max. The bounce is a good way to determine the motion has ceased. Still, tracking motions at that rate seems to be of little use for the most part except for perhaps really intense virtual fighting - which I'm not too worried if the system is unsuitable for.

For absolute orientation, I know the system needs magnetometers. It would almost be possible to use magnetometers exclusively if it wasn't for the fact that they can't determine roll along the axis of the magnetic field - which is where accelerometers fill in the gap. Accelerations along that axis could be useful in place of gyros for determining an approximate roll rate, and once motion ceases, to give an absolute value.

One of the nice things about arm tracking in VR is that slight errors in positioning my not be all that perceptible. It's our eyes that work as a feedback mechanism, so errors of even several inches between the real world and the virtual world shouldn't pose a problem (as long as it doesn't register movements in the wrong direction).

The biggest problem with magnetometers comes from magnetic field disturbances, or other field sources. For a fixed play space, calibration may be able to reduce the effects of noise sources (not too sure about disturbances, but probably not). In a dynamic environment such as free roaming, calibration wouldn't be possible - in those cases, adding a gyro would be desirable, which takes us back to square one.

If sufficient tracking is possible using just 5 $2 sensing elements, that's the route I would like to go. With the limited range of articulation the arms can perform, I'm really starting to think that it's actually viable.

[Krenzo]

There is a single chip 9DOF available. $17 for the chip by itself. Invensense and Sparkfun

I would stay away from that chip. It's about twice the price of a two chip solution plus microcontroller, and there is no documentation on using the on-chip motion processor.

[MSat]

I think it's best that I tone down the goals a bit in order to make the chance of success a bit more probable. Instead of aiming to include joysticks as part of the goal (and the most complex in terms of manufacturing), I'm looking to reduce the system to a body-mound sensor, and a pair of sensors for each arm - one on the upper arm, and one on the wrist. That enables the opportunity to utilize existing enclosures currently available on the market with perhaps only minor modifications. Without the joysticks, the options available to the users are a bit more flexible. They can use something like the PS Move Navigation Controller, the Peregrine glove, or perhaps something hands-free altogether. It could also be useful for augmenting an HMD-mounted Leap Motion - providing arm tracking while not within the Motion's FOV. Last but not least, this should decrease the price considerably. Anyone else think this may be a good idea?


Edit:

As it turns out, the hardware may be even simpler and cheaper than I expected. I was a bit unsure of the connectivity - I liked the idea of wireless, but the convenience of the variety of MCUs with integrated USB PHYs was really tempting. Luckily, there are some highly integrated MCUs with Bluetooth 4 transceivers at very low prices (<$2) that only require some passive external components. The accel/mag chips are similarly priced, and aside from having good specs, they can be configured to have 1 of 4 unique I2C addresses - so at least the 4 arm-mounted sensors can share the same bus while still not saturating it. Another cool thing about the particular MCU is that it has an integrated OPamp and ADC. That means that with the addition of a cheap electret mic, the system can have voice transmission practically for free.