The Adjacent Reality open source wireless head/hand Tracker

[nrp]

Over the last 6 months or so I've been iterating on a design for a small, low cost, open source (hardware, firmware, and host interface software), wireless head and hand tracker. With the current revision, I have something worth showing.



The Adjacent Reality Tracker has 3 axis each of accelerometer, magnetometer, and gyroscope, an RGB LED, an 850nm infrared LED, a lithium ion battery and onboard charger, a USB interfaced microcontroller, and a Nordic radio transceiver. The bare Tracker is around 1.1"x1.1"x0.3" and weighs 9.2 grams. I'll be 3D printing enclosures for these prototypes, but I'm looking into injection molding for the future. While I haven't done full battery life tests yet, the expected usage is on the order of 8 hours. Parts cost for the Tracker in prototyping quantities (100) is around $30, and around $12 for the Base Station.

While the on board microcontroller is capable of doing AHRS sensor fusion and outputting a quaternion at roughly 120Hz, it can more usefully and quickly stream sensor data to a host computer to be processed there. The gyro is capable of sampling rates up to 760Hz, and the Nordic link has the throughput to transmit that fully with under 1 ms latency to the Base Station. (USB will be the long pole for latency)

The Base Station USB stick can interface to up to 6 Trackers, but due to on air collisions at high data rates, 3 is probably the practical limit for real time virtual reality applications. The Base Station exposes any available Trackers to the host computer using the new USB HID Sensor descriptors. The Tracker can also be interfaced to a computer directly via its micro USB connector, where it also appears as a collection of HID Sensors.

The hardware design is approaching complete, now that I have ditched Bluetooth for latency/jitter reasons, but the software is still under heavy development. You can see it in action at GitHub. The software is currently under the GPL, but will be BSD once I remove sensor fusion from the firmware. The hardware design is Creative Commons.

The plan at the moment is to complete the software for this board revision. I only just finished assembling the first couple of boards today, but once I have more done and the software in more complete form, I'll be sending them out to people willing to help with development of the firmware, host interface libraries, or applications. The longer term plan is to launch a Kickstarter towards the end of the year if there is interest. I'll be pricing it as close to break even as possible.

[brantlew]

That's fantastic! It sounds like a great tracker.

Edit: A couple thoughts. Too bad the Rift is almost in production. I'm sure all the deals have been made and the software support added for the Hillcrest. This sounds like it would have made a great companion piece for the Rift. You should ping Palmer anyway.

When you feel ready to start sending out dev kits, I would be happy to push this into FreePIE.

[pierreye]

I'm really interested in your project compare to hillcrest due to wireless version with low latency. Let me know when you ready to sell few prototype for initial tester.

[alexlf]

That's an extremely impressive latency/battery life/refresh rate/price combo. I'd take 3 + the base station the instant it was released.

[WiredEarp]

Looks great, i'd take a couple + base station myself. The 120hz update frequency and wireless are great features...

[Krenzo]

I am curious as to why you went with separate chips for the gyro and IMU+magnetometer. Were you not able to find one that combined them all into one? There was one chip discussed in another thread that did just that.

[nrp]

I am curious as to why you went with separate chips for the gyro and IMU+magnetometer. Were you not able to find one that combined them all into one? There was one chip discussed in another thread that combined them all together.

Invensense is currently the only company putting out a single chip with 3 axes each of gyro, mag, and acc. The problem is that the chip is expensive and difficult to source. They aren't necessarily interested in supporting small projects like this.

STMicro is currently previewing a chip that is essentially just the two that I am using combined into one package. I'm hoping that by the time I would be manufacturing a larger run of Trackers, this chip will be available. It likely wouldn't reduce my board size, but it would reduce the parts count and cost as well as the assembly cost.

[android78]

That's pretty cool!
If only the board included a GPS, and had an option to disable the wireless to be able to use ones own transmission, it would be ideal for FPV RC.

[nrp]

That's pretty cool!
If only the board included a GPS, and had an option to disable the wireless to be able to use ones own transmission, it would be ideal for FPV RC.

The breakout board exposes SPI, level shifted UART, and 6 GPIO pins (all have ADC [glove controller with a flex sensor in each finger comes to mind], some also have PWM). You could interface it to your own radio setup over either UART or SPI. I'm trying to keep the firmware modular to make it easy to write different interfaces to the sensor data.

[cybereality]

This looks awesome! If you need someone to test stuff out, just shoot me a PM.

[nrp]

This is very preliminary, but at the moment, I'm looking at doing some really basic positioning support using the magnetometer along with the base unit from a Razer Hydra. It is, as expected, generating a sin wave. Reverse engineering the intended shape of the magnetic field will take some time though. I've captured some samples of the sin wave, but need to time it more carefully and do a Fourier transform to determine the frequency. Syncing the magnetometer's sampling rate to the Hydra's frequency may prove impossible though.

Edit:
The Sixense SDK is needed to turn on the base station, but it will turn on its magnetic field without controllers plugged in, and once the field is powered on, the Sixense SDK based app can be closed.

Also, just to be clear, the Tracker in its current form can not be as precise in positioning as a Hydra controller. The Hydra uses fairly large inductors through op amps before going to a 10 bit ADC. The Tracker uses a 10 bit MEMS magnetometer intended for much less precise measurement of the Earth's magnetic field.

[brantlew]

That would be great! I haven't seen anyone yet trying to reverse engineer the Hydra, but it seems like a very ripe area of research. You would think that a DIY Hydra knock-off could be made with wireless controllers.

[space123321]

I have recently purchased a Hydra and I am very pleased with it. I am planning on mounting the sensor above mself and have the hmd cables and hydra wands filter down from the ceiling to allow for full rotations.

My main concern will be if the HMD tracker and hydra will be able to be used at the same time. I beleive that mouse control is not present in Portal 2 while the hydra is in use. Once the hydra is placed back on the base, the mouse becomes active. I have not looked into this further, just something than made me say to myself "hmmmmm..."

[Krenzo]

Also, just to be clear, the Tracker in its current form can not be as precise in positioning as a Hydra controller. The Hydra uses fairly large inductors through op amps before going to a 10 bit ADC. The Tracker uses a 10 bit MEMS magnetometer intended for much less precise measurement of the Earth's magnetic field.

Everything I have read about AC magnetic field tracking indicates that surrounding metal causes interference because of eddy currents. Does anything on your circuit board interfere with the Hydra's field?

Are there teardown pics of the Hydra? I'm interested in seeing what is going on inside.

[cybereality]

I remember using the Vuzix tracker with the Hydra and they did work together (might have been using accelerometer only though, can't remember). However the Hydra gets interference from my ASUS 120Hz monitor for some reason. It gets really bad jitters unless I turn the monitor off.

[space123321]

I have no jitters in my Hydra setup - however I did receive a faulty unit and required a replacement. Things are running perfect now!

[nrp]

Also, just to be clear, the Tracker in its current form can not be as precise in positioning as a Hydra controller. The Hydra uses fairly large inductors through op amps before going to a 10 bit ADC. The Tracker uses a 10 bit MEMS magnetometer intended for much less precise measurement of the Earth's magnetic field.

Everything I have read about AC magnetic field tracking indicates that surrounding metal causes interference because of eddy currents. Does anything on your circuit board interfere with the Hydra's field?

Are there teardown pics of the Hydra? I'm interested in seeing what is going on inside.

There will be interference, but I have not yet determined how much. If I can come up with a good, consistant way to calibrate it, I can probably do soft and hard iron error compensation similar to what is normally done for magnetometers.

I didn't take any pictures while I had mine apart, but there is a teardown thread somewhere on these forums.

Edit: I also realized that if I want to do this properly, I can build a daughterboard with inductors, three op amps, and a small microcontroller with ADCs on it to avoid having to sync the main MCU run loop to the Hydra. The daughter MCU can then determine the peak to peak values and send them back over UART.

[zalo]

This is the latest image from his website. It looks like this revision is shaping up!

You're awesome, by the way. I can't understate the enormous potential of this tracker (especially with the dying support of solutions like the move and the wiimotion plus).

I'll also buy a bunch as soon as you put them up to buy.

Thanks!

[fmuniz]

Guys, this is awesomwe.

But what is the range I should expect from this hardware?

Would definitely adopt it!

[nrp]

I didn't get much time to work on the project over the last couple of weeks because of work and family stuff, but I am back at it fully now.

I'm still working on the wireless interface, but 10 meters is typical for Nordic 2.4Ghz using pcb trace and chip antennas.

Over USB directly, I'm currently getting a little over a 500Hz data rate, which is probably sufficient for not having to worry about timing vs the game loop, but is still short of the 760Hz maximum of the gyro and the 1000Hz theoretical max for USB interrupts.

[brantlew]

Wow! 500Hz is awesome. Can the accelerometer sustain that rate as well - meaning you can get 500 unique samples? At that rate it would be interesting to see if you could start getting some positional data out of it. That integration approximation gets closer to reality the closer you can get to infinity.

[NickK]

This is very preliminary, but at the moment, I'm looking at doing some really basic positioning support using the magnetometer along with the base unit from a Razer Hydra. It is, as expected, generating a sin wave. Reverse engineering the intended shape of the magnetic field will take some time though. I've captured some samples of the sin wave, but need to time it more carefully and do a Fourier transform to determine the frequency. Syncing the magnetometer's sampling rate to the Hydra's frequency may prove impossible though.

Edit:
The Sixense SDK is needed to turn on the base station, but it will turn on its magnetic field without controllers plugged in, and once the field is powered on, the Sixense SDK based app can be closed.

Also, just to be clear, the Tracker in its current form can not be as precise in positioning as a Hydra controller. The Hydra uses fairly large inductors through op amps before going to a 10 bit ADC. The Tracker uses a 10 bit MEMS magnetometer intended for much less precise measurement of the Earth's magnetic field.

Even if it is not as good as Hydra, what is roughly the accuracy that you can achieve? Can several base stations be used to improve that accuracy?

[nrp]

Yep, those are unique samples! The accelerometer is actually capable of an insane 5,376 Hz sampling rate at low resolution (10 bit? 8 bit?), and 1,344Hz at normal resolution (12 bit). Unfortunately, the higher the sampling rate is, the higher the error density is, so it doesn't help that much for dead reckoning.

[brantlew]

Just curious. That 500Hz is for real-time data packets. But you're not bandwidth limited at this point right? So if you stacked packets - say 1.5 per transmission could you recover all 760 unique samples? It wouldn't help latency, but it might help smoothing and sensor fusion.

Edit: Ah. Of course higher sampling rate means more errors. But not a higher error magnitude (noise floor) right? So if you could recover all 1300 samples wouldn't you be able to smooth better with more samples approximating your bell curve - or am I missing something? (I'm not a signal engineer)

[nrp]

This is very preliminary, but at the moment, I'm looking at doing some really basic positioning support using the magnetometer along with the base unit from a Razer Hydra. It is, as expected, generating a sin wave. Reverse engineering the intended shape of the magnetic field will take some time though. I've captured some samples of the sin wave, but need to time it more carefully and do a Fourier transform to determine the frequency. Syncing the magnetometer's sampling rate to the Hydra's frequency may prove impossible though.

Edit:
The Sixense SDK is needed to turn on the base station, but it will turn on its magnetic field without controllers plugged in, and once the field is powered on, the Sixense SDK based app can be closed.

Also, just to be clear, the Tracker in its current form can not be as precise in positioning as a Hydra controller. The Hydra uses fairly large inductors through op amps before going to a 10 bit ADC. The Tracker uses a 10 bit MEMS magnetometer intended for much less precise measurement of the Earth's magnetic field.

Even if it is not as good as Hydra, what is roughly the accuracy that you can achieve? Can several base stations be used to improve that accuracy?

I'm probably not going to continue the experiment. The precision would be on the order of inches, rather than the millimeters that the Hydra is capable of, and would deteriorate pretty quickly beyond a couple of feet away. The accuracy would be even worse than that. The refresh rate would be really poor as well.

When I have more time to play with it, which will likely not be until early next year, I'm going to build a proper wireless magnetic positioning version of the Tracker with coils.

[nrp]

Just curious. That 500Hz is for real-time data packets. But you're not bandwidth limited at this point right? So if you stacked packets - say 1.5 per transmission could you recover all 760 unique samples? It wouldn't help latency, but it might help smoothing and sensor fusion.

My data rate is actually largely bandwidth limited. If I send only gyro data (6 bytes, making an 8 byte endpoint), I can reach 750Hz. Adding accelerometer and magnetometer data (18 bytes, for a 32 byte endpoint) within the same packets brings it down to 500Hz. I could use 14 remaining bytes to pack in some older gyro data, but I'm trying to keep this a valid HID device. I'm still looking into why I can't push 32 bytes every interrupt.

[brantlew]

When I have more time to play with it, which will likely not be until early next year, I'm going to build a proper wireless magnetic positioning version of the Tracker with coils.

Yes, yes, yes. I almost PM'd you about this the other day. You would be the perfect guy to bring out some cheap, general purpose, magnetic sensor modules. Sixense is invested in the Hydra and stingy with their technology and Polhemus is just retarded (they quoted me $10,000 for a 4 sensor package!!) Nobody is filling this gap, but magnetic sensors could be the "new IMU" in a couple of years. Hell get into that business and maybe even a big player like Sony would make a big contract if it decides to do more advanced motion controllers.

[brantlew]

My data rate is actually largely bandwidth limited. If I send only gyro data (6 bytes, making an 8 byte endpoint), I can reach 750Hz. Adding accelerometer and magnetometer data (18 bytes, for a 32 byte endpoint) within the same packets brings it down to 500Hz. I could use 14 remaining bytes to pack in some older gyro data, but I'm trying to keep this a valid HID device. I'm still looking into why I can't push 32 bytes every interrupt.

Ah. Too bad. Have you considered some cheap compression like only sending the delta value and stripping off a few leading zeros?

[nrp]

My data rate is actually largely bandwidth limited. If I send only gyro data (6 bytes, making an 8 byte endpoint), I can reach 750Hz. Adding accelerometer and magnetometer data (18 bytes, for a 32 byte endpoint) within the same packets brings it down to 500Hz. I could use 14 remaining bytes to pack in some older gyro data, but I'm trying to keep this a valid HID device. I'm still looking into why I can't push 32 bytes every interrupt.

Ah. Too bad. Have you considered some cheap compression like only sending the delta value and stripping off a few leading zeros?

I forgot that USB HID doesn't require byte alignment. Since both the acc and mag sensors are 12 bit, I can bring the total to 15 bytes, for a 16 byte endpoint. I'll try this when I get home.

[brantlew]

Also consider compression.

You can get some good compression on a non-random feed like this using simple techniques. For example: Say each sensor value has a range of 65535 (2 bytes). Rarely from one sample to the next will the value deviate very far. Let's say that +-256 is a pretty good cap on slow movement deviation and signal noise. +-256 can be represented with 9 bits. So if you represent the feed as a stream of cumulative delta values, then on average each sample value could be represented with 9 bits instead of 16 and all smashed together.

Ok, not entirely true. You will have samples that exceed the 256 delta, so you will need two types of values. A full 16 bit absolute value and a 9 bit delta value. You can use a single bit flag on the front to indicate the data type. So each value will be either 10 bits or 17 bits but on average you will just need the smaller one which compresses the feed by 38% over the 2 byte representation.. That's a big savings for just some simple bit twiddling.

[nrp]

I could do that, but I'm trying to remain compliant with the HID spec so that people can more readily use existing drivers and libraries to interface to it. Windows 7 and later and Android for example have some level of built in support for HID Sensors.

[nrp]

With 16 byte endpoints, it reaches the full 1000 Hz. Getting the full 760 Hz rate of the gyro works! While that kind of sampling rate is probably overkill even for head tracking, being able to do 1000 interrupts per second is useful when multiple Trackers are attached to one base station. Two at 500Hz or four at 250Hz, for example.

It is also possible that the base station will be able to reach 1000 Hz even with larger endpoints, as the current design has the microcontroller clocked at 16Mhz instead of the 8Mhz the Trackers are at.

[brantlew]

That sounds awesome! Can't wait to get my hands on one of those suckers.

[CiNEPHiL]

[glove controller with a flex sensor in each finger comes to mind]

YESSSS!!! It's time to bring to live SOGO 7 DATA GLOVES!

http://members.home.nl/shadowrunner/sogo7/sogo7.htm

[zalo]

Congrats on the Oculus partnership!

Are you still going to sell these separately?

[Chriky]

Massive congratulations nrp! So good to see another thing from MTBS3D in the Rift

[VR2]

Hi NRP, please PM me as soon as possible. I have a nice opportunity for you for this hardware (order and more), you will not regret.

Best regards,

Michael

[PalmerTech]

Hi NRP, please PM me as soon as possible. I have a nice opportunity for you for this hardware (order and more), you will not regret.

Best regards,

Michael

NRP is working for Oculus VR now.

[MSat]

With what I've seen of nrp's work in this thread, I'm glad he's on Team Oculus, especially since now we'll get the fastest IMU on the market


Oh, and Oculus, please keep the GPIO on the board in place with access to them either via open firmware, or through the drivers/SDK. This combined with the new 7" form factor is going to make for a pleasantly hackable device, especially if the display interface board also ends up being flexible and future-looking.

[Chriky]

Is it still the plan to sell the trackers separately? If so, any idea on timescales? Thanks.