DIY 6 DoF Optical Positioning -Update: 7/19/13-

[zalo]

Update: 7/19/2013
Testing out how my system, the rift, and the leap play together. Positional tracking is the enabling technology that allows the leap to be used properly in VR since you have to know where the leap is relative to the rift. At the beginning I calibrated where the leap was by placing the rift on top of it.
http://www.youtube.com/watch?v=sAtwW6pzqGY

Update: 7/17/2013
It took a while but I finally got around to recording a more informative video of my system. In this video the tripod is on my desk and the camera is out of the frame above my head.

http://www.youtube.com/watch?v=6kCjvF2bKO0

The top left is a 3rd person view, the bottom left is rift-eye view, and the right is me. In the beginning you can see there are only two fiducials visible on the top of the rift. I then move through a wide range of motions that are possible with my system. The only flaw in this iteration are that the red LEDs appear dimmer to the camera than blue LEDs would. My next iteration will use brighter blue LEDs to fix the jitter (Brighter=Bigger Blob=More Pixels to Average for position)

To quickly recap for newcomers: What makes my system unique is that it can robustly determine the position of a user's head across a wide FoV and set of angles while using only a single camera and two tracked points on the Rift. The innovation (over similar systems like Johnny Chung Lee's) comes from the fact that I am using the Rift's IMU to fill in the missing orientation information that would normally be captured Optically (but would restrict the freedom of movement). I am using a PSEye and Processing to funnel the point tracking data to Unity at 125 Hz.

OLD:
I recently implemented support for the PSEye.
http://www.youtube.com/watch?v=XIbaxnpYUsM

OLD:
I've invented a new technique for 6 DoF positional tracking that uses a wii remote, a wireless sensor bar, and an Oculus Rift.

This technique is unique because it requires only two tracked points, and supplants the rest of the information with the Rift's IMU.

The goals of this system are affordability and flexibility. Using a single Wii remote and a wireless sensor bar (attached to the Oculus Rift) I am able to obtain absolute 6 dof positioning.
Soon I'll be able to replace the wii remote with a Leap because they will be implementing a blob tracking API compatible with Unity (increasing the resolution enormously).

Here's a picture of the current prototype:
http://i34.photobucket.com/albums/d144/Zalo10/photo_zps9a7f1cc3.jpg
The wii remote tracks the points on top of the rift. This perspective allows for full 360º yaw, a little more than 180º for pitch, and a little less than 180º for tilt.
Future prototypes using the leap will be mounted on a flexible arm attached to the monitor. The wii remote requires that height because of its limited 33ºx23º FoV.

Here's a video of it in action:
http://www.youtube.com/watch?v=RFrCf-O5Bck
Notice how the user is able to look in all directions.

Jerkiness is due to the low real resolution (128x96) of a wii remote IR camera and will disappear with a higher resolution camera.

I realize that now that Sixense has announced their solution, my solution will be competing with theirs. Here are the Pros and Cons comparing the two systems:

• Pros:
• Cost (>$3 for the wireless sensor bar, $30 for a wiimote or $80 for a leap)
• Latency (Leap has ridiculously low latency)
• Weight (LEDs can weigh next to nothing)
• No Magnetic Interference

• Cons:
• Requires Line of Sight
• Conical Capture Volume

Once the Leap is integrated, I suspect the resolution of the two systems will be comparable.

Any questions or suggestions?

[geekmaster]

I've invented a new technique for 6 DoF positional tracking that uses a wii remote, a wireless sensor bar, and an Oculus Rift.

Johnny Lee replaced the sensor bar with a pair of infrared LEDs, when he used your new technique back in 2007.

The nice thing about your implementation of this idea is the overhead mount, making optical occlusions much less of a problem without needed colored LEDs. But that could be a problem with overhead shoulder harness mounts being discussed in DIY ODT threads.

Don't feel bad that people have explored this territory before. Just take it as confirmation that you had a great idea, and you can have many more where that came from. So keep up the great work!

The sensational (since 2007, and still is) infamous Johnny Lee video:
http://www.youtube.com/watch?v=Jd3-eiid-Uw

Here is the discussion thread at Jonny Lee's website:
http://www.wiimoteproject.com/wiimote-d ... iscussion/

You can download his wii head tracking code here:
http://johnnylee.net/projects/wii/

And here are some MTBS3D posts discussing using a wii remote for positional head tracking:
http://www.mtbs3d.com/phpBB/search.php? ... johnny+lee

A more thorough dedicated 6 DOF discussion (including wii head tracking):
http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15040

Here is a much more thorough version, using colored LEDs for more robust (occlusion avoiding) head tracking:
http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=16072

[zalo]

Don't worry about it! I don't feel bad because I'm well aware of past implementations and their short-comings.

Johnny's system didn't use an IMU to compensate for changes in the orientation of the head. It always assumes you are looking straight at the screen with the LEDs on a plane perpendicular to the line-of-sight to the camera. This is bad because the depth component (how far the sensor bar is from the wiimote) is affected when you turn your head, bringing the LEDs closer together from the camera's PoV, making the system think your head is farther away than it is.

This system lets you look all around you and the depth component compensates for tilt of the HMD with the Rift's IMU (adding several layers of complexity to the math).

These improvements add a level of robustness that makes it viable for a consumer product (not that I plan on commercializing it).

[geekmaster]

Johnny's system didn't use an IMU to compensate for changes in the orientation of the head. It always assumes you are looking straight at the screen with the LEDs on a plane perpendicular to the line-of-sight to the camera.

This system lets you look all around you and the depth component compensates for tilt of the HMD.

Well, yeah, virtually all Rift apps do that. Johhny Lee's code works well with Deskope or GMsphere, for example.

There will still be a problem with overhead safety restraint mounts, so you need to allow for off-center mounting as well.

Keep up the development work on this. It certainly looks like a useful solution (especially with an offset wii or leap motion mounting position).

And the sensor fusion (i.e. "math complexity") you are implementing is where the obvious novelty (and utility) is found in your solution. Anything that extends existing solutions is a good thing. Your additional contributions are a good thing, and will benefit the VR (and especially Rift) community.

Thanks.

[brantlew]

Nice work. I'm surprised you are seeing that much jumpiness with the Wii camera - admittedly it's pretty low res but in my experience not to the extreme shown in the video. Is there any chance you are seeing that level of tracker aliasing from somewhere else?

Cool project.

[geekmaster]

... Jerkiness is due to the low real resolution (128x96) of a wii remote IR camera and will disappear with a higher resolution camera. ... Any questions or suggestions?

Although the wii remote camera is only 128x96, it uses grayscale interpolation to determine positional accuracy to 1024x768. This interpolation is similar to how IR touchscreens can return great accuracy with a small number of LEDs and IR sensors. It should not cause such jitter, as shown in the Johnny Lee videos. You do not need a real sensor bar. A pair of IR LEDs will suffice. The sensor can even use a pair of candles for the IR source.

[zalo]

Yeah, I thought the same thing, brantlew.

I just tried it again with some changes to the physical setup (my workspace is so cluttered!) and it appears to be a lot more stable now:
http://www.youtube.com/watch?v=IAXQ5CKzo54

EDIT: I'm aware of the 8x interpolation, but you need the finer details when trying to determine depth from screen-space distance.

Just look at the TrackIR 5:
Raw Sensor Resolution: 640 x 480
Reporting Resolution: 96,000 x 72,000

That would be some handy horsepower, but Natural Point has locked down their SDKs and Freetrack is a mega hassle to get into Unity (if it will even give you the raw IR points).
I had to switch to a mac just to find a wiimote library that would output the raw positions of the IR points.

[blazespinnaker]

Very cool project.

Are you going to extend this for hands and feet as well?

[nateight]

Great stuff, zalo. I've been saying this would be a decent path to quick-and-dirty positional tracking for ages now, but I've been too mired in a related project to get to where you already are. Sending you a PM now.

[budda]

Hi,

Its interesting the wii remote is coming out of 'retirement' in VR.

Much of the potential of the wii remote is still to be exploited, even 7 years after its release.

Many people still think of it as a toy and a gimmick, not as an advanced user interface device at a budget price.

Perhaps zalo can extend the angular range of the wii remote camera by placing a fisheye lens over the front. Both the large camera type fisheye lenses and CCTV type fisheye lenses could do the job.

Fisheye lenses have another advantage in that they concentrate the incoming light, and give better camera range sensitivity.

Its refreshing to see the wii remote being used in this way.


Thanks.

[WiredEarp]

I think the whole bluetooth thing has held back acceptance of the wiimote. When I played with it, it was a real PITA to get going and setup.
If they've improved this (cable connection or something?) then I might give it another shot. Perhaps I could mount one to my belt for tracking my hips.

Re Zalo's setup, it looks really good and I think having the sensor on the ceiling is the only practical way to sense the HMD in the vast majority of positions, when doing freestanding VR. For sit down, the sensor can probably be placed anywhere.

[cybereality]

Cool.

[mattyeatsmatts]

haha ^ textbook Cyber comment

[zalo]

Those are some great ideas, budda! I hadn't even thought about the fisheye lens idea. Using the lens from a Nyko Zoom would probably work best without any hassle, but I wonder if a fresnel lens will work...

Also, some guy on the free track forums suggested sanding down the curved tip of an led until it's flat to increase its emission angle. I'll probably try this one too since I have a sander handy.

Wiimote connectivity is still a pain in the ass (had to borrow a Mac to even make this demo) but its the only thing on the market less than $200 that will let you dump point tracking data into unity. If you have some real skills, a PSEye would also work.

[brantlew]

@zalo: The improvements look a lot better. It makes me think you might be getting reflections. You should try using a clean matte surface and turning off lights just to rule out the possibility of stray IR reflections adding noise to the setup. Also, if you don't want to sand the tips of the LED, you can clip the top with a pair of wire cutter pliers (just not too deep). That flattens the head and increases your emmision angle. Also, I found that WiimoteLib on Windows works just fine for getting the raw data. (FreePIE uses that library if you need reference code)

In my experience the Wiimote can give good performance, but I agree that Wiimote pairing is a huge PITA. . If only they could be auto-paired it would the perfect VR hobbyist device.

[colocolo]

this could be a perfect fit for gait direction tracking and positional tracking simultaneously.
I guess the Rift even wouldn't need positional tracking.
For tracking the position of a skeleton(view) it only needs one reference point. The rest is mainly rotational movements of a skeleton. Moving the head independently from the torso in translateral directions is a very unnatural movement and we never do this movement.

[PatimPatam]

Very good job zalo, fusing optical and IMU data has been on my to-do list for my posittron thingy since I got the Rift DK! Unfortunately been crazy busy for last 3 months and the small free time I got I used it for checking the forums, testing the Rift and other people’s demos, and showing it to friends and family. I finally have a few days off now, so I might start playing with all this again.

I agree that optical can have some advantages over magnetic tracking; another potential PRO that you didn’t mention is tracking distance (well we still have to see how good is the new Sixense wireless system going to be).

Also a bit surprised to see that you can get a little more than 180º pitch with only 2 tracked points, i would have though you could only get a little less than 180, at best!

[zalo]

Thanks brantlew and PatimPatam! The IR LEDs stick out over the top of the rift, so they're visible just barely over 180° but will be a lot more so if they weren't recessed and focused.

The leap is interesting because it will only be able to support tracking its own reflected light.
That's where these come in: http://www.naturalpoint.com/optitrack/p ... s-markers/
A little protruding IR retroreflector ball mounted at the corners of the rift should give better visibility than the LEDs which are slightly recessed/have low emission angles.

I am also considering using a Processing + a PSEye, and funneling the point tracking data into Unity via OSC. Does anyone have any experience with this? Taking the IR filter out doesnt look like it will be easy, but the resolution and FoV gains are too great to disregard.

[mattyeatsmatts]

I thought the leap had a pretty poor sight distance, you'll have to get that 9 iron pretty close to your noggin.

[CyberVillain]

but I agree that Wiimote pairing is a huge PITA.

My brother is working on a Wiimote lib based on the Dolphin project and it can pair very easy (Atleast with the Toshiba stack) works with Integrated Motionplus and all other wiimotes.

Just have to fix some kind of auto calibrater and then integrate it with FreePIE

https://github.com/maxmalmgren/DolphiiMote

[brantlew]

but I agree that Wiimote pairing is a huge PITA.

My brother is working on a Wiimote lib based on the Dolphin project and it can pair very easy (Atleast with the Toshiba stack) works with Integrated Motionplus and all other wiimotes.

Just have to fix some kind of auto calibrater and then integrate it with FreePIE

https://github.com/maxmalmgren/DolphiiMote

That would be great. For a brief time, I had a tele-whiteboard room implemented using the Wiimote whiteboard application and dual ceiling mounted Wiimotes, but it proved to be a pain to use partially because of the issue of having to pair the motes on every use.

[CyberVillain]

Dolphin auto pairs on the Windows stack, but for my Adapter (Asus little dongle) the window stack didnt work so had to change to Toshiba and with that stack Dolphin does not auto pair. But paring in the Toshiba stack is just a matter of clicking the wiimote in the list and you're done so not very hard.

Check out the above lib, its a Cpp lib wrapped with C so it will work with most languages, including C#

[zalo]

http://www.youtube.com/watch?v=XIbaxnpYUsM

I made a few upgrades to the point-tracking system by switching to a PSEye. To get the data from Processing into Unity, I'm now sending messages over OSC.

This makes tracking a lot more smooth and robust (sorry if it doesn't look that way, Fraps slowed everything down and I was holding the camera in my right hand and the rift in my left so it went out of frame a few times).

If anyone is curious, I'm using this example of Processing -> Unity Communication via OSC

I hope to make some more videos soon that do these upgrades justice.

[cybereality]

Awesome!

[Direlight]

Very cool,, wiimote IR runs at roughly 120hz too, it's basically 40 dollar controller and headtracker in 1.

Johnny's system didn't use an IMU to compensate for changes in the orientation of the head. It always assumes you are looking straight at the screen with the LEDs on a plane perpendicular to the line-of-sight to the camera. This is bad because the depth component (how far the sensor bar is from the wiimote) is affected when you turn your head, bringing the LEDs closer together from the camera's PoV, making the system think your head is farther away than it is.

Don't ask me how I know this (actually I was researching wiimote drift ) but they used similar setup in the game red dead redemption. The sensor bar calibrated the motion plus, which ended up not helping me but seemed cool reading about it.

Where is Johnny Lee anyway? Never could get anything he ever coded to work, but they were all his side projects to be fair.


Oh yeah guys, you can also use the new 9 axis wii controller on PC. Some guy is even trying to get the touch screen working too. It would actually make an okay rift controller for strategy oriented games.

[Chriky]

He worked for MS for a bit on Kinect and then moved to Google

[zalo]

It took a while but I finally got around to recording a more informative video of my system. In this video the tripod is on my desk and the camera is out of the frame above my head.

http://www.youtube.com/watch?v=6kCjvF2bKO0

The top left is a 3rd person view, the bottom left is rift-eye view, and the right is me. In the beginning you can see there are only two fiducials visible on the top of the rift. I then move through a wide range of motions that are possible with my system. The only flaw in this iteration are that the red LEDs appear dimmer to the camera than blue LEDs would. My next iteration will use brighter blue LEDs to fix the jitter (Brighter=Bigger Blob=More Pixels to Average for position)

[nateight]

Blue LEDs? Would that even work? I have it on good authority that Wiimotes are best served by ~880nm IR LEDs, and efficiency decreases the farther you move in either direction from that in the spectrum. Are you using visible light LEDs for the benefit of the viewing audience, or is there something I'm not understanding here?

Additionally, though I haven't tried this myself, I don't think you need a "sander" to give your LEDs a wider profile, just a sheet of fairly fine wet/dry silicon carbide sandpaper. The goal of that hack is just to scuff the hell out of the bulb so the light scatters close to its source, not change the actual shape of the plastic (though that could present additional benefits). Two IR LEDs seem to run very well off a single watch battery, though I probably should learn once and for all why you're not supposed to use them without a voltage-limiting resistor. Do you smell something burning?

[garagebattle]

colored orbs, one on each shoulder, one of each color

http://www.amazon.com/Color-Changing-Ni ... uckgo-d-20


with a bigger color balls, you could, in theory, move the camera further away? also going on the assumption you are using PSeye camera.

take it further, add a wide angle lens to the PSeye:

http://www.amazon.com/Detachable-Degree ... uckgo-d-20


thoughts? btw if this is nothing new my apologies, i dont read all the threads on this site.

[zalo]

Blue LEDs? Would that even work? I have it on good authority that Wiimotes are best served by ~880nm IR LEDs, and efficiency decreases the farther you move in either direction from that in the spectrum. Are you using visible light LEDs for the benefit of the viewing audience, or is there something I'm not understanding here?

Yeah, I ditched the wii remote in favor of a PSEye. Higher resolution, more control over the blob tracking algorithms, 125 Hz, visible light tracking etc. Oh, and it works on windows, which is evident in the video.

Also, on the rift are three "bright" LEDs hooked up in series with a resistor to a 9 volt battery. Two red ones and a blue one. I had to tape over the blue one with metal tape to block out its light; it's far more detectable than its red counterparts. Soon I'll be getting a package of those keychain LED lights that I'll simply be able to attach to the sides.


@garagebattle Thank you for your suggestions. I might try colored balls if my current LED idea doesn't pan out. Also, I've seen and been tempted by the wide angle lens before, but it's very difficult to install and I don't want to break my PSEye in the middle of development. I figure at that point I can also take out the IR filter too and switch back to IR LEDs. It also goes against the ease of use that buying an off-the-shelf camera would have; it would also spread my existing resolution across an even wider FoV.

[brantlew]

Wiimotes use 940nm LEDs I'm pretty sure.

@zalo: There is a company that specializes in creating lens mounts for a wide range of lenses on the PSEye. Can't remember the name right now but if you go search through the big PSEye Visual Odometry thread you can find it in there.

[FingerFlinger]

This one? m12lenses.com

[zalo]

http://www.youtube.com/watch?v=sAtwW6pzqGY
Testing out how my system, the rift, and the leap play together. Positional tracking is the enabling technology that allows the leap to be used in VR properly since you have to know where the leap is relative to the rift. Still haven't gotten brighter LEDs (and now I'm pumping TWO video feeds into my poor USB 2.0 ports at over 100 fps) so it's kind of laggy.