POSITTRON: yet another proposal for positional head-tracking

[PatimPatam]

WARNING: If you don't want to read quite a lot I suggest you skip directly to the YouTube videos further down!


INTRO

First of all I wanted to mention that i designed this solution mainly with the Oculus Rift in mind, but this concept could be applied to most types of HMDs, that's why i put it under the R&D section.

I think by now everyone in this forum understands the need for true 6DOF positional tracking so i won't go over it again. I will just stress what Mr Abrash from Valve mentioned at Quakecon: parallax is even more important than stereoscopy for VR/AR. Here's an interesting link:
http://cb.nowan.net/blog/2010/04/28/doe ... s3d-matter

The thing with positional tracking is that there are not only many different approaches to tackle this problem but also many different factors that determine which approach is best fitted in each scenario (seated vs standing play, short vs long distance, occlusion, magnetic interference, complexity of setup/calibration process, usability with backtops, 360 degree freedom of movement, etc). There's already some really good progress by members of this forum on systems designed for specific use cases, like Chriky's fiduciary markers for 1:1 real world location mapping or Brantlew's Red Rovr Motion for redirected movement in open spaces to name a few.

I however wanted to focus on a completely different use-case, a consumer-friendly approach to solve positional tracking for the average gamer playing at home. Here i believe the main priorities should be easy setup, robust seated play without angle restrictions and ideally 360 degree freedom of movement inside a small play area. For me easy setup implies having a maximum of one external tracking device which is easy to position (i.e you don't have to mount on the ceiling), something like the Wii or Kinect for instance. I also believe the overall appearance of the system is really important if you want a successful commercial device.


Future and available solutions:

Focusing on a consumer-friendly scenario i believe some of the most promising solutions on the long-term are magnetic tracking or PTAM/visual odometry. I think they are both very valid approaches, and probably the future for VR, but we're still not quite there based on what i've seen so far. On the short-term (as in available right now) one of the most popular approaches seems to be "outside-in" optical tracking systems. In general some of the advantages of an external, inward-looking approach are low processing requirements and low latency.

One example of outside-in optical tracking is the PS Move method, like seen in Project Holodeck (basically strapping 1 PS Move wand on top of the head and use 1 camera for tracking). The problem i have with this type of system (apart from looking quite dorky) is: what happens if you look 90 degrees up? with only 1 camera and 1 marker you will always face occlusion problems..

Another popular optical solution out there is the TrackIR. This is actually pretty close to what i would want from a tracking system. The only problem with TrackIR is again to do with angles and occlusion: it has a very limited range (about 45 degrees) which is ok if you have your vision fixed on a computer screen but not good enough when wearing a VR headset. Also the working range (about 1.5 meters) is a bit shorter than what i would like (ideally 3 to 4 meters).


My approach:

So my line of thought was: if we already have a proven technology that when it works it works well, why not simply try to extend it and solve the low-angles and occlusion problems by adding more markers on the target? And while we're at it, why not use one of the big hurdles of VR (having to actually wear a head-mount) in our advantage to integrate the solution?

A few months back i was following this very interesting topic on 6 DOF head tracking ideas:
http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15040

After a few posts on this and some other related threads i was quite surprised to find out that apparently no-one in the forum had tried to implement something similar before or given it much thought.. so i decided to give it a try myself as soon as i found some spare time!

Basically my main design goals were:

• 1- Positional tracking (real 6DOF) with 360 degree horizontal and 90 degree vertical freedom
  2- Maximum one external tracking device (camera), easy to setup, no need for user input or calibration
  3- NOT having "protruding antennas" (TrackIR style) or "external glowing balls" (PS move style).
  Reason A: they make the HMD more break-prone; Reason B: they don't look very cool

I think there's a place for a mid-term solution, something which might not be the definite positional tracking for VR but still better than existing alternatives.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


THE PROPOSED SOLUTION (proof of concept)


The HARDWARE:

First of all i would like to present "the truncated cube", in case you don't know him!




The nice thing about the truncated cube is that whatever way you look at it you can always see at least 4 of its faces. For pose estimation of a 3D object you kind of need to see 4 non-coplanar points at all times, so if we were to put markers on each face of a truncated cube this is could be a perfect match for our purpose. Fortunately it doesn't have to be a completely uniform polyhedron, a truncated cuboid also shares this characteristic. Another good thing about this type of polyhedra is that its shape can easily become the base of a HMD shell design.


Marker integration

The next problem that we have is that for pose estimation we not only need to "see" the markers but we need to be able to identify them, as in put a number to each marker and be able to tell which one is which.

Infrared (IR) markers are popular because they are very easy to discriminate from the background with proper filters, but being able to detect several colors can help to simplify the problem a lot. Colors on the visible spectrum are obviously harder to detect than IR but we have something that we can use in our advantage: since we are integrating these markers on a HMD we can actually control the background where the colors are displayed to make blob detection easier.


Here are some of my first experiments, where i decided to use simple color stickers on the darkest background possible. For that i used flocking paper found in telescopes, which is more practical and uniform than using black velvet for instance. Also i think i forgot to mention, i'm using a PS Eye as the main camera for the image processing.



With this setup i managed to get pretty good results; it was good enough to build the main blob analysis algorithm and i was able to discriminate 4 basic colors (r,g,b,y). As expected the problem was that the result was very dependent on ambient light, small changes would make the recognition fail. The obvious next step was moving to color LEDs.


Looking back finding the right LEDs to use was one of the biggest unexpected difficulties i found; they need to have a wide angle and display uniform light, and i had to test quite a few number with different current intensities and camera settings until i found ones that performed good enough.

I also found out that i had to reduce the camera gain and exposure almost to the minimum settings possible in order to get correct colors (not white over burn), this in turn helped to reduce blob "trails" in fast movements and also had another good effect: since the whole image is darkened, blob-detection is already half-solved because only bright lights stand out. This actually makes the flocking dark paper quite unnecessary, but i decided to leave it anyway. Here i made a silly mistake though: i made all these tests with the camera running at 30fps instead of 60 and later i realized this really affects the colors detected.


Here you can see the power source i used and one of the tests i made with LEDs:



So once i was happy with the LEDs i found it was time to build the main prototype. In theory we need to have only one marker per side of the truncated cuboid (16), this way we always have a minimum 4 non-coplanar points visible (and most of the time redundancy which is not bad). In practice though this cuboid has to be split in 2 parts (front and back of the HMD) so i thought that on the sides and top it would be better to have LEDs on both parts, even if they shared the same theorical plane. This is necessary to overcome occlusion by the head/neck itself.

Even though it wasn't strictly necessary i decided to use some pilot lights for the front and the back in order to have 2 more different types of markers that would help with the point detection. I also decided to put 2 LEDs on the top and bottom of each part so we could detect at least 4 when looking straight from the top/bottom, without depending on the other part of the HMD. Later i regretted this 2 decisions though, since they complicate the design, the detection algorithm and actually give more problems than benefits..

Another mistake i made in retrospect was to fix the intensity of the LEDs using voltage regulators LM317 in combination with fixed resistors.. i should have used potentiometers instead, in order to be able to vary the intensity of the LEDs without having to redo any soldering.. oh well..


Anyway in the end my final design contained 22 individual markers of 6 different types: r, g, b, w, B pilot, Y pilot.

Obviously in order to detect correctly the 22 different markers it would be ideal if we could have 22 different colors, but that is really not practical and i would dare say not possible, at least with the PS Eye hardware. However having 4 or 6 different types of markers is enough for point detection if you distribute them more or less cleverly and take into account the surrounding markers while trying to identify an individual one. The LEDs don't have to be on the exact positions i used or same colors, you can get creative with this. The only 2 rules are always 4 non-coplanar points visible and colored in a way it's possible to match them with the complete 3d model.


Some pictures of the building process:










DIY rough cost estimate:

• -20 LEDs -> 15$
  -2 pilot lights -> 8$
  -1 PS camera -> 15$
  -LM317Ts, resistors, wires, etc -> 5$ aprox
  
  TOTAL aprox cost -> 43$

I think if bulk buying the components this could be reduced greatly, probably under 12$ if you don't count the camera cost. If DIYing you may need to add an AC/DC power supply - aprox 15$, which of course is not necessary if it's integrated with the existing HMD supply.


And here go some images of the finished prototype:








The SOFTWARE:

For the software side i decided to go with C/C++ since having the lowest possible latency is really important. I also decided to use OpenCV for the image analysis side and OpenGL to show some results. I've never used OpenCV before and i found it very good in general, apart from some outdated documentation in places.


Brief program explanation:

Initialization:

• - Create / store 3D model of markers
  - Apply gaussian blur to the image to prevent false blobs from being detected

Core functionality:

• - Blob analysis: detect ellipses, obtain center coordinates, color and type
  - Point detection: Match results of blob analysis with 3d model, numbering each marker
  - Pose estimation: use the matched 3d model points and 2d image points to estimate the pose of the object

Show results:

• - Apply Kalman filter
  - Paint results of blob analysis and point detection with OpenCV
  - Paint simple in-game map with OpenGL

To start off i decided to create my own blob detection algorithm in order to have greater control of the conditions and results. I was pleasantly surprised that my custom scanning patterns proved very reliable detecting up to 5 different colors (r, g, b, w, y) under very different light conditions (complete darkness, artificial light and direct sunlight) and up to 1.5m distance approximately; not as good a distance as i would like but I'm quite confident that this could be improved to 2.5 - 3m without too much difficulty. Still, for now it was good enough to move the main focus to other parts of the problem..

For the point detection i faced the problem of having to identify 22 different markers having only 5 different colors. In order to do that I created quite a big number of fairly complex rules for each marker, that take into account not only the color of the current marker but the relative position and angles of the adjacent ones from all the possible views. This part can still be improved as sometimes the detection fails, but at this point i realized it would be better to rethink a bit the distribution of the LEDs on the hardware before trying to improve the software side.

For the pose estimation part i decided to use the OpenCV implementation of the POSIT algorithm. Unfortunately is not as simple as calling a function since the input expects the whole list of points of the object you want to detect, starting always with the same origin point, and for each frame we only have a small subset of these (the points visible by the camera). I managed to get around this by using a few tricks, which i won't go into detail for now

After we have the position and orientation values I thought it would be cool to show a simple real-time representation of a room, so it's easier to evaluate the quality of the output. To do this i created my own OpenGL "engine", so i could have total control of how the combination of inputs affect the movement "in-game". The map i used is a modified by hand version from one of Nehe's OpenGL tutorials: the result is a funky colored temple, with some kind of "tombstones" inside. Head motion detection is exaggerated on purpose for demonstration, should be toned down when using it with a real HMD.


[youtube-hd]https://www.youtube.com/watch?v=EzC-HDpv2xw[/youtube-hd]


Results:

I think the first results although not perfect are quite promising, as i managed to get a fairly reliable tracking with a good response and good precision, which even at 30 fps feels very fluid. I have tested 60 fps successfully but the blob analysis results are not as good as with 30 fps because the intensity of LEDs is not properly adjusted. This should be fixed with a fairly simple hardware modification.

In the video the little "wings" that you can see on top and bottom of the prototype were added to simulate the top part of the head and the neck blocking some of the LEDs. You can still see some small glitches from time to time due to the LED detection failing, as i mentioned it could probably be corrected with more complex rules but i have already thought of some modifications in the hardware and marker distribution to prevent it.

Also most of the jitter you can see (specially with the Kalman filter off) is related to orientation, but this system main purpose is to get positional tracking, so in combination with an IMU for orientation values and leaving the optical tracking to get only the positional information, the final result should be much better. Even though finding orientation was not the main goal of this project, these values are still very usable as you can see on the video.


[youtube-hd]https://www.youtube.com/watch?v=nsl43qbMnOA[/youtube-hd]


I don't really know the latency introduced by the PS Eye camera itself but i decided to take some measurements in order to see the latency introduced by my software:

• - normalized box filter blur = 3.38 ms
  - blob analysis = 4.20 ms
  - point detection + pose estimation = 0.06 ms
  - opencv paint results = 0.60 ms
  - opengl paint results = 0.06 ms
  
  TOTAL 8.30 ms aprox

First i used OpenCVs Gaussian Blur, but that was taking about 5.20 ms. Later i i realized i could get away replacing it with a Standard Blur (Normalized Box Filter). Blob detection still works great, and latency was reduced by almost 2ms (from 5.20 to 3.38 ms). I believe there's still room for improvement in this area though.


Recap of current prototype limitations:

• - Blob analysis distance 1.5m. With hardware and software modifications could be improved to 2.5 - 3m
  - Point detections fails sometimes: could be fixed using a different LED distribution
  - 30 fps. 60 fps should be easy to obtain (simply the intensity of LEDs is not properly adjusted). 120 fps possible (at the expense of detection distance due to reduced resolution)

How would a real HMD that uses this system look like?

They say a picture is worth a thousand words, so i decided to add a few 3d renders to better show how the idea could be applied to real-life HMDs. Please bear in mind that they are not intended to be very accurate, mechanically correct or even have the right proportions..

The HMD would have to be separated in 2 different sections, and these could be connected by a telescopic tube for instance, used to maintain rigidness between both parts and to measure the distance between them (in order to feed the tracking algorithm with the exact 3d model). This same information could be useful for other purposes like feeding binaural audio processing with the approximate size of the head.


Original idea for a full 360 degree positional tracking HMD, on which the cardboard prototype of the pictures and videos is based:



Front and back don't really need to be equal, they simply need to have the markers on fixed positions. I think if this system was applied to a commercial HMD the back part could be used for connectors or to store a wireless receiver and a battery for instance, or simply to counter the weight of the front part and even the center of gravity of the head mount. Maybe following a similar design to some of the 90s Virtuality systems. Also if having a separate back part of the HMD was too much of a problem, the idea could still be used with only the front part, limiting the movement to about 100 degrees in each side (200 total) which would still probably be usable in VR if you keep this limitation in mind while playing.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


SOME ADDITIONAL THOUGHTS

Even though my main priority with this project was seated play, i believe that in combination with a HMD wireless connection and with some improvements on distance detection you could have complete freedom of movement limited to a small area (the FOV of the PS3 camera at a maximum of 3 meters distance approximately). As it is now i think it's good only for seated play, standing/rotating in place could work but you would have to mount the camera on some sort of stand (at head level) and not have much margin to move around.

As mentioned earlier it's clear that for a consumer device it would be ideal not to have an external tracking device, but i believe having only 1 camera is acceptable, as most people are used to devices like move or Kinect. In fact it could even be seen as a "safety feature": when you walk out of the FOV of the camera the system could pause and warn you. Finally i would like to point out that sometimes we geeks tend to forget that looks is one of the most important aspects for a consumer device. I know as it is now my little prototype doesn't look awesome (and looks are very subjective), but i think it has the potential to look pretty cool. Personally i think it has a Cyclops meets Iron Man kind of vibe


General Limitations:

• - The main "problem" of this solution is that the case design of the HMD needs to be adapted to it. I want to make clear that this whole idea is not very practical if you want to add positional tracking to an existing HMD (although everything is possible). It is intended to be part of the shell design for a new HMD built with optical tracking in mind (either DIY or commercial).
  
  - Accuracy degrades with distance, still quite good from 1.5 meters. Could be improved.
  
  - If someone walks between you and the camera while playing you can lose positioning. Again maybe this could actually be turned into a good "feature": If someone occludes the camera for more than 3 seconds it could be a way to communicate they need your attention. It would be good to have a method to do this as someone already suggested before:
  http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=15413

Rift developer kit add-on?

Here's a possible idea for a Rift developer kit add-on, or as i like to call it "Rift-Hugger" (inspired by the face-huggers from Alien!). The back part could be used to store standard batteries that would power the LEDs. My guess is that the front plate of the developer kit can be easily removed with the clips on the side. If not the add-on could be simply attached on top of it. The moving parts on the side that hold the Rift could have some sort of locking mechanism as well.





As you can see, as a separate add-on it wouldn't be ideal, it would have to be targeted to a specific HMD model and it wouldn't be easy to DIY, but it could still be done (and very cheaply in fact if mass-produced in a factory).


Possible next steps / future improvements:

• - Build another improved prototype: modify LED distribution, separate circuits by colors, use potentiometers instead of resistors, etc.
  
  - Leave orientation to nrp's adjacent reality tracker IMU and use optical only for positioning; unfortunately it's not available yet. Reality tracker should be much more accurate and precise for orientation, play with fusion algorithms.
  
  - Adapt prototype to developer Rift size and shape. As i mentioned before the ideal path would be to design the casing with the optical tracking in mind not the other way around, but after seeing the prototype pictures i think we could do something with it. As it is (without adding a rear part) it would not be possible to have the full 360 deg but i reckon i could get about 240 deg lateral tracking (120 rotation to each side), something still very usable in many scenarios, and a nice step up from TrackIR for instance.
  
  - Increase capture distance by software (adjust color thresholds depending on distance between LEDs).
  
  - Improve blob recognition to allow lower resolution input (120 fps mode).
  
  - Use a cheap graphics card (CUDA) or a FPGA to speed up the image processing?
  
  - Track 2 headsets at the same time 1 with camera?
  
  - Extend system for hand tracking?
  
  - Add FreeTrack support / create a plugin for FreePie / create a new VRPN driver?
  Still have to look into these options and decide which one should i try first..
  It would be really cool if I could test the prototype with an existing game that supports 6 DOF, like Arma 2 for instance.

About the POSITTRON name:

I have to say that i'm not really planning to patent it or anything, i put a name just for fun. I think it suits the project quite well, and sounded better than the boring alternative thread title "HMD optical marker integration".

The POSIT part is quite obvious, as it's a positional tracking system, and i also happen to use the POSIT algorithm to solve the pose estimation problem. On the TRON side, well TRON is like THE original movie about VR (although not very realistic) and they also happen to use helmets with bright LEDs surrounding them, not too different to my design really! Oh, and I quite like the English band Ladytron

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


TO FINISH OFF (SALES PITCH STUFF)

So I know the proposed solution is not the holy grail of tracking, but i believe it's a realistic and ready to go solution that solves some real problems. Let me finish with a list of benefits (box ad style), they are NOT all true with the current prototype version, but could be on a more finalized design:

• - Precise (sub 1mm) positional tracking (6DOF) with 360 degree horizontal and 90 degree vertical freedom of movement. No magnetic interference, no self-occlusion, no drift.
  - Working range from 50cm (in front of a computer) to 3 meters (in front of a TV set/play area). Seated or standing play. Wireless!
  - 120 fps ready, low latency, fast processing (not computationally expensive).
  - Uses open-source software and of-the-shelf available hardware (cheap components).
  - Only one external camera needed which can be placed anywhere in front of the player, no need for calibration.
  - It can look pretty cool, i believe

Oh and of course in marketing terms, if this was to be added on top of a 9 DOF tracker like the adjacent reality, i guess it could be called a 15 DOF system:
3 DOF accelerometer + 3 DOF gyroscope + 3 DOF magnetometer + 3 DOF optical orientation + 3 DOF optical positioning = 15 DOF!


I have to say that it has been quite a challenge to find the time to implement all this; i had the idea burning in my head since the end of july more or less, but couldn't start working on it until september. Since then, with a full-time job as web developer, i have been able to spend only a few hours a week on this project..

Anyway it seems like Oculus are looking into using PTAM for localization/positioning, which i think is the right move in the long term; I really hope they can manage a properly robust and low-latency implementation. Still I think something closer to my design could be a simpler and safe alternative / plan B, in case the PTAM research doesn't pan out in time for the consumer Rift. I also think this design fits quite well with Palmer's approach of "doing more with less".

Personally i believe this solution would be ideal for the developer version that we will get in about 3 months, as it would let developers start programming and experimenting with positional tracking. Also it would help the Oculus team design the SDK with positioning as a main feature, and to receive feedback from the community. Unfortunately it's probably a bit too late for this to happen on the Kickstarter revision.. who knows, maybe there's still time to add a few holes on the Rift shell as part of the fabrication process heheh.

In any case i think this could still be useful to developers and DIYers as an alternative to strapping a PS Move, a Razer Hydra or a mobile phone on their heads I'll be more than happy to help if anyone wants to give it a go. Also of course any good ideas are more than welcome, either from the HW or SW side. I'm sure this particular design and tracking software could be greatly improved but i think my little demo is enough to prove that this is an avenue worth exploring.

Ok, finished (for now), and sorry for the long ramblings!



Acknowledgedments:

I would like to thank some members of this forum like Brantlew and Chriky who kindly answered some of my newbie questions when i got started, and specially Edz who gave some good advice when we were discussing all this back in July/August. Last but not least i would like to thank my lovely girlfriend for her craftsmanship input while building the physical prototype and for her constant support in all my crazy endeavors!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE LOG

Update 01: (01-26-2013)

- Replaced Gaussian Blur with a Standard Blur (Normalized Box Filter)
- Added 3d model pictures to show how the project could be integrated with an actual head-mount
- A few spelling corrections

[Fredz]

Very good job indeed, nice that you had a try at this !

I have only skimmed over your post but I've seen that you mentionned Kalman filters. I don't know if you use one but you may have a look at the Madgwick algorithm which does basically better with less resources and complexity : http://www.x-io.co.uk/open-source-imu-a ... lgorithms/

Too much to read for now, I'll probably comment a bit more tomorrow.

[Naru]

Nice! I'm working on something similar, but am more focused on my diy Rift at the moment so haven't been able to begin experimenting. I'm glad you got it working and that you went as far as you did in building and testing it all.

I think for a larger setup where people want to walk around, multiple cameras wouldn't be a bad idea. It's not necessarily a bad requirement either. Have you come up with a head mountable design of the truncated cube? When I finish my diy Rift I'll be trying out different things for positional tracking, and this would work if it had a more hmd friendly design. Optical flow from a head mounted camera is my last resort. I'm holding off on that to see if Oculus gets it done faster.

[cybereality]

Wow! That's some post, dude! And, yes, I read the whole thing and watched both videos.

I think what you have is looking really solid. There were a few small hiccups with the tracking, but overall very nice.

The one thing I don't understand is how will you wear that thing? Is the idea that it would be part of the HMD? Does the whole thing cover your head? Seems like it might be a little bulky to have that thing over your head. Otherwise it seems like a good idea.

[brantlew]

Wow! This is some really great work. I'm surprised you were able to sit on it this long, but it's a really nice surprise to get it all at once like this almost fully implemented. I'm so glad someone finally did this because the concept has been talked about so many times as a "given", but nobody just sat down and worked through it all. Kudos.

A couple thoughts. I notice that the gaussian blur eats up quite a bit of your time budget. Have you tried using simpler filters? A simple averaging filter runs probably 100x faster than a gaussian and might work almost as well. If not then there are ways to create sort of a "stepped pyramid" filter that mimics some properties of a gaussian but can be optimized a lot more. It would require custom filtering code (instead of OpenCV) but since that's half your CPU it might be worth spending some time trying to eliminate the gaussian.

While I'm on the subject, I wonder what the "acceptable" CPU budget for tracking should be? 5% of the CPU for your algorithm certainly seems acceptable (especially if you could cut that in half). It's even more of an issue with optical flow type tracking. I wonder if 10% or even 15% CPU would be an acceptable loss? As an experiment, I ran libviso simultaneously while running a couple of games, and I only dropped about 5 fps when optical tracking was operating. I guess it would depend on the games and the computer but that seems like it could be an acceptable loss.

Anyway, good work. It seems like a very good solution. Especially for seated play and fused with an IMU.

[Chriky]

Really awesome stuff! Can't write a proper post at the mo but just a quick mention that blurring horizontally then vertically is way faster than doing a 2d blur in one go. A few other things to say but will have to wait!

[PasticheDonkey]

great work getting it all working. i'd still prefer the camera on the head and the LED markers placed around a room tho.

[Randomoneh]

Why are you so awesome? Do you have a history in electronics / mechanics / IT?

[PatimPatam]

Thanks for the feedback guys, it really means a lot!

@cybereality and @everyone
Yes i should have probably stressed this out a bit more: the idea is that it would part of a HMD shell design, but it wouldn't need to cover all your head. It could be split into 2 different "small" sections (front and back), the only limitation is that both parts would need to be connected semi-rigidly, so you can feed the algorithm with the 3d model of the markers. There's some more info on the "General approach limitations" section, still I will try to post a 3d sketch or something so that it's a bit more clear!

Also with a bit more work on the hardware side and some IMU fusion, the small hiccups should be gone!

@Fredz
Thanx man! I will definitely have a look at the Madgwick algorithm to see if it gives better results! I find that Kalman already has almost negligible computational cost though (is part of the 0.06 ms taken by point detection + pose estimation).

@Naru
I agree with you, even though is not my main focus right now, i think this idea could work as well on a larger setup using multiple cameras. On a laser-tag type of scenario for instance you could have each player wearing a HMD-posittron design and have a few cameras on each room or corridor covering all possible angles.

I have only made some sketchs by hand of a head mountable design, i will try to do something a bit more professional-looking, but it will probably not be 3d-printer ready! Unfortunately the big-ass 7 inch screen of the developer Rift is probably not the best fit for this type of design, but i'm quite confident something can still be done with it.

@brantlew
Thanks for the suggestion brantlew, yeah i didn't realize gaussian was so cpu-hungry until a few days ago.. i really should have a look at other filter options!

@Chriky
Thank you for the tip, I will try that!

@ PasticheDonkey
Well i think it really depends on the use you want to make of your HMD. My thought was that for the average gamer it would be easier to simply stay in front of a camera than having to mount markers around the room.

@ Randomoneh
Hahah, thanks man but i wouldn’t call myself awesome! I have a master’s degree on computer science, specializing on computer vision and 3d graphics. Unfortunately for circumstances of life I never managed to get into this industry professionally (although i wouldn't mind to!).

Also I don’t really know much about electronics, I just did a bit of research once I decided to build this prototype.. Actually I had never soldered a single thing before this!
Still, all of it was a very interesting learning experience!

[EdZ]

Once the Adjacent Reality tracker is available, could you use the orientation input from that to constrain your model, cutting down on tracking time (already pretty low!) and relaxing the requirement for unique marker layout?

[PatimPatam]

Hello EdZ, yes having the orientation info could definitely help relaxing the marker layout requirements, although having some redundancy is never too bad..

About cutting down the tracking time, the fact of having the orientation info probably wouldn't help much directly, because most of the time is taken by the blob recognition algorithm. However it would help indirectly if we could reduce the number of colors that we need to detect for instance.

Also there are tons of things to try on the sensor fusion side: for example (and simplifying) if we have a 60 hz input from the positional tracking (PosiTTron) and a 120 hz input from the orientation tracking (Adjacent Reality tracker) i guess we could predict with fairly good accuracy the positional values "missing" and have a pretty nice 6 DOF 120 hz output.

[mahler]

.... you may have a look at the Madgwick algorithm which does basically better with less resources and complexity : http://www.x-io.co.uk/open-source-imu-a ... lgorithms/

The Madgwick algorithm is a specialized filter for acc, gyro & mag sensors.
I'm not exactly sure what the major improvements are over Kalman-filter based approaches (read the pdf for that), but you can't just simply apply that algorithm in this case. It will be helpful when he starts fusing with an IMU or MARG for orientation.

@ PatimPatam
This post is really wonderful. It taught me new things, gave me hope about a realistic and cheap solution for positional tracking and is really impressive how you did work on so many aspects: programming, electronics and creating the cubes. As you said, it seemed possible, but it just hasn't been done and/or documented before. This makes it tangible and people can better focus on the issues that still remain.

[PatimPatam]

Thanks a lot mahler, that is very encouraging!

[Chriky]

Do you think the system would work with an LED setup more like this...



If so, I think it has life as a competitor to TrackIR, but essentially using colours to get more angle coverage (and also, being way cheaper!)

[PatimPatam]

Well, i think the best way would be to have the LEDs integrated with the HMD itself, but an external “adapter” similar to your sketch would be possible I believe.

It would need to be a bit bulkier though, because the LEDs cannot be co-planar like in the front of your drawing, and having some of the markers on a thin frame would be problematic as well. The reason is that it helps a lot that the LEDs sit on an actual surface with a different angle than the rest of surfaces, so that 2 LEDs cannot “mix” from any possible viewing angle.

With all than in mind I think it would be complicated to make a design of an add-on than fits on any HMD, but it would be possible to have an add-on designed for an specific HMD model. I’ll post soon some 3d renders of both my original idea for a 360 angle HMD, and how we could adapt it to the Rift developer kit.


Oh and managed to reduce blur time by 2 ms btw! will update that on the next post too.

[PasticheDonkey]

i'm thinking it's a bit over complicated now. more lights than you need. 9 lights with three colours positioned correctly would be enough. i may try too make a diagram too show the arrangement i'm thinking of.

edit: it's essentially this but with a green triangle of lights on the back which could be attached to the strap. so you should always be able to track a red, blue or green triangle, or a red red green triangle, or a blue blue green triangle, or a red blue green triangle. tracking can be handed off from one to the other based on the constraints of the triangle you are tracking first.

[PatimPatam]

Good thinking PasticheDonkey, but here are a few reasons why i believe your approach wouldn't work:

1- From a pure geometrical perspective, a 2D view (image) of 3 3D points with known relative positions to each other (triad) is not enough to determine the 3D plane which contains them (and therefore position / orientation). This is known as the P3P problem, and there are actually up to 4 different solutions for each possible input. I believe TrackIR can get away with it because they have a very particular distribution of markers, constrained to fairly narrow working angles, that allows them to discard the false positives. The problem is that these properties would be very hard to maintain when you are mixing markers from different triads around the whole 360 degrees.

As an example you would have this issue when the camera is looking straight to the right side of the HMD, when you would have one green point on the left and 2 red points on the right.



I think 4 triads where you can see at least 1 whole triad from any perspective would be on the limit of being usable, but that makes 12 markers already and you would probably have to use sticking out "antennas" like trackIR does, which i didn't want to do. Maybe in combination with the orientation information from an IMU it could also be possible to filter all the false positives, but so far i based my solution purely from an optical perspective. Apart from that you would still have the following 2 problems.

2- What happens if you look 90 degrees up? the camera would only see 1 red marker on the left and 1 blue marker on the right, hardly enough to determine position and orientation. Similar thing would happen on the sides and back of the HMD.

3- The "green" triad at the back could not be attached on a flexible strap. That would make the distance between the front and back triads completely random, and impossible to determine a constant point at the center.


Having said that i agree that my design could be simplified, as i mentioned on my first post. I think we could easily go from 6 types and 22 total markers to 4 types and 16 total markers, probably less if you add an IMU. These past few days I've been very busy with other things.. i hope this weekend i will have time to finish off some renders which will help understand how this whole idea could be applied in real-life.

[brantlew]

In the diagram you provided - confusion between those two projection is only possible if the size of the triangle is allowed to stretch. If you have a strictly known and constant triangle size, then this particular projection is not possible (at least within the limits of your camera resolution) - no?

[Mystify]

In the diagram you provided - confusion between those two projection is only possible if the size of the triangle is allowed to stretch. If you have a strictly known and constant triangle size, then this particular projection is not possible (at least within the limits of your camera resolution) - no?

Think of it this way:
If the triangle is parallel to the viewing plane then there is only 1 place it could be.
If you rotate it along and axis containing those 2 red dots 15 degrees, you will see the green dot move towards the red dots. Did it rotate forwards or backwards? You can't tell, both positions would look the same(ok, if it was known to be 15 degrees, perspective may come into play, but there is some pair of angles that you could not distinguish).
And if the red dots are not perfectly vertical, you have to figure out which angle they are at as well. There are several configurations of lights that could explain the same image.

[PasticheDonkey]

i thought the constraints of the neck and a camera above the seated hight of the player head would mean you wouldn't get 90 degrees up from it. and yes you could need sensor data aswell.

wouldn't perspective make the triangles not the same. you could also mostly track 2 triangles at times where that confusion would occur as only one could be rotating around it's central axis giving that problem.

with the green triangle you'd sort of have to calibrate it's size and shape from watching it's points movements interms of other triangles. once it's on the head it'd remain the same size tho. or it's on a fixed piece attached to the strap buckle.

it all makes the programming too tricky i guess for amateur stuff. too much like work.

and my mock up photo doesn't show this but i would have the triangles more angled so they would be angled towards the equidistant point between the other 2.

[brantlew]

Yes I can see now the situation where 2 angles (one forward and one backward) will occupy the same projection. It's just not the one illustrated in that diagram.

[PatimPatam]

@brantlew

I believe the diagram is correct. Maybe it's not easy to appreciate because of the perspective, but no the triangle is not stretched. It's exactly the same shape and size, only slightly rotated on the axis formed by the 2 red points. As Mystify explained think of the 2 possibilities as mirror images, where the mirror is a perpendicular plane to the line of sight, situated between both triangles.


@PasticheDonkey

As I explained on my main post one of the points was that the camera should be easy to position in front of the player, without having to mount it on the ceiling. And no, perspective would not allow you to differentiate between the 2 possible solutions.


I made this very quickly, maybe I’ll make another diagram tonight from another perspective. This is actually a very well documented and studied geometrical problem.

[PasticheDonkey]

mines clipped on top of my tv/monitor. i know this isn't the case for everyone tho.actually when leaning back the bottom two green lights should become visable but they could get occluded.

and again you need to uses parts of other triangles to make it work they'd just get bigger or smaller in those rare cases. part of the design was that you could always see a triangle and the edge of another. make them equilateral and it's easier

[brantlew]

I believe the diagram is correct. Maybe it's not easy to appreciate because of the perspective, but no the triangle is not stretched. It's exactly the same shape and size, only slightly rotated on the axis formed by the 2 red points. As Mystify explained think of the 2 possibilities as mirror images, where the mirror is a perpendicular plane to the line of sight, situated between both triangles.

Ah ok. I get it.

[PatimPatam]

and my mock up photo doesn't show this but i would have the triangles more angled so they would be angled towards the equidistant point between the other 2.

If you don't want to get into self-occlusion problems, i don't see how you can do that without using antennas (a la TrackIR) that stick out from the HMD (which i didn't want to do).

Sorry man but i don't really have time to discuss more in detail your particular design here, i tried to explain why i didn't use it myself.. Maybe with the changes you mentioned on the last posts and a bit more refinement it could actually work; perhaps you or someone else interested could try to implement it, or simply post a more detailed mockup on the R&D section in order to have a starting point.

Best of luck! (no irony/sarcasm there!)

[PasticheDonkey]

thanks for helping me keep in mind the concerns some people would have. good luck with your own refinements and i look forward to seeing your version through out it's stages.

[PatimPatam]

Ok, small update:


1- After a few tests i realized i could get away replacing the Gaussian Blur with a Standard Blur (Normalized Box Filter). Blob detection still works great, and latency was reduced by almost 2ms (from 5.20 to 3.38 ms). Thanx brantlew for the suggestion, it's not like 100x faster, but a nice boost! I believe there's still room for improvement in this area though.


2- Since i realized the main problem of my first post was that it wasn't clear enough on how this whole idea could be applied to real-life, i decided to add a few simple render mockups. Please bear in mind that they are not intended to be very accurate, mechanically correct or even have the right proportions.. Here are some examples (more on the edited main post):


- Original idea for a full 360 degree positional tracking HMD, on which the cardboard prototype of the pictures and videos is based:




- Rift developer kit add-on, or as i like to call it "Rift-Hugger":




Anyway i think these 3D models show what i have mentioned before, that this could be a pretty simple solution if it was integrated as part of the design of a new HMD shell. As a separate add-on it wouldn't be ideal, it would have to be targeted to a specific HMD model and it wouldn't be easy to DIY, but it could still be done (and very cheaply in fact if mass-produced in a factory). I also took this chance to learn a bit of Sketchup + Maxwell, which was pretty cool for me.


Let me know what you think!

[brantlew]

Looks very nice. Glad the faster smoothing works for you.

[Mystify]

Just thinking bout this from a stylistic perspective, but do the lights need to be dots? You just need to detect the blob of color, so the actual device could have a glowing logo or other cool symbol something instead, right?

[cybereality]

Looking good, PatimPatam. The renders definitely help with visualizing the product.

[PatimPatam]

Thanks guys!

Just thinking bout this from a stylistic perspective, but do the lights need to be dots? You just need to detect the blob of color, so the actual device could have a glowing logo or other cool symbol something instead, right?

The lights don't really need to be dots but they would have to have a simple shape which is easy for the blob detection algorithm to find where it starts and where it ends. Simplifying a bit they couldn't have "holes", so complex symbols like letters wouldn't work very well. But triangles, squares, pentagons etc should be fine. Actually rectangles is a particular case that could be quite interesting.. if you take it to the extreme you could create some cool "paths of light".

Something like this should work for instance (quick and dirty example!):

[Mystify]

Thanks guys!

Just thinking bout this from a stylistic perspective, but do the lights need to be dots? You just need to detect the blob of color, so the actual device could have a glowing logo or other cool symbol something instead, right?

The lights don't really need to be dots but they would have to have a simple shape which is easy for the blob detection algorithm to find where it starts and where it ends. Simplifying a bit they couldn't have "holes", so complex symbols like letters wouldn't work very well. But triangles, squares, pentagons etc should be fine. Actually rectangles is a particular case that could be quite interesting.. if you take it to the extreme you could create some cool "paths of light".

Something like this should work for instance (quick and dirty example!):

Yeah, that looks cooler.

[WiredEarp]

This is great work PatimPatam! It looks like it will be very good for Rift tracking - optical, and with much better coverage than TrackIR.

I like your renders as well, they look very professional. What software are you doing them in?

[virror]

Very cool!
Maybe i missed it but what are your plans with the software for this product? Are you planning to sell it or go open source?

[PatimPatam]

@WiredEarp

Thanks a lot man!

I'm using the free versions of Google Sketchup + Maxwell render, which i just learned how to use during the last few days.. I had some previous experience with Lightwave and Blender, but i was quite surprised on how easy is to learn Sketchup. After watching 4 or 5 basic tutorials on youtube i was able to produce some pretty decent results i believe..

You won't be able to create Pixar quality models, but i think is very intuitive, especially for non-organic architecture / interior design / prototyping.


@virror

I'm still considering different options.. i think probably going open-source is the most likely to happen.

I'm not sure if that would work very well though, because at the moment the software implementation is very dependant on the hardware / LED distribution.. The problem is that i don't think the hardware would be easy to DIY, even if i gave very specific instructions. So basically i don't know how it could work without having a common consumer product or "shared base"..

[BillRoeske]

I'm not sure if that would work very well though, because at the moment the software implementation is very dependant on the hardware / LED distribution.. The problem is that i don't think the hardware would be easy to DIY, even if i gave very specific instructions. So basically i don't know how it could work without having a common consumer product or "shared base"..

As far as the housing goes, one of the cafe press-style 3D printing websites that have sprung up might be able to get the job done. The end user would still need to be able to source and solder the LEDs, but that's at least a more common skill than fabrication.

PS - Great demo, btw!

[PatimPatam]

I'm not sure if that would work very well though, because at the moment the software implementation is very dependant on the hardware / LED distribution.. The problem is that i don't think the hardware would be easy to DIY, even if i gave very specific instructions. So basically i don't know how it could work without having a common consumer product or "shared base"..

As far as the housing goes, one of the cafe press-style 3D printing websites that have sprung up might be able to get the job done. The end user would still need to be able to source and solder the LEDs, but that's at least a more common skill than fabrication.

PS - Great demo, btw!

Yeah that could work i guess, but there would still be a lot of soldering involved.. more than most people would be willing to do probably (maybe i'm wrong).

[Moriarty]

Nice work PatimPatam ! I like the elegance of both the working principle and the design.
Posittron is a cool futuristic name also, in combination with the LEDs it reminds me of Data's "positronic" brain from Star Trek :

[virror]

The soldering seems very basic and there are lots and lots of DIY projects that has a lot more advanced hardware that that, so i don't think that it would be any problems at all as long as the instructions were clear enough. I wish i had enough money to preorder the Rift : /

[PatimPatam]

Hahahah good find DrZimmerman!

I never really made that connection before (at least not consciously). I guess it is a nice fit then!


@virror

I think you could be right, but still would be much simpler if a company (or Oculus itself) could provide the hardware, as long as it was at a decent price.. that would allow the design to be a bit more sofisticated as well.