Oculus SDK - Ideas and Feature Requests

[Fredz]

The performance of any of these homebrew solutions is annihilated by the professional solutions available.

Instead of marketing blurb, what about some real numbers ? Precision, FOV, range, frequency, latency, number of tracked features, axis of freedom ?

$1899 also is nearing consumer price range. We dont hesitate to pay $2k for a television in our living rooms and this VR build is barely more.

I don't anyone who did pay $2K for a TV, and only a company would spend the same amount for a head tracking system. People must be quite rich where you live...

[cybereality]

Well I don't doubt that a system like that would be worth the money for an academic or commercial entity needing it for business. For example, gaming studios pay *way* more than that for their mo-cap setups. But $2K is a little much for a hobbyist. I mean, its not like I couldn't find the money if I had too, I've spent probably twice that on my computer over the years. But its just a little much for just a tracker. Not when you can buy 3-9DOF orientation trackers for around $100-200 and they are good enough for most purposes.

There are also methods for positional tracking, like brantlew mentions the Wiimote which you can get for like $40. It works, and its accurate. Just look at what Johnny Lee did. There are also things you can do with the PS Eye or the Kinect. Its not like the Wiimote is the only option. Not to mention FreeTrack. Can't imagine why any hobbyist or gamer would pay that kind of dough on a tracker.

I mean, you can build an entire 3D movie theater for less than that. I did. Got a 720P 3D projector for $600, and then spent another $800 to build a HTPC w/ Blu-Ray and everything. So for $1,400 all inclusive I've got a 100" 3D theater in my living room. Now THAT is worth spending money on and still cheaper than that tracker.

[Gunshot]

Video games represent years of a team of developers work, and they sell for $60...

[vvortex3]

I have tried allmost all of the freeware and cheap solutions mentioned and none of them are good enough for 6dof head tracking. The latency requirement for head tracking to avoid motion sickness is quite low. I honestly think $1899 is a bargain. VR isn't cheap people. Well, the HMD and input devices have become cheap but the 6dof trackers simply are not cheap yet and there is no cheap solution that is even similar. Experience: I have tried most of them and also owned an optitrack system. I'm open to try other things but I have not seen or heard of anything else yet which is comparable. There is a huge gap between "it works" and "it's accurate".

Edit: Maybe these will help convey the difference between hacking a raw stream from a wiimote and using something like tracking tools:
http://www.naturalpoint.com/optitrack/p ... rials.html

Please note that I'm not affiliated with naturalpoint in any way. I am just very satisfied with their system.

[Pyry]

Honestly the only expensive part of this system is the camera(s). The software side of things is just the well-studied three point perspective pose estimation problem with a Kalman filter on top. I mean, yeah, you could spend endless engineer hours to produce a low-latency, low-jitter, high-frequency pose estimation by unloading the processing onto FPGAs and whatnot, but it won't matter that you're calculating the pose at 2000hz on a hard-real-time embedded system if that pose then has to go into a desktop PC where it will be read into a game engine that renders at 60hz (most of the time with no guarantees).

[divide]

I agree $2K is way too expensive for just a hobbyist tracker setup, I would never go above $200 for tracking.
For instance on the hardware side, you can have 2 synchronised PS3 Eye @187fps for $40...

[vvortex3]

Honestly the only expensive part of this system is the camera(s). The software side of things is just the well-studied three point perspective pose estimation problem with a Kalman filter on top. I mean, yeah, you could spend endless engineer hours to produce a low-latency, low-jitter, high-frequency pose estimation by unloading the processing onto FPGAs and whatnot, but it won't matter that you're calculating the pose at 2000hz on a hard-real-time embedded system if that pose then has to go into a desktop PC where it will be read into a game engine that renders at 60hz (most of the time with no guarantees).

Which software can I use that does this? I'm only aware of Freetrack, TrackIR, and Naturalpoint's solutions (staying within things that are "affordable"). Are there others? I'd be open to trying them. I have used freetrack with multiple cameras /etc and was disappointed with it. It is jittery and loses tracking easily. That was my experience anyhow. I don't want to spend $2k as much as anyone else but I have not yet seen or heard of any cheaper and comparably useable systems that I don't have to brew myself.

And to the wiimote users, what is the positional accuracy of the wiimotes tracking?

[crespo80]

The oculus team is probably in contact with many leading tracking companies like optitrack, and if these companies believe in the success of the project they can provide a way cheaper custom solution for the consumer rift so it can be sold at an attractive sub 1000$ price point (I mean the complete package, the HMD with 6DOF tracking, a camera system for body tracking, some kind of controller for the lower body and a hydra-like standard controller).

[vvortex3]

If that happened I would be very happy

[kalabalik]

...I don't want to spend $2k as much as anyone else but I have not yet seen or heard of any cheaper and comparably useable systems that I don't have to brew myself.

Well that's where you're obviously wrong as the vast majority of people are so opposed to the idea of chugging 2 thousand dollars on just tracking that they're simply not buying it, when a much much cheaper system can manage OK compared to the ocean of money between the setups.

So if we're ever going to see decent tracking being widely used we need to have it at a price point of 100-200 dollars tops. Not thousands.

[Fredz]

And to the wiimote users, what is the positional accuracy of the wiimotes tracking?

Okay, some hard numbers now since you didn't provide any :

http://www.pitt.edu/~nak54/Wiimote-poster-a4.pdf

System : WiiMoCap
Material : 2 Wiimotes, USB Bluetooth dongle, IR LED, Power supply, 2 Camera brackets
Cost : $160
Acquisition rate : 98Hz
Latency : 49.6ms
Accuracy : 1mm

Their own conclusion : the WiiMoCap system performance is on par with Phasespace motion capture system.

[brantlew]

@Fredz: Nice find.

[vvortex3]

Fredz: could you please link to the source code? Because there are people out there doing accurate tracking with a single camera and mountains of extremely academic and closed source. There is a big difference between possible and available. See DSLAM: http://www.youtube.com/watch?feature=pl ... ZiSK7OMANw. (I have also compiled and tested PTAM).

It is neat that it can be done with a wiimote, it can also be done with normal cameras, my point is that it is not the same quality without lots of software effort. Somebody has to do the same work as these academic/private organizations and then make it cheaply available.

[bobv5]

vvortex3,

I don't think anybody is saying that it is not a nice piece of kit. But it costs far too much. The reason Rift is so poular is because people can afford it. If you have a lot of money decent VR has been available for many years. If you don't have much cash, you have to hack some things, build some things, and tolerate the faults. Why is it that you do not like people thinking 2k$+ is too much?

[FingerFlinger]

Also note that motion capture is VERY different from DSLAM. The WiiMoCap implementation is basically just blob tracking and some trigonometry. The family of optical flow/density algorithms have a completely different purpose and are far more complex.

[vvortex3]

It's not that I don't like anyone at all. I am just trying to share my personal experience with these systems. Maybe it will help those who have not played with these systems personally understand what 6DOF tracking is, why it's important, and avoid being misled. I also very much root for any alternatives and would gladly try anything that I can get my hands on for comparison. For me, personally, I dont want to deal with constant jitter and loss of tracking and I want to help those who have not personally tried these systems understand that there are options out there now which do not have these problems (even though they are still costly).

Tracking systems have other markets than VR so anyone that puts in the effort to make a good system is going to realize that it can be sold for motion capture, video, robotics, weapon systems, etc. Before they are even listening to those who are unwilling to pay more than $1-200. We need a good open source project that is meant to compare with these systems. An open source version of tracking tools would be awesome and I would even help (although my math skills are not as good as needed).

I like Johnny Lee's system with the wiimote but please understand that you cannot turn your head away from the wii sensor and thus can't really turn your head eliminating alot of immersion. Maybe the solution here is for us to start or support an open source project for multi-camera infrared tracking of rigid bodies.

[brantlew]

Maybe the solution here is for us to start or support an open source project for multi-camera infrared tracking of rigid bodies.

An multi-camera extension to FreeTrack is sorely needed and might be a good place to start.

[vvortex3]

Here is an academic project which calibrated multiple wiimotes: http://web.mst.edu/~vram/proj-aasumc.html Maybe we should hunt down each of such projects and harass them for source code.

[EdZ]

I'm waiting on the camera module for the RaspberryPi. Add an IR LED ring, and you have a Vicon-camera-a-like for under £100, with on-board blob-discrimination and point coordinate output. Slap a bunch around a space, hook them all up via ethernet, and have a central machine just take the coords from each and use OpenCV or the like to provide you with your point-cloud.

[Fredz]

Fredz: could you please link to the source code? Because there are people out there doing accurate tracking with a single camera and mountains of extremely academic and closed source.

Here : http://labviewhacker.com/wiiremote.php# ... %20library

You are not a programmer are you ? Because tracking blobs like it's done in this project has nothing to do with mountains of academic research. Even a beginner programmer could handle that quite quickly.

There is a big difference between possible and available. See DSLAM: (...) (I have also compiled and tested PTAM).

As someone else said, that's not the same thing at all. Tracking blobs with an IR camera is a piece of cake, tracking moving features in an unknown environment is an entire different beast. And it's not that much useful for head tracking because of the lack of precision, lag and difficulty to track fast movements.

It is neat that it can be done with a wiimote, it can also be done with normal cameras

Sure, I just wanted to point out a project that does work and with factual numbers. Using a PS3 Eye with IR leds inside ping-pong balls would be even better, less lag (22ms vs 50ms), more precision (knowing the diameter of the balls), fast frequency (120Hz vs 100Hz) and better FOV (75° vs 33°).

my point is that it is not the same quality without lots of software effort. Somebody has to do the same work as these academic/private organizations and then make it cheaply available.

Software like this is cheap and hardware shouldn't be that expensive either. These people sell a service, that's why you can't find prices on their website. It's dependant on how much you are prepared to pay for that, by calculating how much you'll win in the end. Clearly not for home users.

That's the exact same situation with the Rift compared to the Sensics zSight although the specifications are not that much different. The first one cost $300 and the other $11,995 or $23,980, depending on where you look on their website. This shows clearly that the price they sell their product has no correlation to its manufacturing cost.

[zalo]

tracking blobs like it's done in this project has nothing to do with mountains of academic research. Even a beginner programmer could handle that quite quickly.

I can confirm as a beginner programmer that Blob detection is not a problem at all.

The two bigger problems are extrapolating position from a set of points, and integrating it with the software.

Problem number one can be solved for free by anyone with a higher level math education and familiarity with a Position from 3 Points Algorithm. Chriky did this in the VR/AR Research and Development subforum.

Problem number two, on the other hand, requires participation on both the VR developer's part AND the game developer's part. Hopefully, the Rift SDK will solve this problem by opening up an API that we can use to pipe in custom rotational/translational data to any software that has integrated it.

I know I can do full 360 6dof tracking (getting everything but pitch and roll optically, and by using an accelerometer/gyro combo to fill in the gaps) with a single PS3 eye camera placed above the user's head, and no more than 2 or 3 led tracking markers mounted to the top of the HMD. The problem here is that you won't be able to move very far from your starting position (unless you have a tall ceiling), limiting this application to standing-still VR.

But at least we'll be able to do it with the Rift SDK (I hope!). Then we can judge the quality of DIY VR Tracking setups

[vvortex3]

I'm not sure if you have been reading my posts. I'm referring to multi-camera solutions. Also, please don't bother with personal attacks because I will not read nor respond to them and quite frankly it's a waste of everyone's time. I have no idea why there are so many trolls on here. Fredz, I am not interested in arguing with you over what I know to be true.

To those of you who have been contributing in a helpful and useful manner to the conversation I thank you.

What we need is an open-source multi-camera infrared tracking software package that can use things like the wiimote or any camera that can be modified to view infrared. This will involve things like calibrating the cameras, defining rigid bodes, correcting for error, smoothing jitter, etc. Basically all of the things that are not really accomplished by what is cheaply available now.

[crespo80]

Looking to a future consumer version with integrated body motion capture, with an all-in-one input peripheral that's easy to set-up and reasonably cheap compact and safe to be used while standing up, what about the idea of controlling the forward/backward/strafe movements with a single pedal to be attached to the ground and controlled with one foot, just like my previuous idea of the round board to be used while sitting?
The user attaches this pedal to the ground with some suction cups (or some combs if he uses a carpet) so he has always a real clue of his position inside the room, and this pedal foot can rotate over three axis as well, the pitch controls the forward/backward and the roll controls the side strafe, the yaw controls the body turn.
So you turn your body in any direction by pivoting around this pedal with the other free foot, and control your virtual legs by rotating the pedal.
It's very precise just like a keyboard/controller, very intuitive and you don't have to dangerously walk on spot.
The second thought is that you need a tracking camera system and a "safety area" free of obstacles in case you fall down, let's say a 6 feet diameter circle.
We can achieve both the goals by placing three cameras on the floor that have to be attached to the pedal foot 3 feet away at the same distance from each other and point to the user, so the user is forced to find a large enough spot where to play. Or the cameras and the foot pedal can all be mounted over a single large board, but the concept is the same.
These cameras can precisely triangulate the head position (by placing some markers on the rift) and the whole body as well, maybe putting another marker on the chest to precisely track when the user crouches/jump, and a pair of coloured and inexpensive gloves to precisely track even the fingers. Two added cameras can be mounted on the rift itself for augmented reality use or to integrate with the other three to calculate hands position. Then, if you like, a standard hydra controller as an added input controller.

[FingerFlinger]

I agree vvortex3, that would be a really good open-source project. A complete mo-cap suite would also enable many smaller indie developers to incorporate it into their workflow. That said, it is a big investment of time. I will probably roll my own solution sometime this winter, and I suspect that's why there isn't a full-featured OS project out there already: people do it on their own and don't take the initiative to clean up the code and release it.

[brantlew]

It's a good project idea and would probably be the most wide reaching solution for the average gamer if it could be made to work - especially with Wiimotes.

[PalmerTech]

I want to give some more detailed feedback, but a short note: vvortex3 is right. There is nothing out there that can come close to matching the performance of these professional systems, at least not right now. That Wiimote project matched Phasespace, sure, but comparing newly released tech to an older professional system in a 2D workspace environment is not fair. The current Phasespace systems have a fifth of the latency, and are incredibly robust; They cost $25,000, sure, but they are so freaking badass.

That said... With VR taking off like it is, there is no reason that these systems need to cost so much anymore Most of the cost is in R&D and software, not the actual cameras! If there are tens of thousands of people buying tracking systems instead of tens of people, then a lot of savings can be passed onto the consumer. Oculus is having some very cool discussions, and I can assure you that the consumer Rift is going to have amazing tracking.

[MrGreen]

Oculus is having some very cool discussions, and I can assure you that the consumer Rift is going to have amazing tracking.

[brantlew]

A post in another thread got me thinking about web-cams. Not necessarily for AR, but just a simple hazard/navigation camera so you could see the outside world without taking the HMD off. That seems like a basic component that Oculus should include with the consumer-Rift for usability. I wonder what kind of SDK support would be necessary for this. A simple model that I can imagine would be to provide a button on the Rift that activated the camera and displayed the outside world. So you might need some type of signal from the SDK to tell the game to pause while the camera was being used. A more complex interaction might be to provide the video feed directly to the game via the SDK, so the game could pop up a video HUD within the game for a more seamless integration. And third would be a stereoscopic video feed that a game could use for augmented reality games. But this might require a good deal more integration that just a simple video feed.

[MrGreen]

A post in another thread got me thinking about web-cams. Not necessarily for AR, but just a simple hazard/navigation camera so you could see the outside world without taking the HMD off.

Michael Abrash mentioned that it was an absolute must for a retail product on his blog and it surprised me a little. I mean, is removing goggles that hard? This goes in the "nice to have" category as far as I'm concerned but it sure would be a useful feature nonetheless.

[brantlew]

If it was really hard or expensive to do, then I would say it's not worth it. But because it's so simple and cheap and it is so EXTREMELY convenient to have I would say it's a must-have. Once you've got yourself all strapped in, the optics adjusted, the headphones situated and your body in position - it becomes really intrusive to then have to undo all of that just because you forgot to hit some key-combination, or forgot to plug in your headphones, or misplaced your controller, etc...

[MrGreen]

Yeah I guess there are more occasions where you'd have to remove it than I thought at first. And as an added bonus you get an AR ready HMD.

[rhinosix]

A camera seems like something that wouldn't be too difficult or expensive to add, and would come with a lot of benefits for the consumer version.

One thing I'm thinking about is transitions. A big component of slick interfaces like iOS are the fine detailed transition and animation effects.
You don't want to tap a button, and have it coldly open the next window. You want to see the downstate of the button, and the smooth transition to the next screen without doing anything too extraneous.

For VR, you may not want to strap the device on, and have the VR world just... there. Or have the game just open. You want to feel like you're stepping through a portal into another world.

With a front-facing camera you would be able to transition between worlds. You could have it default to camera-view as the natural state. When you're ready to step into the game, you press a button, and it transitions you - it could be like an electronic visor coming down over your eyes; or just a smooth fade to black. If something happens - like if the game crashes, or there's something that needs attention in the outside world, you don't want the player seeing a blue screen of death and having to rip the thing off their face. Just a nice fade to black, and the outside world comes into view. Everything is ok.

Might not be a big deal to enthusiasts, but it could add some benefits to everyday consumers.

One thing I was thinking about today is bootup screens like the Dreamcast, the original PlayStation. And also stuff like the Dolby, Paramount, Universal, and other intros they show at the start of films to show off their audio and visual effects. When I see one of those Dolby intros I'm like "screw yeah, I'm about to have a good time."

It might be fun to have a cool short intro when the player boots up for the first time. You could show the real world, do a quick transition through clouds, stars, or the oculus of a building - and show the logo before the game comes in to focus.

[Pyry]

Most of the cost is in R&D and software, not the actual cameras! If there are tens of thousands of people buying tracking systems instead of tens of people, then a lot of savings can be passed onto the consumer.

I don't know; if you want >60fps, global shutter machine vision cameras, you're looking at paying $400+ per camera, and these cameras are already sold in the (tens?) of thousands. You're not going to be able to get the same performance out of a $40 rolling shutter webcam (but maybe in this case you don't actually need that level of performance).

[EdZ]

Most of the cost is in R&D and software, not the actual cameras! If there are tens of thousands of people buying tracking systems instead of tens of people, then a lot of savings can be passed onto the consumer.

I don't know; if you want >60fps, global shutter machine vision cameras, you're looking at paying $400+ per camera, and these cameras are already sold in the (tens?) of thousands. You're not going to be able to get the same performance out of a $40 rolling shutter webcam (but maybe in this case you don't actually need that level of performance).

This is exactly the problem a group I was in at Uni faced when trying to build a 'Vicon-on-the-cheap' system. Even if you hack some above-average (SPC900NC) webcams to share sync, they just don't cut it for machine vision when spread around a room. The PS3eye was a nicer (well, higher framerate at lower resolution) camera available, but with the uncompressed USB interface we'd have ended up with pretty much having a PC running every camera, and that still wouldn't have guaranteed sync.
The RaspberryPi with (yet to be released) camera module might be a more viable option. The camera will connect directly, you can do any sort of distortion correction and blob tracking on-board at whatever framerate you can achieve with the camera hardware, and just stream coords over ethernet to a central server.

[MrGreen]

There has to be a way to make a good tracking system without having to resort to cameras...

Just don't count on me to find it.

Surely what's used in the Razer Hydra can be improved upon right?

[vvortex3]

I've been digging around a bit.

I'm thinking the best cheapest solution available may be an overhead mounted TrackIR 5 pro + tracking clip + sdk for head tracking combined with razer hydra for input.

The approach is discussed here: http://www.mtbs3d.com/phpbb/viewtopic.php?f=140&t=15399

The problem with this approach is that, as far as I can tell, Naturalpoint's site claims that they do not release the TrackIR SDK unless it is for commercial game development. I have put in a request for a copy of the SDK for evaluation purposes and we'll see. I have tried Freetrack, but not TrackIR and I'm hearing that TrackIR is actually much higher quality.

The neat thing here is that both TrackIR and the Hydra have plugins for Unity 3D. Here is the TrackIR plugin:
https://github.com/byBrick/Unity-TrackIR-Plugin

I am planning to purchase and test this solution. The total cost is $150 for the TrackIR Pro, and $110 for a Hydra.

[MrGreen]

TrackIR limits you to 180° or so doesn't it?

[brantlew]

I would say less than 180. The advantage of overhead is that you can preserve 360 degree yaw angles. The other directions are limited a bit. Roll is not a problem since you rarely roll your head 90 degrees, but pitch will be limited. If you try to look directly up or down TrackIR will lose you. I think a nice addition to this setup would be a simple 6DOF gyroscopic tracker (ie. Hillcrest, Wiimote, etc). The inertial tracker could compensate for those extreme angles and the system could transfer to inertial tracking when the TrackIR lost it's view.

[MrGreen]

Having to mount something to the ceiling would be a deal breaker for a lot [most] people in my opinion.

What's the technical wall that would prevent to use Razer Hydra's tech for head tracking?

[brantlew]

I think the "wired" thing really holds back the Razor Hydra from being a general purpose tracker. If Sixense had gone with their original vision and offered wireless controllers it would be twice as useful as it is now. My far-fetched hope is that Oculus can partner with Sixense to license their motion sensors and package it as a wireless solution. A three-point configuration (1 built-in magnetic head tracker + 2 hand controllers) would offer a general solution for most scenarios and provide a standard control platform for game companies to build upon.


Edit: To simplify the design you could wire the hand controllers to the head set. But it would be really amazing if they had wireless transmission between the headset and the base station so you could achieve an untethered setup.

Another idea....right now the Hydra base station is the brains of the Hydra but I wonder if you could split the base station so that the computational logic was enclosed in one module and the field emitter was a separate passive device. Then the Hydra system could still be wired (and worn) with just the transmitter separated and acting as a fixed reference point. No need for wireless communication then. It seems like a best-of-both-worlds solution.