Dev V2 Cameras and positional tracking

[crobarpro]

now we've all seen the pictures of the two camera concept from oculus, I kind of want to start a discussion on how this could be implemented and what this means for the end experience. I've seen some suggestions on AR, and others that they would be just for positional tracking / quick Reality view. Can anyone say with experience how the two cameras could be used for positional tracking? Perhaps markers set up around the VR environment? Would this add additional latency to the experience? What are your guys speculations on the four glowing dots between the eyes? perhaps depth sensor? I've got so many questions! I need your speculative answers!

[phillypro]

im scared....i dont want the oculus rift to become a playstation move experience for your head

i like the idea of gaming in a chair...i hope the positional tracker is moreso just for being able to move your head in and out of certain depths

ex: being able to look over a map...by leaning forward

but i sure as hell dont want us going the route of the kinect...where we turn the computer room into a play space and are now limited by furniture

[lordvtp]

Those would be infa-red sources. You know, I've thought several times that the best way to get your hands into the game is not a hydra or Kinect per say, but a leap motion type device mounted to the front of the rift. I think this might just be what they are going for with these mockups.

[mattyeatsmatts]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

[Parallaxis]

Those would be infa-red sources. You know, I've thought several times that the best way to get your hands into the game is not a hydra or Kinect per say, but a leap motion type device mounted to the front of the rift. I think this might just be what they are going for with these mockups.

I think leap motion technology is highly unlikely for tracking your arms and hands. You need something that works where ever you place your hands and not just en a little area in front of you. It is much more likely to be used as kinect to track your whole body with cameras placed away from you.

Hand gestures is something that seems cool, but doesn't really work that well in reality. It's just a novelty.

[Parallaxis]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

Hydra has a precision of 1mm / 1 degree and that's not nearly good enough for head tracking. The magnetic field is also easily interrupted by other things and that will cause it unusable for head tracking. It's fine for hands though.

[mrklaw]

What is the likely refresh rate of cameras? I think the PS eye has a max of 120Hz at 320x240, would that be fast enough for positional tracking?

[Marcel]

Those would be infa-red sources.

If they are IR sensors then why does it say "Stereo Cameras" on the slide?

Also, if the slide says 1080P HD, I assume this means 1080P per eye. That would need a 3840 pixel screen!

Awesome sales predictions, I hope they make it!

[Butters]

If they are IR sensors then why does it say "Stereo Cameras" on the slide?

Because there are stereo cameras which you can clearly see in place of where the eyes would be. We're talking about the sensor in the middle which is unrelated.

Also, this is clearly 1080p for the whole screen, not per eye.

[Parallaxis]

Those would be infa-red sources.

If they are IR sensors then why does it say "Stereo Cameras" on the slide?

The IR emitters would be the 4 leds in the middle of the device. Who said sensors?!

Also - cameras are IR sensors. Try pointing your remote to your phone or video camera while shooting a video. Wow! It sees the infrared beams!

[Krenzo]

Hydra has a precision of 1mm / 1 degree and that's not nearly good enough for head tracking.

Why do you say that? I have a position tracker that doesn't have 1 mm precision, yet it seems to work ok for head tracking. You don't generally have things close to your face to even be able to tell if you've moved your head a few millimeters.

[Ben]

Can anyone say with experience how the two cameras could be used for positional tracking?

All sorts of ways. But you only actually need one.

The first approach uses trackable points on known objects, but it's not really suitable because you need an object visible to the camera at all times. That would be difficult without restricting movement considerably. Switching things round makes it easier, with tracked points on the user and the camera stationary in the environment, similar to TrackIR. This also means there's little to no added weight on your head. Similar problems persist though, and once the camera loses sight of the points it's useless.

Another approach would be to project a pattern onto the environment and then compare it from two different perspectives. This is how the Kinect works. It can do it with a single camera because the pattern is always unchanged when viewed from the point of projection, so the original pattern can be used instead of a second image. The downside with this approach is that to guarantee a clean(ish) image of the projection you need to use IR, and that means the image from the camera is kind of useless for anything else. That's why the Kinect has a second camera, to capture a human-readable color image. The other downside is that it doesn't work very well.

Small, light, cheap cameras which can produce noise-free images quickly, and a system that can process those images in real-time to determine your position so fast that any delay is imperceptible? Maybe one day, but not any time soon. I'm not aware of any positional tracking techniques where two cameras would help when placed so close together. So what are those cameras for?

Here's my theory: To look cool. That's it. It's a concept. A designer was likely given a picture of the prototype and told to make it look awesome, and that's exactly what they did. A pair of cameras on the front looks neat, and if you use your imagination it's easy to come up with all kinds of interesting things they could be used for. But making any of those things actually work? That's a lot harder.

And finally, I'll reiterate what I said in the other thread. This is taken from the Emerging Companies Summit at GTC, which is an event where CEOs from a load of startups are given 5-10 minutes to promote their companies to a panel of judges who then pick their favorites. It's Miss World for Silicon Valley, and a lot of airbrushing and plastic surgery is to be expected.

[Ben]

Why do you say that? I have a position tracker that doesn't have 1 mm precision, yet it seems to work ok for head tracking.

It's probably cleaning up the data a lot before applying it to the scene. Filters like that are great for removing jitter, but they also introduce a lot of latency and lose some of the genuine motion. That stuff will become much more apparent when using a HMD instead of a monitor.

[mahler]

You may be interested in these two Topics
PS3 Eye - PTAM/Visual Odometry
Computer Vision Headtracking Prototype

[nanicoar]

perhaps depth sensor? I've got so many questions! I need your speculative answers!

It's to track your hands and legs since people mention not having those is weird. That gets added into the wide category of things that cause nausea, and the industry sees those as deal-breakers.

[Skylead]

So what are those cameras for?

Here's my theory: To look cool. That's it. It's a concept. A designer was likely given a picture of the prototype and told to make it look awesome, and that's exactly what they did. A pair of cameras on the front looks neat, and if you use your imagination it's easy to come up with all kinds of interesting things they could be used for. But making any of those things actually work? That's a lot harder.

I'm fairly certain it's not "to look cool", but instead to allow for rudimentary augmented reality integration. You need two cameras to make the image stereoscopic so that's why it's not just one camera. They said they have been experimenting with using desk objects as controller interfaces like cubes and other shapes. The cameras would be a way to track the "cockpit" area around you and overlay visual controls onto such objects.

[crobarpro]

The "looking cool" theory kind of makes sense to me... but its no fun to talk about something like that. Do you think the team could be working on a positional tracker based on relative motion on previous frames? like tracking whatever ambient objects are in the background? that might be hard, especially if you had a wall of white in front of you with no distinguishable objects to track.

[juniordee]

This could be a good use for the dual cameras: http://youtu.be/R0-dsbeasgA

[PasticheDonkey]

The "looking cool" theory kind of makes sense to me... but its no fun to talk about something like that. Do you think the team could be working on a positional tracker based on relative motion on previous frames? like tracking whatever ambient objects are in the background? that might be hard, especially if you had a wall of white in front of you with no distinguishable objects to track.

i think it's possible with enough processing time or perhaps a dedicated chip. it's like digital stabilising of an image, but instead of stabilizing it you just take the movement information it works out from how the image changes as the position and angle info. fuse that info with your sensors. i don't know if that can be done in real time right now tho.

[Machinima]

The cameras could be for something as simple as allowing you to see in the real world without having to take the HMD off, so that for example you can see your keyboard or you cup of coffee at the press of a button, rather than taking you headphones off, taking the rift off then puting them both back on again.

[MSat]

There are a lot of uses for stereo cameras - some of them probably won't even be realized by the time it ships, but it would be stupid not to add them as there is significant potential (assuming that the sensors are capable enough in the first place) for various uses.

Other than that, I wouldn't read too much into what you see in those conceptual renders.

[Ben]

I'm fairly certain it's not "to look cool", but instead to allow for rudimentary augmented reality integration. You need two cameras to make the image stereoscopic so that's why it's not just one camera.

Augmented reality is severely limited by current technology. Capturing a detailed image, sending it to the computer, processing it, then sending it to the display takes far too long to maintain the illusion, at least with consumer grade hardware. Finding a camera which is small, light, cheap, and has a resolution and frame rate high enough for AR is, as far as I know, impossible.

They said they have been experimenting with using desk objects as controller interfaces like cubes and other shapes. The cameras would be a way to track the "cockpit" area around you and overlay visual controls onto such objects.

This is cool. You'd have to be looking at an object whenever you wanted to interact with it though. A single separate camera which didn't move with your head would make more sense, and if the image wasn't actually being displayed to the user then it wouldn't need to be anywhere near as high quality.

Do you think the team could be working on a positional tracker based on relative motion on previous frames? like tracking whatever ambient objects are in the background? that might be hard, especially if you had a wall of white in front of you with no distinguishable objects to track.

This is how most image-based tracking works at the moment, so I'm sure it's something they're playing with. As you said, there's no guarantee the environment will be suitable though. The only thing you know for sure is that the user will be wearing the HMD, so tracking that with an external camera is a better solution. I intend to try sticking some retro-reflective tape on my dev kit and use it with a TrackIR camera for basic positional tracking and additional drift correction. It may be that the fast rotational tracking does a good job of masking any delay with the positional tracking, but I'm not enormously confident about that.

I'm not actually trying to be all doom and gloom about this, I'm just going by what I know, which is that image-based tracking isn't yet possible at the same speed and precision of the motion tracker in the Rift. It wouldn't make sense to increase the weight and price for a feature which doesn't perform anywhere near as well as everything else. I'd love for the concept to be an accurate portrayal of the consumer release, revealing that image to be a photo rather than a render. I'd love it if someone at Oculus had made the cameras work as well as our imaginations want them to. And that's completely possible - the people there are far better at this than me.

If you'd told me a couple of years ago that I'd soon be excited about VR again I'd probably have given you overly verbose reasons for why that won't happen as well. As a confirmed pessimist, I actually love it when I'm wrong.

[Parallaxis]

I think you can be fairly certain that the stereo cameras is mainly for positional tracking since resolution and positional is the two most important areas they are investigating.

Sure they can be used for other things too, like orientation in the real world and maybe even AR - but that is definitely not the main reason they are there.

I don't believe in the use of external cameras at all. That too much hassle for any non-enthusiast.

[Ben]

I agree, external cameras are a pain, and I hope they don't go that route. I hope they don't go any image-based route really, because in my experience it's just too slow for this purpose. Using accelerometers and then correcting drift with a camera might be more viable.

But I still don't think the picture should be taken as representative of anything. The cameras make no sense. A video feed which matches the quality of the display would be too slow and expensive. A low quality video feed isn't worth it even if it's fast and cheap. If they're for tracking there only needs to be one and the front is the worst place for it to be from a balance point of view. It's just not a practical design. It's a cool picture to impress people and get them talking. It does that really well.

[Bmitch1012]

It says supporting the Oculus Rift and f/riftvr...Hmmm

[PasticheDonkey]

that's their facebook address.

[squibbfire]

im scared....i dont want the oculus rift to become a playstation move experience for your head

i like the idea of gaming in a chair...i hope the positional tracker is moreso just for being able to move your head in and out of certain depths

ex: being able to look over a map...by leaning forward

but i sure as hell dont want us going the route of the kinect...where we turn the computer room into a play space and are now limited by furniture

EXACTLY! I really hope they keep it simple...

[PasticheDonkey]

positional tracking for the rift or move or wii controls don't mean getting out of your chair. in the rift's case it just means more true to life immersion.

[mattyeatsmatts]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

Hydra has a precision of 1mm / 1 degree and that's not nearly good enough for head tracking. The magnetic field is also easily interrupted by other things and that will cause it unusable for head tracking. It's fine for hands though.

fair enough I guess, ill just play sitting on a wii balance board

[Kazioo]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

Carmack did it a year ago and it wasn't good enough. Hydra is precise but not accurate and it has 8 times bigger latency than Rift's tracker (effectively: 62.5hz vs 500hz).

[mattyeatsmatts]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

Carmack did it a year ago and it wasn't good enough. Hydra is precise but not accurate and it has 8 times bigger latency than Rift's tracker (effectively: 62.5hz vs 500hz).

Thanks for that, very informative.

[Butters]

ill just play sitting on a wii balance board

That's actually not a bad idea
Not necesarily for position tracking, but for movement control : sit in the middle of the board and shift your weight to control movement of your avatar (instead of WASD or a joystick). Lean forward to walk, backward to step back. You'd probably need a swiveling chair for 360 orientation. Coupled with head positional tracking in the headset, I think this may feel quite natural, making the head position in game more consistent with the avatar movement and that movement more consistent with some sensory feedback for the player (possibly mitigating motion sickness ? Wouldn't your body accept perceived acceleration better if your inner ear was detecting a proportional quantity from your body lean input ?). You're going to need a chair with a back rest - I can see people falling over while trying to run away from things in a horror game ! (it actually would be very cool to have your character take a step back when you have an instinctive surprise reaction - monster jumping at you from the shadows or something)

A problem would be not to confuse actual lean (looking around a corner in a FPS) with movement lean (strafing left-right). But I guess you can learn to move your head to the side without shifting your weight either way. That would force you to keep only your head (and gun, presumably) peeking our of cover, adding to immersion ?
The other problem is of course that this makes it impossible to play slouched back in your chair or sofa, so probably limited to more involved experiences.

This type of control scheme would have no interest over a regular joystick when you have access to one (be it on a Hydra or a guntroller or anything else), but could be interesting with a "hands free", Motion Leap style control scheme. I can see myself preferring that over a foot controller (I guess that makes it an "ass controller" ? ).

[Drewbdoo]

Couldnt Oculus hack a razer hydra position sensor from the hand controller of the hydra and put it into the rift headset so it tracks your head, the cable could just be incorporated with the rifts cabling.

Carmack did it a year ago and it wasn't good enough. Hydra is precise but not accurate and it has 8 times bigger latency than Rift's tracker (effectively: 62.5hz vs 500hz).

Happen to notice around 2:09, his solution for positional tracking is 2 cameras on the front...

[nateight]

wii balance board..."ass controller"

This is absurd, ridiculous, preposterous - exactly the sort of wackiness that deserves a little testing. I don't think for a second it's going to work (on top of being a damned uncomfortable seat, the Balance Board is really designed for the weight distribution of feet), but I'm adding it to my list of things to experiment with now that I have one ($30 USD at Gamestop!).

As for the actual topic (Christ, how did we get from cameras to...), I'm entirely with Ben: Visual odometry is messy, slow, and frequently wrong. Maybe Oculus will blow the doors off the concept with some amazing new algorithms and techniques, and while they're at it also invent a time machine capable of killing Hitler without causing an alternate timeline worse than the Holocaust, etc. I'd be utterly thrilled to be surprised on this one, but please forgive me if I don't cheer this effort on too vigorously - I know a "will be ready for mass production in 5-10 years" tech when I see one. It is of course possible that we really are looking at the consumer Rift here, and that those really are two cameras 2.5" apart, but that people expecting the Rift 1.0 to be a combination VR/AR device are still going to be disappointed. Infrared marker-based tracking is a very good solution for calculating spacial translation (good enough for the Wii, remember, good enough for TrackIR, good enough for FreeTrack, and sharing some similarities with the Kinect and the PS Move, and that's all of the trackers). If Oculus wants to (prudently) avoid having users place cameras around their Rift-spaces, a marker-based approach could move the cameras into the HMD and allow users to instead place IR markers. Ben, one camera in the Kinect is enough for the Kinect because it is stationary - isn't the whole point of this exercise that the capture point does move? Two cameras and one multi-LED marker beacon would be enough for most seated movement, four or five beacons should be able to cover 360º spinning. If the POSITTRON system doesn't win the positional head tracking race, it's probably only because Oculus saw the sense in flipping it around; the underlying tech is too suitable a solution to be ignored.

Drewbdoo, both you and Carmack are completely right: Stereoscopic tracking is going to win...eventually. The very next thing Carmack says in that keynote is "we need a few more cranks of Moore's Law", and we do. If you think AR is happening right now, educate yourself. Mediated reality is way, way harder to do than virtual reality. The Rift 3.0 will use something like amazing 4k 480FPS cameras feeding real-time, 3D pictures to an even better display; the tech simply won't be there in time for the Rift 1.0.

[Moggle69]

The 2 cameras are for x-ray vision and to shoot laser beams from them. LOL

In all seriousness, its obviously for positional tracking and not AR. I LOVE what Oculus is doing here! We are going to have one kickass HMD soon.

[brantlew]

Carmack did it a year ago and it wasn't good enough. Hydra is precise but not accurate and it has 8 times bigger latency than Rift's tracker (effectively: 62.5hz vs 500hz).

That's not all that he said though. If you listen closely he says that the right thing to do is sensor fusion of the inertial orientation tracker and magnetic position tracker. Something that he admitted he didn't spend enough time to get right.

[Ben]

Happen to notice around 2:09, his solution for positional tracking is 2 cameras on the front...

And around 2:10 he says the hardware doesn't exist and the software problem has remained unsolved for decades. If the people at Oculus have managed to get it to the point where it's good enough for VR then that's awesome. I'll have a big smile on my face while I eat my virtual hat. But I'll wait for details before getting excited.

[nateight]

Here's an additional free one because I'm already seeing a patent attorney for something else this week: If you agree with the marker-based approach but hate the idea of external markers for seated Rifting, you're definitely going to be right next to a computer and a boring 2D monitor, right? Turn the control box into a pass-through box (neatly obviating the need for an additional DVI/HDMI switch) and have the formerly bypassed monitor display a visual marker the cameras in your Rift can see.

the right thing to do is sensor fusion of the inertial orientation tracker and magnetic position tracker.

Hey wait a minute, aren't you the visual odometry guy?! Mr. Today I Took Two PS Move Cameras Apart And? You're supposed to be inventing some magical stereoscopic hoozitz the world has never seen before, not fiddling with magnetometers! Combination VR/AR is forsaken!

[brantlew]

Don't read too much into it - I'm just quoting Carmack. The work I do at Oculus is actually a lot more pedestrian than you might think. There are specialists that do a lot of the heavy-lifting.

[nateight]

Don't read too much into it

This is the Internet, sir, the Internet. And just what do you mean by "quoting Carmack"? Is he working for Oculus now?? Or - buying the company? And then selling it to Microsoft to fund his rockets?! OH GOD.

GUYS COME QUICK - I THINK BRANTLEW JUST TOLD US MICROSOFT IS KILLING THE RIFT