HMDs, dynamic focus, and distance cues

[xef6]

Hi, I'm new to the forums, so I'm not familiar with any of the parlance/culture!

A common problem I read about is headaches resulting from invalid/unnatural depth cues in two categories: 1) vergence and 2) accommodation. I never see/read/hear about HMDs dynamically changing the focus of light coming into the eye, but I do know that many have adjustment wheels for manual/one-time changes per person (far/nearsightedness). My question to anyone familiar with HMDs is: why not connect that to a servo? Has anyone heard of this being done before?

The reason I ask is because when you try to focus on things that are close (with an HMD), your eyes don't experience accommodative kinesthesia (http://en.wikipedia.org/wiki/Depth_perception). Wouldn't it be possible to change this dynamically, depending on what the user is looking at? I guess this would require eye tracking as well to know what the user is looking at, but shouldn't this work in theory? I've worked with eyetracking before, and it seems like this should totally work.

Just trying to think out loud

(I guess this might belong in another part of the forum, but the reason I put it here is that the Oculus might have such an adjustment wheel, and computer control of the focus seems like it would be very useful anyways, such as having personal profiles for your focus preferences.)

[PalmerTech]

If the servo is fast and accurate enough, and the eye tracking is also fast and accurate enough, then yes.

Doing so is a huge engineering effort, though, especially in a package that remains lightweight while still allowing a large field of view.

[2EyeGuy]

That's an interesting idea. Probably too technically hard for the Rift, but it might be possible in one of the high-end systems that cost tens of thousands of dollars, if the Rift doesn't make those systems obsolete. It's basically just the opposite of the autofocus in a camera. You'd have to combine it with a depth of field effect in the software, that changes to match which depth is in focus.

I see a couple of problems though: it might cause more headaches than it prevents, since things would be going in and out of focus in ways that don't exactly match what your eyes are doing. And it might be hard to do eye tracking in a dark space when your eyes are covered by a screen in almost every direction due to the high FOV of the Rift.

I like the idea of a device that would automatically change its focus to match my vision requirements when I log in. But I think that might have to wait for a later generation of HMD. You could even combine that with iris or retina recognition, so you would just have to put it on and it would know who you were. I don't know if you can recognise people from their tracking data, that would be cool. I was a bit surprised Wii Fit didn't do that with the balance board and recognise people by their weight or posture.

[xef6]

Yeah, sorry if I made it sound like I was expecting a feature to magically appear in an already amazingly high-value product. I'm curious what others think about it in context. I imagine it would be quite difficult to make wearable hardware that does something like this.

And it might be hard to do eye tracking in a dark space when your eyes are covered by a screen in almost every direction due to the high FOV of the Rift.

Usually, eyetracking is done in infrared, so that lighting is less of an issue. The iris/pupil appear in much higher contrast that way too. FOV is also another issue, dunno about that. Thanks for the response, more than anything I'm curious about what other people think about this, because HMDs are still pretty niche.

[faker]

I think that a custom implementation of a leap motion device would make an excellent eye tracker that could double to configure software and hardware automatically for every new or recognised user as they put it on. Of course such a partnership must also include a larger front mounted sensor array to provide hand/arm/object tracking and locking onto the surrounding environment for full positional tracking.

Here's hoping this makes it into the consumer Rift, I know I will be strapping a leap on the front of my Rift dev kit as soon as I have them both!

[Mart]

With eye tracking you could also solve another problem - rendering performance.

The key would be taking advantage of the fact that the human eye only sees a very small part of the image in high resolution.

The eye position would be sent to the host machine, which the game engine could use to render reduced resolution relative to the eye position. The center of our eyes would see the normal 1:1 image, but our peripheral vision would see reduced resolution. (1x1,2x2,4x4...)



In theory if the eye tracking was fast enough, and the implementation matched the relative resolution of our peripheral vision perfectly, you wouldn't be able to notice a difference in image quality.

It would be a great feature to compliment a 4K/8K screen...

[brantlew]

Is this what you had in mind?

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15154

[xef6]

Is this what you had in mind?

http://www.mtbs3d.com/phpBB/viewtopic.php?f=120&t=15154

brantlew, that's exactly what I was talking about! Thanks for posting that paper about it. I like how he ends the component list with "magic". With that many moving parts/variables, I'd be impressed if it was responsive/precise enough to be helpful. I wonder if he'll ever write about the voice coil version. It is pretty skimpy on details (like response time of system/eye), but the overview style is pretty useful. Nice find.

I guess this exists, but just barely (May 2012). It's really encouraging to know that someone has actually made a functioning prototype. Well, I assume with magic he should be set.

Looking around, there's an NCBI article listing that looks at how fast the accommodation reflex acts. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3018118/

[zalo]

I still foresee that eyetracking will be a big issue. Your eye is nearly touching the Rift's lens so there's no room pre-lens to get an image of the pupil.
Then, ideally, your entire viewing frustum (with eye movement) is taken up by image, so there's no room for the camera to exist behind the lens (since it should be all screen)

That is, unless you want to hide a camera behind a transparent LCD... but there are many problems with that too.

Also, when you are looking at an object up close, you can voluntarily control whether to focus on the up-close object or the background (even though you are technically looking at the same place). Just an eye-tracked position isn't even enough to determine the "proper" focus distance.

So now you'll need to measure the curvature of the lens just to see what he's trying to focus on, you'll need to add computer generated dof, AND you'll need to have dynamic focusing and eye tracking. I don't see how one will fit all of this together in a light package unless one shatters the electrooculographic code so one can use some non-invasive electrodes on the surface of the skin to measure the potentials of the eye movement and focusing muscles.

And while we're at it adding electrodes, let's just stick some GVS on there for good measure.

[android78]

I think the only way to do eye tracking with the rift would be with a semi-silvered mirror at 40 degrees between the screen and lens, and tracking camera at the bottom of the case. I'm not sure how low reflective could work, but maybe with a 10% reflective (90% transparent) could be ok and not darken the image too much.

[EdZ]

I think the only way to do eye tracking with the rift would be with a semi-silvered mirror at 40 degrees between the screen and lens, and tracking camera at the bottom of the case. I'm not sure how low reflective could work, but maybe with a 10% reflective (90% transparent) could be ok and not darken the image too much.

As most eye-tracking cameras operate in the near Infra Red region, a dichroic mirror of the correct wavelength to reflect only the nIR wavelength the camera senses (commonly known as a 'hot mirror') would introduce the minimum of distortion into the optical path.

[alexpez]

As a layman I'm thinking that the latency involved in reading the eye position and then adjusting the lens could cause more issues than it solves, but I'd love to see the result! If the latency issue could be sorted, then the accommodation issue disappears, but everything on screen remains in focus the whole time (no defocus blur?).

Looking ahead, would there be any mileage in adaptive micro lens arrays? If you built an adaptive micro lens that could cover 1 pixel on a screen, and then built an array to cover the screen, and if they were independently controllable, would you be able to assign correct focal depths to objects on the screen allowing for whatever you looked at to appear in focus and everything else out of focus?

[zalo]

I really like the concept of hot mirrors. They look look perfect for eye tracking:

Hot Mirror Transmissiveness over Spectrum


IR LED Peak Spectrum

They fit comfortably within each other's spectrums so an eye tracking camera could be easily placed on the top of the HMD between the lens and the screen (above the screen) with the hot mirror doing a 45 degree reflecting thing. That is, if there is enough space and it doesn't weigh too much.

[xef6]

I really like the concept of hot mirrors. They look look perfect for eye tracking

That definitely looks perfect spectrum-wise. After actually thinking about high FOV vs. camera placement, I realized that I have no idea how you'd pack everything into the same space. It seems like the only good spot left would be near the nose/under the eye, or something with reflections. I don't really know where to source cameras that small in the first place. Alas!

I think it would probably be accurate enough to reasonably guess whether or not you're looking+focusing at something near (torso, etc), if you're careful about having really skinny things tripping up the focus all the time. I dunno. Either way, I know it's already been done with regular displays + eyetracking. Bonus points for gaze-based head bob :

[youtube]http://www.youtube.com/watch?v=HdW1v9TPNYw[/youtube]

Funny to imagine trying to cram it all together, I guess. If there was a way to get a decent camera angle, it could be pretty fun to mess with. Even if the HMD can make things *feel* up close, I guess you'd still need post-processing blur to make it *look* blurred appropriately anyways. I tend to think of crysis' weapon customization as an example of nice post processing focus blur. I wonder how accurate the tracking would need to be.. the ps3 eye probably wouldn't fit into an HMD, but gets nice images at VGA res:

[youtube]http://www.youtube.com/watch?v=8edm_LGQWDA[/youtube]

[crespo80]

A servo system that adjusts lens distance based on the user's focus is great but will cost way too much in the near future.

In comparison, it's fairly simple and really cheap to only track the pupil of one eye in Infra-Red: all you need is a IR illuminator, a "hot mirror" and a tiny IR VGA camera, the software does the rest.
It could be implemented even in the consumer Rift if they wanted, and if it would permit a huge increase in performance if implemented corectly in game engines, as showed a few post ago by Mart in its illustrative picture

[brantlew]

A servo system that adjusts lens distance based on the user's focus is great but will cost way too much in the near future.

In comparison, it's fairly simple and really cheap to only track the pupil of one eye in Infra-Red: all you need is a IR illuminator, a "hot mirror" and a tiny IR VGA camera, the software does the rest.
It could be implemented even in the consumer Rift if they wanted, and if it would permit a huge increase in performance if implemented corectly in game engines, as showed a few post ago by Mart in its illustrative picture

These are two entirely distinct goals however and can't be compared. Dynamic eye-tracked downsampling is simply an optimization technique. Variable focusing is an immersion technique.

[zalo]

The leap motion is using small IR cameras. The Wii Remote used small IR cameras. I'm sure they exist somewhere.

I was thinking the hot mirror thing could be used with reflections.
There is no room for the camera without the mirror, however:

With the Mirror, you could put it between the screen and lens facing down:

Diagram.gif

This is overly simplified (partly due to my lack of artistic ability), but I imagine some adjustments to the angle of the mirror will need to be made to accommodate the small space between the lens and screen (unless the consumer version uses bigger screens and lenses so we have more room )

[brantlew]

@zalo: That's a pretty slick design.

[Namielus]

why does the mirror need to be 45 degrees and the ir-cam 90 degrees in relation to the eye/screen?
Will the hot mirror block light from other wavelengths than infrared if its steeper?
The camera can still be in the correct angle relative to the mirror?


PS:

Sorry I managed to miss the fact that you pointed out the angle adjustment yourself. I only saw the comment about the possibility of a bigger screen/more space.

[zalo]

The best artist on the forum decides to modify my crappy drawing instead of a new one. You can't even tell that it's not the original!

My only hesitation with steeper angles is the lens. It could refract in that weird thing lenses do when you're outside the exit pupil, making the image useless. (No time to look up what it's called now, but you end up only seeing the inside of the lens).

[Namielus]

Okay so Im lazy. Getting glass to reflect infrared only and let everything else pass through is a hard.
Also I could not remember the distance between the eye > lens > screen.
There is room to put the screen and lens much closer to the eye. Just easier to see this way.
The red beam of light you see here is not an actual simulation of light but a thin material tube emitting light.

eyetrack.jpg

The image is full of noise because it takes along time to calculate light (all the colors) from a screen passing through a convex lens after first passing through angled glass.
I might make a real simulation if I figure out how. These are double convex lenses and a 5.6" screen with
doom 3 in sbs on it

It was fun to see the actual image from doom3 projected onto the retina/eyeball. Kinda crazy how well you can simulate these things (for someone who is more skilled than I am).

[EdZ]

You may have heard of the Lytro lightfield camera: it sacrifices output resolution* for the ability to capture images at different focal lengths simultaneously by adding a microlens array on top of the sensor and computationally reassembling the image into multiple focal planes.

I wonder if the inverse is possible, producing a display that sacrifices output resolution to allow you to focus on it at different apparent depths. Most lightfield displays I can find have been aimed at producing different images from different angles (autostereoscopic displays), whereas a display for a HMD would only need to tailor for one viewpoint (or rather, a very small viewing window, due to the physical translation of the pupil as the eyeball rotates). The lenticular images you sometimes see on cards and game boxes (e.g. Vanquish) are a 1-dimensional example of the former. I'm not quite sure if you could produce an arbitrary focus lightfield with an homogeneous array of lenses, or if you'd need to pack in a microlens for each desired focal distance.


*According to wikipedia, a 16 megapixel sensor produced a 60 kilopixel image, so a factor of 266 reduction in resolution. A hypothetical 4K display (3840x2160) would have an apparent output resolution of 235 x 123. Less potential focal distances would presumably increase this.

[foisi]

Hi EdZ, it is possible and it's called integral imaging
http://www.mtbs3d.com/phpBB/viewtopic.p ... 56&#p76056

[Randomoneh]

Okay so Im lazy. Getting glass to reflect infrared only and let everything else pass through is a hard.
Also I could not remember the distance between the eye > lens > screen.
There is room to put the screen and lens much closer to the eye. Just easier to see this way.
The red beam of light you see here is not an actual simulation of light but a thin material tube emitting light.

eyetrack.jpg

The image is full of noise because it takes along time to calculate light (all the colors) from a screen passing through a convex lens after first passing through angled glass.
I might make a real simulation if I figure out how. These are double convex lenses and a 5.6" screen with
doom 3 in sbs on it

It was fun to see the actual image from doom3 projected onto the retina/eyeball. Kinda crazy how well you can simulate these things (for someone who is more skilled than I am).

That's your image? What did you use?

[Namielus]

Google sketchup for simple stuff like this and 3dsmax/blender for more complex stuff.
Rendering is Indigo Renderer

[EdZ]

Hi EdZ, it is possible and it's called integral imaging
http://www.mtbs3d.com/phpBB/viewtopic.p ... 56&#p76056

That's what I was talking about with existing autostereoscopic displays, but all existing implementations (that I've found) have been directed at showing independent images to multiple viewpoints, not multiple focal planes to a single viewpoint.

[zalo]

Polhemus (of all places) just posted this video:
http://www.youtube.com/watch?v=5xOzglScJ7g

It depicts an eye-tracking system using a transparent dichroic mirror (hot mirror!!!).

If someone were to use system like this in the rift, they'd have a pretty slick design indeed.