Real-time variable focusing (Volumetric) HMD

[brantlew]

Wow!! I just ran across this little paper from a guy at Stanford. Basically the system handles pupil tracking and can dynamically adjust the focal distance to compensate for your cornea lens. In essence it allows you to focus your eyes at any distance in the scene just like the real world. That has got to be insanely immersive !! Anyway, he's working on a commercial prototype HMD version of it.

http://dl.dropbox.com/u/13352595/volume ... isplay.pdf

DualLens.jpg

I've got a feeling he's lurking around here somewhere. Care to comment Jon Rodriguez?

[CyberVillain]

I've been dreaming about pupil tracking since the first HMD! Very cool!
That would also take away alot of eye / brain strain from Stereocopic 3D if you converge were the eyes want to converge...

Only problem is software support, the converge stuff could be integrated into Nvidia stereo vision and you get generic support, but for full support you want the view port to move when you move the eyes, but since Arma2 is the only game which support Head tracking I have a hard time believing eye tracking will be generic supported any time soon :/

[FingerFlinger]

Yeah, how could you hack that into an existing game? You'd need to cast a ray from the virtual pupil.

[brantlew]

It's my understanding that at least one of the 3D injection drivers (nVidia, Tridef) uses a similar algorithm where they cast a ray through the center of the view to the closest "signficant" object and dynamically sets the convergence to that Z-axis. That's pretty much the same thing isn't it? So it seems like you might be able to "hack" content for this type of system.

[Fredz]

I don't think casting a ray would be needed. You should be able to calculate the 3D position of the point the user is looking at only using the 2D coordinates of the points in each screen and extract the Z-Buffer value at these points.

Nice project indeed, although a bit low on details. I'd love to see some videos or more information about the effectiveness of the simulation and the impact of latency.

[suckmysound]

the bane of HMDs has been fixed focus and that too blocking the peripheral vision by forcing the user to experience 'tunnel vision' .
That's the reason these HMDs products are NOT Mainstream - meant only for niche category [VR].
regular LCD/LED TVs provide 3D but do not force tunnel vision or block peripheral vision hence they being 'Mainstream' products.
AR HMDs tried to get over this 'issue' by allowing peripheral vision or a part of it the way to look into real world without being totally secluded. But not totally eradicating the issue of fixed focus virtual imagery.
Moverio from epson is one such project in focus.
moverio is not truly AR but an attempt at that with transparent lenses connecting to 'real' world at the same time.
the unit is pricey - else i'd have definitely settled for it.

But we need a tech. like this which utilizes peripheral vision and gives us panoramic view of AR and VR without fixed focus.

[Omarzuqo]

It was an interesting paper, just one thing:

I'm no mechatronic engineer, but wouldn't it be easier to just artificially add the focus/unfocus blur in the game engine? Some games already do that (see picture).



By just having retina tracking data you could just blur the background or foreground and skip lens movement part and it's corresponding latency.

I'm not trying to be a smarta$$, just wondering.

[FingerFlinger]

I think that part of the effect is sensing how the muscles in your eyes need to contract in order to focus correctly on the scene. Just changing the blurriness of the image wouldn't create this effect, although it may still add a level of immersion.

[brantlew]

By just having retina tracking data you could just blur the background or foreground and skip lens movement part and it's corresponding latency.

I'm not trying to be a smarta$$, just wondering.

You're missing something fundamental here. When you look at a desktop monitor or an HMD your eyes are focused at a fixed distance. It doesn't matter if the scene is blurry or clear, the eye muscles have that cornea lens squashed into a constant shape. Now do this. Look at the edge of your desktop monitor and focus on some text. Now look just beside your monitor at the wall behind it. See how your eyes refocus and the screen text becomes blurry and the wall becomes clear? That is the effect they are trying to achieve with this lens system. They want you to be able to refocus your cornea and have the lens system compensate for that. So even though you will actually be looking at a physical screen only a few inches away, the lens system will allow you focus your eye at any depth and resolve objects at that depth in the virtual scene.

They could also do what you are saying and just track eye movement and blur the image around the point you are looking at. But that would be at a fixed focal distance which does not feel as realistic as when you actually allow your eyes to change shape.

[android78]

Imagine if you could have displays with extremely high refresh rates and have a system that is entirely integrated with the graphics card and the z-buffer. If it was to, at 30 cycles per second or so, run through the z-buffer and display the pixels that are at the correct depth to match the focus, then you could have many focal distances matching the actual depth. THAT would be cool. In the meantime, I think that having a system that will set the focus just based on the divergence of your eyes 'could' give a good alternative. I say could because moving your eyes round the scene may result in weirdness, given how quickly our eyes move and the potential tracking rates.

[brantlew]

It doesn't seem like the system has to do much while your eyes are moving. When I move my eyes around it doesn't feel like they smoothly refocus. It feels like my eyes have to stop moving before the cornea adjusts. Maybe that's just me.

[android78]

It doesn't seem like the system has to do much while your eyes are moving. When I move my eyes around it doesn't feel like they smoothly refocus. It feels like my eyes have to stop moving before the cornea adjusts. Maybe that's just me.

Maybe, but I think that if your eyes are trying to focus while the focus of the subject you're looking at changes, this could cause more nausea then having just a fixed focus.
These are just guesses though. At least it appears that someone is working on it.

[specfreq]

Finally, the thread I've been hunting for.

A couple hours of searching and I've only found one person that has put somthing like this into reality.
Here's a couple related links to his stuff:

https://docs.google.com/viewer?a=v&q=ca ... Fe0qOZAfGg

http://people.rennes.inria.fr/Anatole.L ... llaire.pdf

2:40 is the demonstration, you could imagine the effect this would have in 3D not to mention the Rift.
[youtube]http://www.youtube.com/watch?v=HdW1v9TPNYw[/youtube]

He's using an ASL 6000 for eyetracking, It's probably absurdly expensive (I couldn't find a price tag).
I did, however, come across this though http://www.cnn.com/2012/09/24/tech/mci- ... index.html "If no commercial partner comes forward within the next year or two, Faisal says they will publish all the information about their device and software online and make it freely available for anyone to replicate."

I wonder if there is a way to track the focus point of your eye by measuring the shape of the lens or something similar (not eye tracking). You can contract your eye muscles at will, maybe there is a way to replicate this in game for a more natural immersion.

keywords: gaze tracking

[mahler]

He's using an ASL 6000 for eyetracking, It's probably absurdly expensive (I couldn't find a price tag). I did, however, come across this though http://www.cnn.com/2012/09/24/tech/mci- ... index.html "If no commercial partner comes forward within the next year or two, Faisal says they will publish all the information about their device and software online and make it freely available for anyone to replicate."

I wonder if there is a way to track the focus point of your eye by measuring the shape of the lens or something similar (not eye tracking). You can contract your eye muscles at will, maybe there is a way to replicate this in game for a more natural immersion.

keywords: gaze tracking

And then today as CES 2013 this was demonstrated:
http://arstechnica.com/gaming/2013/01/l ... h-my-eyes/
At $995 for the developer version and no price given for the consumer version, this doesn't seem to be mainstream any time soon.

More info on Tobii products here:
http://www.tobii.com/en/eye-tracking-in ... -services/

___________________________________
On another note, I read this Microsoft Research Project on Foveated 3D Displays (20 Nov 2012)
http://research.microsoft.com/en-us/pro ... eddisplay/
Not really directly useable as an HMD's though.