Light field display
Posted: Thu Jul 12, 2012 11:23 pm
Here's my idea for a light field display.
First off, though, what is a light field display? A light field is just the set of all the various rays of light within a given volume of space. A light field display is one that can replay all of those rays, or at least some interesting portion of them. You'd really only care about the rays that could possibly enter your eye, from wherever you're looking at the display from.
A regular 2D screen only replays those rays that are emitted in uniform cones from a flat surface. That is to say, each pixel emits the same color (and intensity) light in all forward directions (at least, ideal screens do this pretty well; with cheap LCDs, the color gets "wrong" as you look from off axis).
A light field display would operate similarly to a regular 2D screen, except that for every pixel, different color rays would go out in different directions. You could say that each pixel becomes like a little image itself, except that its sub-pixels correspond to different angular directions rather than 2D positions.
Such a display is what you'd use to make a Holodeck, if you wanted to go that route. With this display, you'd have to focus your eyes to see things at different distances. It would also offer motion parallax, in that if you move your head to see behind something, you'd be able to (while the display is showing a static light field). Multiple viewers could do this at once. In short, the ideal version of this display would pretty much be able to replicate anything that you could naturally see.
So now let's think about something else: the Lytro light-field camera. You may have heard about this camera that let's you take a picture and then focus it later? That's the Lytro. It captures a light field, and lets you choose how you want to reduce that light field into a regular 2D image later.
How does it work? It's quite simple: it uses a very high resolution image sensor with a microlens array in front of it. In essence, it's a big array of little cameras (or think about images within pixels again).
You might also want to consider another light-field camera: the "bullet-time" camera setup used for The Matrix. This was a (curved) linear array of cameras that captured the same shot from many different positions at the same time. By choosing which camera image to look at, you could achieve motion parallax. The Lytro is similar, except that instead of a big long line of cameras, they've been condensed into a 2D array and put into a little box.
The other difference is that the resulting images are all kind of interleaved. That's fine though, because you can sort them out in software.
Anyway, my final idea is quite simple: the Lytro can capture a lightfield using a high-resolution sensor, a micro-lens array, and the right software to sort things out.
To go in the opposite direction, you just need to replace the sensor with a display panel. Of course, the display panel needs the same kind of pixel count that the sensor has (which seems to be 11 - 16 megapixels). Also, the Lytro only captures light fields that enter its main lens, which is pretty small.
You can begin to see the problems: for a monitor-size light-field display, the number of "total pixels" (rays or cones, really) would be massive. As I said before, multiple each ordinary pixel by a whole image worth of pixels. The computation needed to generate such an image would scale similarly (though there would be lots of "shortcuts" you'd take to scale the problem down).
To make this practical, you'd want to limit the number of rays, and this is where HMDs come in. By putting the displays in front of your eyes, you really cut down the number of rays that you'd have to display and compute, since you only care about the ones that can enter your pupils.
This means that the microlenses don't have to project the micropixels into such wide angles. Also, you don't need symmetry, since micropixels on the left side of the display don't need to be projected towards the left, as you wouldn't see them. Of course, making each microlens a custom shape makes this a bit more difficult.
That's the beginnings of the idea. I need to hit the hay now.
First off, though, what is a light field display? A light field is just the set of all the various rays of light within a given volume of space. A light field display is one that can replay all of those rays, or at least some interesting portion of them. You'd really only care about the rays that could possibly enter your eye, from wherever you're looking at the display from.
A regular 2D screen only replays those rays that are emitted in uniform cones from a flat surface. That is to say, each pixel emits the same color (and intensity) light in all forward directions (at least, ideal screens do this pretty well; with cheap LCDs, the color gets "wrong" as you look from off axis).
A light field display would operate similarly to a regular 2D screen, except that for every pixel, different color rays would go out in different directions. You could say that each pixel becomes like a little image itself, except that its sub-pixels correspond to different angular directions rather than 2D positions.
Such a display is what you'd use to make a Holodeck, if you wanted to go that route. With this display, you'd have to focus your eyes to see things at different distances. It would also offer motion parallax, in that if you move your head to see behind something, you'd be able to (while the display is showing a static light field). Multiple viewers could do this at once. In short, the ideal version of this display would pretty much be able to replicate anything that you could naturally see.
So now let's think about something else: the Lytro light-field camera. You may have heard about this camera that let's you take a picture and then focus it later? That's the Lytro. It captures a light field, and lets you choose how you want to reduce that light field into a regular 2D image later.
How does it work? It's quite simple: it uses a very high resolution image sensor with a microlens array in front of it. In essence, it's a big array of little cameras (or think about images within pixels again).
You might also want to consider another light-field camera: the "bullet-time" camera setup used for The Matrix. This was a (curved) linear array of cameras that captured the same shot from many different positions at the same time. By choosing which camera image to look at, you could achieve motion parallax. The Lytro is similar, except that instead of a big long line of cameras, they've been condensed into a 2D array and put into a little box.
The other difference is that the resulting images are all kind of interleaved. That's fine though, because you can sort them out in software.
Anyway, my final idea is quite simple: the Lytro can capture a lightfield using a high-resolution sensor, a micro-lens array, and the right software to sort things out.
To go in the opposite direction, you just need to replace the sensor with a display panel. Of course, the display panel needs the same kind of pixel count that the sensor has (which seems to be 11 - 16 megapixels). Also, the Lytro only captures light fields that enter its main lens, which is pretty small.
You can begin to see the problems: for a monitor-size light-field display, the number of "total pixels" (rays or cones, really) would be massive. As I said before, multiple each ordinary pixel by a whole image worth of pixels. The computation needed to generate such an image would scale similarly (though there would be lots of "shortcuts" you'd take to scale the problem down).
To make this practical, you'd want to limit the number of rays, and this is where HMDs come in. By putting the displays in front of your eyes, you really cut down the number of rays that you'd have to display and compute, since you only care about the ones that can enter your pupils.
This means that the microlenses don't have to project the micropixels into such wide angles. Also, you don't need symmetry, since micropixels on the left side of the display don't need to be projected towards the left, as you wouldn't see them. Of course, making each microlens a custom shape makes this a bit more difficult.
That's the beginnings of the idea. I need to hit the hay now.