Head tracked movie viewing

[JohnCarmack]

While Palmer and I have said many times that the Rift won't be a good device for 3D movie viewing, there is an idea that I would be interested in seeing someone pursue: head tracked pan-and-scan movie viewing.

Despite the great colors and ghost-free 3D display on an HMD like the Sony HMZ-T1, any head motion while watching a movie is very disruptive, because the world comes with your head, which is not at all what your brain expects.

If head tracking were integrated, you could put the movie on a screen properly positioned in a virtual space, and dial the size/seating position anywhere from TV screen to IMAX. I suspect that movie watching like this in a completely enclosing HMD may be rather compelling. The Rift resolution would still be low, like watching a DVD on a giant projector screen, but the immersion might (over?)compensate, at least for some people, and when panel resolutions get better, it might really be something.

There are probably more optimal screen layouts than just a flat quad in perspective -- curved screen, etc. You could even go so far as to do image/stereoscopic analysis to recover extra depth information to inform the final warp.

Copy protection would get in the way of doing this with standard DVDs / BluRay, and you would ideally want IMAX format video to take advantage of the wide field of view, but I'm sure some cool proof-of-concept video could be put together.

BTW, Despite all the great attention Rift is getting, I am going to be a bit scarce in coming weeks as I focus on other projects...

John Carmack

[FingerFlinger]

It would interesting to shoot with a fish-eye lens and map it to a sphere for head tracking and fixing distortion, as a "quick-and-dirty" implementation.

[Alkapwn]

It would interesting to shoot with a fish-eye lens and map it to a sphere for head tracking and fixing distortion, as a "quick-and-dirty" implementation.

Well I do have a GoPro HD Hero 2 I try try recording some crazy intense stuff on and see how that would map out.

[BillRoeske]

From a film history perspective, it would be kind of neat to see old specialty formats like Cinerama presented this way. You could also probably apply a filter to the frames of anaglyph 3D movies to split them out for a per-eye display. There's something funny about the notion of strapping on a HMD to recreate the viewing environment of old movies, sort of like cinematic archeology.

I wonder how far you could take sound integration. A first pass would obviously be to have the sound to pan along with the screen. Beyond that, it would be interesting to create virtual speakers at the ideal angles from the listener and spatialize them for the chosen "seat". I'm really not up on the state of real-time sound algorithms, but there might be the possibility of simulating various materials in the viewing environment if someone wanted to take it in the direction of architectural planning.

It would be neat to see something like recreations of famous theaters or other locations for viewing environments. A really silly pie-in-the-sky thought would be something like a recreation of Shakespeare's Globe theater with captured performances that can be viewed from any seat in the house.

Aside from fun experiments, though, are movies on a HMD something that people genuinely want for regular use? I mean that as an honest question. Movies are a social thing for most of the people that I know (even in the home), and there's a growing social stigma about paying more attention to cell phones than the real people you are near. I know that watching a movie on a huge screen while on an airline flight is one of the big promises of HMD marketing, but I'll be kind of floored if I ever actually see that happen out in the wild (most are working, sleeping, or reading). At minimum, it's more junk to haul through security.

[Okta]

Not sure how much of an issue this could be but the the fixed physical side by side nature of the Rift could pose extra problems for tracked viewing. A fixed SBS image is great but what happens when you track across it you see the split. Tracking code would need to take into account the split region halves, which would need to be zoomed anyhow, and overlapped to correct the fixed lens centre focus per side.

I played briefly with the VR920 head tracking support in steroescopic player which was pretty nifty so its not a bad idea..

[NRainville]

If you treat each SBS image as a 2D image occupying the same location in 3D space, then tracking around that should give you an effect similar to what happens in real life when you shift your head around a 3D monitor. You would just have to select which version of the scene to display to each eye before sending it to the HMD.

To make this idea more general though, it would be interesting to have a generic 2D interface modeled in a 6DOF 3D sim. Something like a VNC client mapped to a modeled projection screen. Translation of the perspective might be entirely useless, but who knows until you try?

[FingerFlinger]

Even if you just spoof the same image to each eye for 2D, the head tracking component would still pump up the immersion level big time.

[PalmerTech]

This is something that could be awesome, especially with IMAX screens. A few ideas on my end:

1) 360 degree video capture. The captured resolution already had pretty low resolution, so the step down would not be too bad. Imagine using this, but instead of moving your mouse, you have full head tracking: http://edition.cnn.com/interactive/2010 ... index.html

2) If you are going to render a video onto a big curved screen in a virtual environment, you could even do something like model the whole theater, so you can look behind yourself and see seats. It would give you a good frame of reference for how big the FOV is, kinda of like how the image appears larger in an HMD when you are comparing it against something far away.

[Nick3DvB]

IMAX on your head?! erm, yes please!!!

Video is starting to sound very do-able, either with optics on the camera, distortion mapping in software, or some combination. The Kinect guys are already doing a lot of great imaging work, the RIFT could be a good way of experiencing "bullet-time" style effects http://www.youtube.com/watch?v=WYezwsFfxvE and and then there's all the plenoptic array / light-field stuff - like the NHK SuperHiVision and Lytro cameras: http://blog.lytro.com/wp-content/upload ... ad-400.gif

I really don't think the resolution thing is a big issue, hell I still watch video on my old SVGA DLP sometimes, even a 1.85:1 or 2.39:1 aspect movie scaled down into it's 4:3 window is still pretty watchable if you ask me (and I've got 20/20 vision). The Slysoft guys can deal with the content protection and hopefully the chroma-aberration correction and other warping could all be done in real-time with MPC shaders or AVISynth scripts, I have used something call DGDecNV to get CUDA acceleration on AVIsynth scripts before (for basic de-anaglyph, Sisvel 3D conversion etc).

The tracking ideas sound really interesting, I have never experienced it myself but the video movement problem seems like another facet of the "missing parallax" effect I was going on about the other day, your brain just knows when something isn't quiet right. But at least now we might actually be able to do something about it... A cinema type experience for standard 3D movies is a great secondary feature but the stuff shot specifically for the RIFT could be a really amazing proposition.

The RIFT is the gift that keeps on giving!

[brantlew]

Until much higher resolutions are available this is probably the only way to use the Rift for a PC desktop as well. Projecting the desktop onto a virtual rectangle or maybe a hemispherical shape combined with head tracking would allow you to look at all parts of the screen. With accurate 6DOF tracking (FreeTrack) you could implement head zoom which would help the low resolution problem. You could display a blurry high resolution virtual desktop and then just move in closer to resolve the details - like a near-sighted person viewing a physical screen.

[cadcoke5]

Billrift says; "A really silly pie-in-the-sky thought would be something like a recreation of Shakespeare's Globe theater with captured performances that can be viewed from any seat in the house."

I don't think the idea is silly. A play that is simply turned into video by a camera recording a live performance is normally a very boring show. But, something different happens when you are there watching the performance live. A HMD might be able to recreate that experience. This also makes an entire world of entertainment content available to the VR experience.

For something like the Globe, as you enter the theater, you are experiencing the theater itself. That is an important thing when you are watching an historical play. It really adds to the experience. But, even for modern plays, in modern theaters, there is something special about a live show that just does not exist when watching a movie.

I don't think, however, that multiple seat viewing of the performance is a benefit. You typically watch an entire play from one seat. Even slight head movements make little difference to detecting the 3-D aspects of the play, since it is so far away. I wonder if the play even has to be shot stereoscopically, since stereo vision does not really do much for you at the typical distances involved. Though, if you were fairly close to the stage, it might. But, theater often makes use of the poor stereoscopic vision at typical viewing distance. It is common to have background scenery painted using the rules of perspective, to convince the mind that it is 3D scenery, and not a flat image. If you keep the viewer in one seat, the burden to do 3D capture of the actors and the scenery itself is greatly reduced. The possibility of using any video of a live play then becomes possible.

I do think that the resolution of the current Rift is a problem. You need to be able to see actors faces clearly. In a movie, you typically get close-ups, but watching a live play with faces only occupying a few pixels is likely to be a problem.

Regarding the resolution issue, I hope the people working on some of the warping software will plan on incorporating adjustments for chromatic aberration mapping in addition to the geometric mapping. Getting better resolution will likely require a prism in the optics to allow for larger, higher resolution displays. The prism will separate the R,G & B colors and unacceptable amount. This simple prismatic shifting of the colors would be easy and fast to correct in the driver. Though, lens chromatic aberration will probably be more intense. Fancy optics can help with this, but designing and making those will be beyond the hackers reach. Chromatic aberration has to be dealt with in the software.

I just said the software has to do the chromatic aberration processing, but I suppose the entire warping processing could be handled in hardware. If a special board is eventually made to drive the display, it might be good to try to put this sort of processing on board. It would be comparable to the way the NEC PX700W Projector does geometric mapping. I suspect that hardware geometric mapping and chromatic aberration processing would be a great way to decrease lag and make the actual drivers easier. This sort of mapping is really is a problem that is inside the HMD and does not vary with the program that is displaying on it.

Joe Dunfee

[Mark2036]

I also have hoped for something like this to be utilised with HMDs. In addition to virtual cinemas and stage shows, you could also imagine what sitcoms could be with 360 view.

I imagine video cameras can be replace by panoramic 360 fish eye type cameras with binaural microphones set at multiple directions, placed at key locations around the sitcom set (such as in the audience, on the couch inside the show's lounge room, standing behind the shows kichen counter, etc). Video and sound can be captured from all angles at the capture point, the AV data can then be displayed inside a HMD depending on the orientation of the users head.

Sensics has a video of doing something similar with iMove (using a HMD to experience a 360 video of a drive around Washington)

www.youtube.com/watch?v=ilXUqXPnPOY

[Klaus]

Maybe a bit off-topic since static, but an interface to google street-view would be cool. Of cource no 3D but I think the immersion would be nice, even though the street-view resolution is not that great and reloading is quite slow.

[brantlew]

I imagine video cameras can be replace by panoramic 360 fish eye type cameras with binaural microphones set at multiple directions, placed at key locations around the sitcom set

I think it would be extremely challenging to direct and frame a full 360 screenplay. Right now most shots are easy to setup because you only have to frame a tiny little window. All the equipment, staff, lights, boom mics, etc.. are just out of frame and you don't have to worry about them. To frame an entire 360 set would be a monumental undertaking. All the equipment and people would probably have to be hidden under floor or behind green screen. Lighting would be nearly impossible because all "natural-looking" light sources would be required. Multi-camera shots would be impossible since every camera would be able to see each other - so every scene would have to be reshot multiple times for multiple viewpoints. It's hard to imagine a lot of directors embracing this type of project because all the technical photographic details would overwhelm the creative aspects. Liberating for the viewer - but stifling for the filmmaker.

Maybe a bit off-topic since static, but an interface to google street-view would be cool. Of cource no 3D but I think the immersion would be nice, even though the street-view resolution is not that great and reloading is quite slow

That's a good idea. Not especially useful, but it would be an easy way to get a lot of panoramic content.

[CyberVillain]

I imagine video cameras can be replace by panoramic 360 fish eye type cameras with binaural microphones set at multiple directions, placed at key locations around the sitcom set

I think it would be extremely challenging to direct and frame a full 360 screenplay. Right now most shots are easy to setup because you only have to frame a tiny little window. All the equipment, staff, lights, boom mics, etc.. are just out of frame and you don't have to worry about them. To frame an entire 360 set would be a monumental undertaking. All the equipment and people would probably have to be hidden under floor or behind green screen. Lighting would be nearly impossible because all "natural-looking" light sources would be required. Multi-camera shots would be impossible since every camera would be able to see each other - so every scene would have to be reshot multiple times for multiple viewpoints. It's hard to imagine a lot of directors embracing this type of project because all the technical photographic details would overwhelm the creative aspects. Liberating for the viewer - but stifling for the filmmaker.

Cameron would

[cybereality]

Using traditional camera rigs, yes, it would be difficult. But if you had more advanced depth cameras (ie like the Kinect) you could record the scene as point clouds (and merge data from multiple cameras to handle occlusions). This would allow for not only 360 degree viewing, but stereo (with adjustable depth/convergence), head-tracking, even walking around the scene to look from different angles. Probably a few years away from having the technology to do this, but in theory it could work.

[Alkapwn]

I imagine video cameras can be replace by panoramic 360 fish eye type cameras with binaural microphones set at multiple directions, placed at key locations around the sitcom set

I think it would be extremely challenging to direct and frame a full 360 screenplay. Right now most shots are easy to setup because you only have to frame a tiny little window. All the equipment, staff, lights, boom mics, etc.. are just out of frame and you don't have to worry about them. To frame an entire 360 set would be a monumental undertaking. All the equipment and people would probably have to be hidden under floor or behind green screen. Lighting would be nearly impossible because all "natural-looking" light sources would be required. Multi-camera shots would be impossible since every camera would be able to see each other - so every scene would have to be reshot multiple times for multiple viewpoints. It's hard to imagine a lot of directors embracing this type of project because all the technical photographic details would overwhelm the creative aspects. Liberating for the viewer - but stifling for the filmmaker.

Cameron would

Crank 3. Jason Statham would be running around with that on his head and you'd capture everything happening. and then other scenes the directors could just run in front of him with it on their heads so that was they can capture his motion as well as the whole environment.

[DougWolanick]

I just tried out putting a 3D video into a 3D environment. Looks pretty cool, but a pretty impractical way to watch on a monitor, maybe with the immersion of the Oculus Rift it could be a fun way to watch things.

I can imagine watching Lord of the Rings in a giant floating screen in Skyrim. Although pretty much anything would be amazing in a VR Skyrim.

[youtube-hd]http://www.youtube.com/watch?v=9pIZZi0JRks[/youtube-hd]

[brantlew]

That's great! SkyRim drive-in mod

[DougWolanick]

That's great! SkyRim drive-in mod

Exactly what my wife said when I tested it on the TV. Maybe 'Need For Speed: Drive-In'. Complete with parking simulation, walking to the concession stand and hiding friends in your trunk.

[cybereality]

Wow! Great stuff DougWolanick.

[cadcoke5]

brantlew said: I think it would be extremely challenging to direct and frame a full 360 screenplay.

I used to work at Sight & Sound Theatres (in Stasburg PA and Branson MO). This is a theater with an extremely large, wrap-around stage. There is no question that the additional space creates a challenge. You have to lead the audience's eyes, so that they know where to see the important action.

I also think failure to control the audience's gaze is a contributing factor with getting headaches at a 3D movie. If you look away from where the director wanted you to look, or they fail to keep good control of this aspect, the convergence can be off enough that you strain your eyes more. Though that would be resolved if the entire scene is recreated with 3D modeling.

As I posted before, I don't think that most existing entertainment would benefit from the ability to walk around for the actual show. Something that benefits from VR is 3D sculpture, which is meant to be enjoyed by moving around it. Historical reenactments of a famous speech, such as Lincoln's Gettysburg Address would surely benefit, since you would be immersed in the times.

Joe Dunfee

[brantlew]

@cadcoke: I agree. Giving the audience free-look changes a fundamental aspect of cinema - the directors viewpoint. None of the current directing or editing techniques would work properly with this added dimension. It would be an entirely different art-form and would require a fundamentally different method of direction and presentation.

[PalmerTech]

It would be hard for live-action... But animated/CG heavy movies would be a lot more plausible.

[profvr]

Here is a camera that could film such scenes PTGrey's Ladybug 3. The optics is a problem; because the individual cameras don't have a common optical centre, you can't get a completely surround image at all depths.

There was an agency, yellowbird, that was making these surround videos, but their website seems partly broken at the moment, youtube has an example of what they proposed. I've seen their snowboarding video on a CAVE-like display, it was very immersive.

[Skaven252]

How about a slightly different approach. Rift was planned for playing first person shooters and to do virtual reality. But how about first person video?

I have for a while been experimeting with a filming technique that allows me to record stereoscopic video from a first person perspective, and combine it with binaural sound. Together the two create a highly immersive experience. As if you were in the camera(wo)man's shoes, so to speak.

When Rift was announced, I immediately thought it would be the perfect for watching these videos. Head tracking is not needed for this application though: actually the view has to stay locked to the head rotation, together with the sound (just like your eyes and ears are) in order to retain the sound's directionality. It's the camera (wo)man who decides where to look. The viewer can move their eyes to look at different parts of the FOV but that's it.

If any one of you have an ultra wide FOV HMD and good headphones, you are more than welcome to try out the 3D first person footage I've shot.

I treated the older videos with Defish to make them rectilinear, but then I learned that Rift actually works better with fisheye footage. So the last post in the thread links to a fisheye edit of my roller coaster video. Presumably that one works better with Rift?

[Cromfel]

Skaven, if you used multiple microphones and panned between them combined with omnidirectional camera. That would allow you to use rift with headtracking and get decent sound placement also. you could even quite easily prototype it with using unity3d? (Maybe there is easier ways). It would be darn nice to reproduce your rollercoaster experiment with accelerometer and combine this rig with one of those cheap rally motion platforms... hmmm...

[Skaven252]

That's why "stereo camera glasses with ear mikes", which you could plug into your smartphone, would be a clever product. The smartphone would contain the accelerometers, the motion tracker and even a GPS. Most of them do anyway.

[Namielus]

Hello, i recall the disney world resort had a 360-degree cinema at epcot at some point,
as well as a local norwegian special kind of cinema had 225degree footage recorded with 5 cameras.
The norwegian thing had about 35 movies produced and was called "supervideografen"
I assume lots of that footage could be adopted for Oculus Rift

[Mark2036]

There appear to be some cameras out there that could be used for full surround vision 3D movies.

Below is an article from 2010 about one such camera based on the eye of a fly, and about the size of a golf ball, that can film real 3d scenes. Although haven't been able to find much on what happened to the development beyond 2010 ...

http://phys.org/news/2010-12-degree-cam ... video.html

[profvr]

That camera is really just a surround (omni-directional) camera, the article states that there is little overlap between the lenses. The Ladybug camera has been available from PointGrey for years produces lower-resolution versions (still 2K wide I think). They mention reconstructing the depth, so as to be able to do stereo; you could do something similar with two Ladybug cameras mounted above each other and use vertical stereo disparity to re-project horizontal stereo pairs

[FingerFlinger]

@profvr

Do you mean you could just pipe the top video stream to the left eye and bottom to the right eye, or actually reconstruct the depth information for each screen, region-by-region? I'm curious how the brain would handle situation #1...

[profvr]

@profvr

Do you mean you could just pipe the top video stream to the left eye and bottom to the right eye, or actually reconstruct the depth information for each screen, region-by-region? I'm curious how the brain would handle situation #1...

That wouldn't work I don't think, though it might be interesting to look at!

The difficult thing would be that with two vertically mounted omni-directional cameras you would have two images with optical centres that were vertically above each other. You need to reproject that in to pairs of stereo images for left and right eye, offset about the vertical axis.

You couldn't place the omni-directional cameras side by side as they would see each other, and the eyepoints would be static: you might as well just use two wide angle cameras (e.g. GoPro 3D which is a common choice for hobbyists in the areas).

[FingerFlinger]

Has this actually been implemented? Does it need to be done manually? It sounds pretty cool.

[Owen]

Until someone finds a way to capture (not to mention store) a volumetric representation of real scene, probably the best approach is to use something like a panoramic depth sensing camera. Maybe use an array of two or three so that you can fill in the holes that depth re-projection will leave as you shift the viewpoint around.

[MemeBox]

Using traditional camera rigs, yes, it would be difficult. But if you had more advanced depth cameras (ie like the Kinect) you could record the scene as point clouds (and merge data from multiple cameras to handle occlusions). This would allow for not only 360 degree viewing, but stereo (with adjustable depth/convergence), head-tracking, even walking around the scene to look from different angles. Probably a few years away from having the technology to do this, but in theory it could work.

I watched Prometheus, the new Alien movie, an insult to the original, but:

It has lots of 3D scanning in it.
Probes which fly around producing a 3D map.
Ancient holograms which are clearly coloured point clouds.

Quite prescient I think.

Films do seem to be quite good at feeling the pulse of the Zeitgeist, I'm looking forward to Skyfall, for the same reason.
I'm hoping there will be lots of "banker bashing" or something similar...

[rhinosix]

One method for making movies I've been thinking about:

* Remove all people/ moving objects from a location, and use a LiDAR to scan the scene when the weather is overcast and light is diffuse. Remove any moving objects or people.
* Or alternatively, build an original set in CG.
* Add the moving objects as CG (trees, flag poles, water, etc.) and apply physics or animate by hand.
* Have actors filmed from every angle against a green screen/ stage to create a 3D model of their actions in motion (similar to L.A. Noire/ Quantic Dream/ other types of mo-cap).
* Add a skybox and lighting/shadows last.

The director would be able to move actors around and change lighting. If they wanted they could shoot the film with virtual cameras to display on a flat screen, or have the viewer able to walk around in the 3D scene. I was thinking it would be cool to watch a movie on a flat screen inside of a virtual environment, but still be able to step inside the screen and walk around if you wanted.

Another thing I was daydreaming about is a room with a laser scanner on each wall which is scanning frames at high speed. So instead of just having the visuals scanned, you would also get a 3D model of the contents of the room for each frame. The whole room would be like a video camera/ laser scanner.

I'd like to make a movie where the viewer can walk around inside a scene, pass their hand through actors, sync their vision with any of the characters.

For now, watching regular films in a virtual environment is still pretty exciting though.

[Alkapwn]

One method for making movies I've been thinking about:

* Remove all people/ moving objects from a location, and use a LiDAR to scan the scene when the weather is overcast and light is diffuse. Remove any moving objects or people.
* Or alternatively, build an original set in CG.
* Add the moving objects as CG (trees, flag poles, water, etc.) and apply physics or animate by hand.
* Have actors filmed from every angle against a green screen/ stage to create a 3D model of their actions in motion (similar to L.A. Noire/ Quantic Dream/ other types of mo-cap).
* Add a skybox and lighting/shadows last.

The director would be able to move actors around and change lighting. If they wanted they could shoot the film with virtual cameras to display on a flat screen, or have the viewer able to walk around in the 3D scene. I was thinking it would be cool to watch a movie on a flat screen inside of a virtual environment, but still be able to step inside the screen and walk around if you wanted.

Another thing I was daydreaming about is a room with a laser scanner on each wall which is scanning frames at high speed. So instead of just having the visuals scanned, you would also get a 3D model of the contents of the room for each frame. The whole room would be like a video camera/ laser scanner.

I'd like to make a movie where the viewer can walk around inside a scene, pass their hand through actors, sync their vision with any of the characters.

For now, watching regular films in a virtual environment is still pretty exciting though.

So pretty much Avatar:
http://www.youtube.com/watch?v=OJ1JzYPjcj0#t=5m

[rhinosix]

Hmmm, similar. But I'd like to be able to capture actors while fully clothed instead of wearing motion capture suits. Like a full body version of what L.A. Noire did.

And I was thinking more of creating digital versions of real world locations so they can be used across multiple projects, rather than making original fantasy locations from scratch.

[Nick3DvB]

For standard 3D movie content I think we are always going to have to push the image back onto a virtual screen (but not necessarily a flat screen) then use tracking for panning, it might be more natural just to create a "virtual cinema" (as that is what the video was created for), but we could try and do something in software to fill the remaining FOV and remove the hard edges of the screen, a bit like Philips Ambilight for TVs.

I just stumbled across the "Smooth Video Project" and it seems they have accidentally created a great Rift video player for regular & 3D content! SVP is really designed for motion interpolation (and it does a better job of it than most HDTVs) but it also has a great "AmbiLight" type feature. It can do all the cropping / scaling already, we just need to integrate the de-fish avisynth script or a pixel shader to deal with the warp and hook the zoom / pan&scan up to the head-tracking. 3D Ambilight is broken at the moment but should be fixed in the next version:

http://www.svp-team.com/wiki/Setting_up ... pic_Player

I've been testing it on my 4:3 aspect ratio 3D projector and it works very well.

Subtle settings work best, try 4,2,100,10. I took these screenshots but stills really don't do it justice:





You need to edit the script to add ambilight to the sides, good for 16:9 projector "overscan":




We obviously need to shrink the video frame right down for the Rift:






As you can see my first attempt at Rift barrel warping went very wrong! (aspect ratio etc)

But there is no real need to correct the Rift's natural warp, the screen area is flat enough and it might help the ambilight effect:




We could even add some more inwards warp:




But its probably best not to warp the video window itself?