Meant to be Seen

You really don't get the AmbiLight effect with the still captures above so I've uploaded a few videos, they are only crappy low rez Fraps captures but hopefully you can get the idea, bear in mind the video window will be *very* low resolution on the Rift anyway, (about 20% of the FOV, which is only 50% of the panel rez!) so these videos probably aren't too far off. LOL We really need that 4K panel!

[youtube]http://www.youtube.com/watch?v=nFlRHRXdHqw[/youtube]
http://youtu.be/nFlRHRXdHqw

[youtube]http://www.youtube.com/watch?v=FU-9D9ATQps[/youtube]
http://www.youtube.com/watch?v=FU-9D9ATQps

(I've given up trying to get the embed working!)
[edit by cybereality: I fixed the embed for you.]

We are stuck with the low rez but at least SVP can interpolate everything to 60fps, which looks great. I still need to correct the aspect ratio, try different amiblight patterns and find a way to blend those hard edges. Then its just a matter of integrating head-tracking for pan & scan, the big idea is to zoom the video window in a bit so when you see something in your peripheral vision (from the ambilight) you move your head slightly and the video frame pans to reveal more of the object that triggered that effect, this should also help solve the general video on HMD head movement issue to. If anyone can test SVP Ambilight out on a DIY Rift I'd be very interested to know what you make of it, I hope to be able to test it myself very soon...

Nick3DvB wrote:We could even add some more inwards warp:

Wouldn't you still want to barrel distort rather than pincushion, even near the centre?

The warp needs to go the opposite way, but otherwise looks promising.

Also, you can probably make the screen a lot bigger. It should almost reach the sides and just be letterboxed on top and bottom.

Thanks for sorting the embed problem Cyber, is there a bug with SD videos?

As you can see in the screenshots I did try warping both ways, but looking at the distortion grid I posted above there is no need to correct the optical warp because it won't effect the video frame too much unless I make it much larger (the uncorrected optical warp will be much less extreme on the Rift than in the last few screenshots). I want to try it uncorrected because I think some pincushion warp actually helps the ambilight effect, I thought about adding even more (in software) for a "wrap-around" screen effect, but there is probably a better way to do that for the screen frame only, it depends a lot on how the head-tracked pan & scan is going to work.

I'm trying to work out how large you can make the screen without blowing your head off, but I only have my CRT and projector to test with at the moment, I can't stand in front of my projector screen and I can't simulate infinite focus on the CRT (I tried, now I have a headache!) But I do have an old video "mp3 player" with a 1 inch OLED screen so when I've recovered I'm going to try some more tests using a map magnifying glass with that.

I have finally worked out the aspect ratio issues so the video bellow should be more like it:

[youtube-hd]http://www.youtube.com/watch?v=eTs6Sd_SCN8[/youtube-hd]

(slow-mo was a bug in the encode but helps show the ambi-light better)

I've uploaded some samples here:

https://rapidshare.com/files/1648185226 ... 0Loop.m2ts
https://rapidshare.com/files/3565215252 ... z60fps.mkv

ADDED: Full frame SBS with some barrel warp correction (but black boarder & wrong IPD?)

https://rapidshare.com/files/248979335/SBS.mkv

If any of the DIY Rift crew want to try it out then Stereoscopic player's SBS output mode should do it. In terms of the resolution issue I think we might actually be pleasantly surprised, it shouldn't be much worse than watching a 2.40:1 widescreen movie on a SVGA projector, and many are happy doing that. Yes, the horizontal resolution will be much lower, I guesstimate about 420x240 when cropped to a 16:9 frame, but what it actually looks like depends a lot on how much the optical pixel compression effect increases the number of physical LCD pixels in the center.

One other thing you don't get in the youtube videos is the frame rate interpolation, which I can push up to 144fps on my CRT! Even at 60pfs the perceived increase in temporal resolution could help offset the low video frame resolution a bit. It will be interesting to see how the "motion blur" persistence effect interacts with it, which is down to the response characteristics of the panel itself (switching time etc) I wonder if we can find a way to push the new panel to 72Hz?

Nick3DvB wrote:Thanks for sorting the embed problem Cyber, is there a bug with SD videos?

As you can see in the screenshots I did try warping both ways, but looking at the distortion grid I posted above there is no need to correct the optical warp because it won't effect the video frame too much unless I make it much larger (the uncorrected optical warp will be much less extreme on the Rift than in the last few screenshots). I want to try it uncorrected because I think some pincushion warp actually helps the ambilight effect, I thought about adding even more (in software) for a "wrap-around" screen effect, but there is probably a better way to do that for the screen frame only, it depends a lot on how the head-tracked pan & scan is going to work.

I'm trying to work out how large you can make the screen without blowing your head off, but I only have my CRT and projector to test with at the moment, I can't stand in front of my projector screen and I can't simulate infinite focus on the CRT (I tried, now I have a headache!) But I do have an old video "mp3 player" with a 1 inch OLED screen so when I've recovered I'm going to try some more tests using a map magnifying glass with that.

I have finally worked out the aspect ratio issues so the video bellow should be more like it:

[youtube-hd]http://www.youtube.com/watch?v=eTs6Sd_SCN8[/youtube-hd]

(slow-mo was a bug in the encode but helps show the ambi-light better)

I've uploaded some samples here:

https://rapidshare.com/files/1648185226 ... 0Loop.m2ts
https://rapidshare.com/files/3565215252 ... z60fps.mkv

ADDED: Full frame SBS with some barrel warp correction (but black boarder & wrong IPD?)

https://rapidshare.com/files/248979335/SBS.mkv

If any of the DIY Rift crew want to try it out then Stereoscopic player's SBS output mode should do it. In terms of the resolution issue I think we might actually be pleasantly surprised, it shouldn't be much worse than watching a 2.40:1 widescreen movie on a SVGA projector, and many are happy doing that. Yes, the horizontal resolution will be much lower, I guesstimate about 420x240 when cropped to a 16:9 frame, but what it actually looks like depends a lot on how much the optical pixel compression effect increases the number of physical LCD pixels in the center.

One other thing you don't get in the youtube videos is the frame rate interpolation, which I can push up to 144fps on my CRT! Even at 60pfs the perceived increase in temporal resolution could help offset the low video frame resolution a bit. It will be interesting to see how the "motion blur" persistence effect interacts with it, which is down to the response characteristics of the panel itself (switching time etc) I wonder if we can find a way to push the new panel to 72Hz?

Hi Nick,

I tried out the SBS clip in my DIY HMD, and it looks surprisingly good. The barrel effect is not fully corrected, and the image could use scaling up to take up more of the available screen real estate, though. This definitely has big potential. Movies would be easily watchable, even as it is.

Here's what I see:
Edit to fix my constant misspelling of HMD (HDM?).

Thanks Mel!

It looks like the warp is actually over-corrected, I was expecting to see a black boarder on the sides to but the image seems to be compressed vertically, I think you might need to set the mediaplayer to 100% scale or "fit to screen" mode, and maybe turn off any aspect-ratio lock (or force it to 4:5). Is the white bar at the bottom the edge of the LCD? The lens seems to be further away than I expected, is that for personal focus reasons? I'm going to encode some 3D samples with the AmbiLight baked-in and the fps fixed but I want to make sure the warp and IPD are ok first, thanks again for your help.

Nick3DvB wrote:Then its just a matter of integrating head-tracking for pan & scan, the big idea is to zoom the video window in a bit so when you see something in your peripheral vision (from the ambilight) you move your head slightly and the video frame pans to reveal more of the object that triggered that effect, this should also help solve the general video on HMD head movement issue to.

I've just been playing with head-tracking for Pan&Scan on my projector, MPC has very comprehensive controls you can map to but it's nowhere near smooth enough and the main problem at the moment is that it pans everything, including the ambilight bars, to achieve the effect above we will need to hook the head-tracking into SVP directly, I'm looking into that now.

The pan & scan effect I'm going for is quiet hard to describe, when watching a 3D movie through SVP on the Rift with head-tracking enabled it should be just like looking through glasses where the ambilight bars form a semi-transparent "frosted" border around the lenses, the movie world appears stable because instead of looking at a "screen" the ambilight bars form a viewport window that pans across the stereo image as you move your head (even a few mm) I hope that makes sense...? It should look awesome at 60fps but for your brain to buy it we HAVE to get the panning just as smooth as the interpolated video framerate. The rift tracker will be perfect for this because, we don't really want to map head translations anyway, it might be cool for zoom but lateral motions would produce an uncomfortable stereoscopic effect.

Nick3DvB wrote:Thanks Mel!

It looks like the warp is actually over-corrected, I was expecting to see a black boarder on the sides to but the image seems to be compressed vertically, I think you might need to set the mediaplayer to 100% scale or "fit to screen" mode, and maybe turn off any aspect-ratio lock (or force it to 4:5). Is the white bar at the bottom the edge of the LCD? The lens seems to be further away than I expected, is that for personal focus reasons? I'm going to encode some 3D samples with the AmbiLight baked-in and the fps fixed but I want to make sure the warp and IPD are ok first, thanks again for your help.

Yes, the white bar at the bottom is the LCDs aluminum bezel. As you guessed, I have the lenses set further away for focus reasons.

I'm going to try your settings suggestions, and I'm looking forward to anything else you've got...no pressure, of course

I'll try and post some 3D ambilight clips tonight, but I need to work out the best depth to render the frame at first. If you can live without ambilight you should be able to play almost any 3D video on the Rift using Stereoscopic player's fullscreen un-scaled mode, just set it to half height and correct the aspect-ratio to 4:5, then output to SBS, you can use the zoom presets to reduce the screen size and the separation controls to tweak IPD, had anyone tried that yet? You might get some wrap-around effect when the "screen" is very large, you could try WarpInject but if that won't work I'll try and post an avisynth script that should do the trick.

Really nice work. I think its important to have an easily accessible option to adjust the strength of the ambilight. So that people can adjust it themselves in case its a little too strong for them

Thanks! I am still working on that, you can reduce the length of the light flares but that gives you a black boarder again, I need a way to reduce their brightness relative to the screen. I thought about trying to blur / blend the edge of the screen but I am not sure about how that would effect the 3D viewport. I think is probably best to render the ambilight frame with no depth but I'm not sure, I will do some more tests and post some videos to compare. How is your project going? Are you making a type of "virtual cinema" experience - will we be able to choose where to sit ?!

yes its also multiplayer compatible so you can sit with your mates and watch live streams of football, starcraft streams - you name it.

I have gotten far on that now.

Cool! Multiplayer is a great idea, the isolation issue could be a big problem for HMD adoption, with everything now trending towards social gaming etc. I have no idea how the video texture update is synced etc. but you might want to try using SVP on your video input, just for frame interpolation, so the video matches the rendering frame-rate. But for your idea it might actually be better to keep the video judder in, so your virtual screen looks more "real" ?

It might be a good idea to have the choice atleast, but thats something you would have to set before putting on the HMD and launching the player.

The user interface has to be really simple and basically symbol based. Or very large letters. I can forget having the menu on the virtual screen. Its too small for that.
You would have to browse your folder on a bigger user interface in the hmd, but so far its seems better to set all that up before loading it and just browsing based on large symbols.

So far I am progressing into having support for all 2D content you can play with quicktime and mediaplayer/directshow so things like that sounds possible.
I am really trying to get 3D stereoscoping video rendering to the texture as well.

The good news is I will be supporting streaming from a capture device. So people will be able to play ps3 over component on their Oculus Rift in a rather unexpected way.

Watching a "3DTV" in a 3D world would be great, I wonder if you could add even more levels to it - a 3DTV in a 3DTV in a 3D world etc

If you support DirectShow you can use the BDA input for live TV as well as input capture cards,

I wonder who will be the first one to loose their TV remote control in the virtual world...

As soon as I get the dev-kit version of rift I am going to UK for some meetings. Maybe we can meet up since we are working on the same things.
I will let you try my demos.

That would be great, are you going to see Palmer's keynote speech in the UK at Evolve?

Sadly it is too expensive for me but a few of us have started thinking about meeting up in London at the same time.

http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=15721

Just wanted to jump in here and say super great friggin work here guys! A bunch of the stuff you guys are talking about I have no idea about, but this looks super crazy fantastic! Keep up the good work!

Nick3DvB, I think its going to be expensive for me too both in terms of time and money.
I dont have head tracking yet, and I think the SBS aspect ratio is wrong.
Hoping that getting an actual rift and the SDK will let me fix that so I have something substantial to bring to England and
show to fellow riftoholics.

No problem, it' s a long way to come just for a few hours. I have just moved house and am going to be really busy up to Christmas, but after people have had their hands on the kit for a few months we should definitely organise something in the new year. The 4:5 aspect ratio is causing me a few problems to, and the head-tracked pan & scan is going to be much harder than I thought, I'm going to ask Kirill Gavrilov if he's planning a Rift output mode for sView, I'm hoping we can integrate SVP with that. Fingers-crossed we'll be able to start testing stuff for real very soon!

I finally got around to uploading some more test clips, a few 5 minute scenes from Smurfahontas in 60fps 3D with ambilight and different warp corrections. I have increased “screen” size / resolution to 288x512 but I think it’s probably too big now? It’s hard to tell without a Rift to test with, I’ll probably have to drop it back down to 480x270 (which happens to be exactly 25% full HD at 16:9). The ambilight bars should give you motion cues in your peripheral vision but the screen should still be “small” enough to allow you to see the whole frame as intended (head-tracked panning will give us a bit more scope here but still be limited by how the movies were shot) Note that the ambilight bars are not just a blur mask, the video is not cropped at all, they are “extra” detail generated from motion vectors in the source (look out for it when the ikran are flapping their wings)

All clips are interpolated to 60fps and should now playback at the right speed in any media player but I’d recommend something like MPC with a Direct3D / EVR sync renderer for best effect. Also remember to set the panel to native resolution (1280x800@60Hz), the media player to full screen (unscaled) and turn off any aspect ratio locks etc. I think they look pretty good but I had to do a couple of encoding passes for the ambilight and warp so “live” SVP output does look a bit cleaner, there is some bad artifacting in the large ambilight bars but I think this is an SVP bug not a bitrate issue, they are 32Mbps encodes which should be over-kill even at 60fps!

I’m not sure if the warp correction is 100% yet, the FIX110 degree ones should be closest, but again it’s hard to tell without a Rift, if any of you DIY guys want to try them out I’d be interested to see more photos, especially of the flat / unwarped SBS clips. I did some quick tests on my projector but I’m not sure how the ambilight frame will interact with the pop-out scenes on the Rifts IMAX size screen. I’m going to try adding a few more ambilight flares next but don’t want to make the frame too busy or it could be distracting. I did a few tests with inwards warp just for fun, it doesn’t seem to hurt the 3D as much as I thought, but probably best just to distort the horizontal screen plane in future, rather than the corner wrap-around effect?

60fps 3D:

https://rapidshare.com/files/4200273682/SBS2_3D.mkv
https://rapidshare.com/files/3503306078/FIX90_3D.mkv
https://rapidshare.com/files/2917471896/WARPIN70_3D.mkv

https://rapidshare.com/files/2245506381/SBS_3D.mkv
https://rapidshare.com/files/2967655910/FIX110_3D.mkv
https://rapidshare.com/files/1828680122 ... N90_3D.mkv

60fps 2D:

https://rapidshare.com/files/4136566042/SBS_2D.mkv
https://rapidshare.com/files/691599072/FIX110_2D.mkv
https://rapidshare.com/files/1588061140 ... 100_2D.mkv

I have fixed my Rapidshare account, the download links should actually work now, doh!

If you still get an error try again in a few hours, the FIX90 file might actually be best for DIY Rifts?

Nick3DvB wrote:I have fixed my Rapidshare account, the download links should actually work now, doh!

If you still get an error try again in a few hours, the FIX90 file might actually be best for DIY Rifts?

I replied to this on the wrong thread. Here's the link:

http://www.mtbs3d.com/phpBB/viewtopic.p ... 780#p88798

Nick3DvB wrote:I finally got around to uploading some more test clips, a few 5 minute scenes from Smurfahontas in 60fps 3D with ambilight and different warp corrections. I have increased “screen” size / resolution to 288x512 but I think it’s probably too big now? It’s hard to tell without a Rift to test with, I’ll probably have to drop it back down to 480x270 (which happens to be exactly 25% full HD at 16:9). The ambilight bars should give you motion cues in your peripheral vision but the screen should still be “small” enough to allow you to see the whole frame as intended (head-tracked panning will give us a bit more scope here but still be limited by how the movies were shot) Note that the ambilight bars are not just a blur mask, the video is not cropped at all, they are “extra” detail generated from motion vectors in the source (look out for it when the ikran are flapping their wings)

All clips are interpolated to 60fps and should now playback at the right speed in any media player but I’d recommend something like MPC with a Direct3D / EVR sync renderer for best effect. Also remember to set the panel to native resolution (1280x800@60Hz), the media player to full screen (unscaled) and turn off any aspect ratio locks etc. I think they look pretty good but I had to do a couple of encoding passes for the ambilight and warp so “live” SVP output does look a bit cleaner, there is some bad artifacting in the large ambilight bars but I think this is an SVP bug not a bitrate issue, they are 32Mbps encodes which should be over-kill even at 60fps!

I’m not sure if the warp correction is 100% yet, the FIX110 degree ones should be closest, but again it’s hard to tell without a Rift, if any of you DIY guys want to try them out I’d be interested to see more photos, especially of the flat / unwarped SBS clips. I did some quick tests on my projector but I’m not sure how the ambilight frame will interact with the pop-out scenes on the Rifts IMAX size screen. I’m going to try adding a few more ambilight flares next but don’t want to make the frame too busy or it could be distracting. I did a few tests with inwards warp just for fun, it doesn’t seem to hurt the 3D as much as I thought, but probably best just to distort the horizontal screen plane in future, rather than the corner wrap-around effect?

60fps 3D:

https://rapidshare.com/files/4200273682/SBS2_3D.mkv
https://rapidshare.com/files/3503306078/FIX90_3D.mkv
https://rapidshare.com/files/2917471896/WARPIN70_3D.mkv

https://rapidshare.com/files/2245506381/SBS_3D.mkv
https://rapidshare.com/files/2967655910/FIX110_3D.mkv
https://rapidshare.com/files/1828680122 ... N90_3D.mkv

60fps 2D:

https://rapidshare.com/files/4136566042/SBS_2D.mkv
https://rapidshare.com/files/691599072/FIX110_2D.mkv
https://rapidshare.com/files/1588061140 ... 100_2D.mkv

Here's a photo of FIX110_2D inside my HMD. The top edge of the picture is representative of the over-barreling. Ignore the bottom edge...it's the lens (result of my bad photography skills).
Any chance you could create another version that fills up the entire width of the screen? I assume, as someone mentioned, you're trying to stay with the max. 50 degree FOV used in theaters, but for my HMD, the full use of the available screen real estate would be preferable to any minor eye strain that the full width rendering may produce.

Hey Mel, thanks for testing these, soz about the slow reply, I'm away from home at the moment but will try and post a full width clip (360x640) with the correct warp next week.

Remember Avatar's 16:9 aspect ratio is not typical, most modern movies are 2.39:1 or so and could fit at the current scale without the side bars:



I still think that we will want to scale down a bit, at least until the head-tracking is working, or watching a 2hr+ movie could be quiet hard on your eyes, especially the 3D, which is probably more uncomfortable than usual in these clips because the warp is not properly corrected. It would be handy if you could post a screenshot of one of the un-corrected SBS clips for comparison, thanks.

Nick3DvB wrote:Hey Mel, thanks for testing these, soz about the slow reply, I'm away from home at the moment but will try and post a full width clip (360x640) with the correct warp next week.

Remember Avatar's 16:9 aspect ratio is not typical, most modern movies are 2.39:1 or so and could fit at the current scale without the side bars:



I still think that we will want to scale down a bit, at least until the head-tracking is working, or watching a 2hr+ movie could be quiet hard on your eyes, especially the 3D, which is probably more uncomfortable than usual in these clips because the warp is not properly corrected. It would be handy if you could post a screenshot of one of the un-corrected SBS clips for comparison, thanks.

Not sure what you're asking for with an 'uncorrected SBS image'. If I understand correctly, isn't that just the raw image as displayed by the player (and thus, don't you already have that)?

Sorry, not a "screenshot", a shot of the screen, through the lens again, when its showing an un-corrected SBS clip would give me a better idea of the lens' optical warp.

I have several examples of shader code all with seemingly different takes on the correction, but it's all greek to me,

my math is not up to much so a picture would be a lot more use to me lol...

Nick3DvB, someone posted a grid pattern here that shows the distortion needed.
Would that be helpful? I cant find it right now but I know its here.
The level of distortion was pretty easy to see in that picture.

Thanks, but I think it was me! I've been trying to judge it "by eye" using various images found here and screenshots taken from WarpInject etc

But I think I'm just going to have to get to grips with the maths at some point...

This is the pincushion distortion that you would see from the oculus optics if you sent a perfect grid to it.




Take this image in your software, adjust it until it looks flat. Those settings should barrel distort a flat image enough to cancel it out.

This is a photoshop approximation made by Dycus (Oculus Employee)

Yup, that's the grid I posted a few pages back, but the aspect-ratio is wrong (1:1 not 4:5) so it's either cropped or distorted.

I've been using Fraps video of a game running through WarpInject, making it rectilinear then inverting the numbers,

but I think there might be something wrong with the plugin I'm using, I'll try a different one when I get time.

Thought this was interesting. I wonder how this camera system is put together and what recording formats they use for this? Amazing potential for immersive VR movies and experiential videos.


http://thechive.com/2012/11/29/this-360 ... ind-video/

brantlew wrote:Thought this was interesting. I wonder how this camera system is put together and what recording formats they use for this? Amazing potential for immersive VR movies and experiential videos.
http://thechive.com/2012/11/29/this-360 ... ind-video/

Uhoh, very interesting indeed. Been interested in 360-videos since I bumped into the vaporware Tamaggo with the ImmerVision panomorph lens (hoping for their webcam-cube to become real), Immersive Media with their multi-camera systems, Yellowbird (ridiculous url) as well. Also the 0-360 panorama lens.

I'd guess they might be using two panomorph lenses, I can see two seams in the video. They cover 360x182 degrees, meaning a 4 degree overlap if the units are slim enough.[/d] The helmet itself is added in post to hide the mount, quite obvious as the lighting on it never changes. Sure would like to see what they are using :3

Edit: Ah, a link to the setup was in the comments. To me it looks like a bunch of cobbled up GoPro cameras, seeing the small displays below the lenses, much like the DIY-streetview (more visible on the old site). Oh, and here is the camera itself form MakingView, not many details though. Also, sorry to go off topic, I do this all the time, in normal conversation too -_-* meh. Still, I would love to experience these kinds of videos in the Rift with headtracking >_> if at all feasible unstabilized.

Nick3DvB wrote:Yup, that's the grid I posted a few pages back, but the aspect-ratio is wrong (1:1 not 4:5) so it's either cropped or distorted.

I've been using Fraps video of a game running through WarpInject, making it rectilinear then inverting the numbers,

but I think there might be something wrong with the plugin I'm using, I'll try a different one when I get time.

Here's a link to the one I did up in Photoshop. Not sure how accurate it is, but I think it should be very close.
http://www.mtbs3d.com/phpBB/viewtopic.p ... hop#p74879

Thanks alot alkapwn I have been looking for that. Much better

Yes, that's a lot better, will have another shot at this when I get a minute. Thanks.

Hi Nick,

I'm viewing through the 5x pocket loupe magnifier with Aspheric Lens. What I've found is that the pincushion distortion from these lenses is very minimal. The Doom3 BFG edition warping is even slightly over barreled for these lenses but not by much.

I took two pics, one of the non-prewapred SBS and one of the F110 SBS. As the other poster mentioned the F110 is over-barreled (for these lenses at least).

Maybe the second file below could be used with your lens correction plugin/software to try to get it square.

Thanks for the photos tmek, that's really helpful. Have you got any black borders or does the image fill the whole panel? The FIX90 file will probably be closer, but still over-corrected. Looking at the SBS clip, if we drop the screen size back down a bit then there's probably no real need to do warp correction at all, as I suspected all along, and it should be more comfortable to watch anyway. It's strange how the pixel density actually appears to increase at the edges of the screen, you'd think it would be the other way around, I assume that's just an optical illusion caused by the the additional camera lens, or maybe just jpeg compression on a moire pattern?