Rift as a passive POV video player

[Skaven252]

I've been working with a POV video project similar to the 2001 MindFrog project, but without realtime streaming. The idea is to record video with head-mounted bullet cameras and binaural ear microphones to get a strong sense of first person immersion. This should in theory produce pretty immersive "virtual tourist" footage. You won't be able to look around, but you get stereoscopy. Also, the binaural sound greatly enhances the immersion.

My earlier viewing tests were done with a standard desktop display + headphones. The probem was, that the viewers often turned their heads while viewing, which decouples the direction of the binaural sound from the picture. Also, you had to lean very close to the display to get a wide FOV. I thought that even without head tracking, the Rift would be the ideal device for viewing these videos due to its wide FOV, and the fact it keeps the picture locked in front.

My Oculus Rift devkit arrived yesterday, so I immediately proceeded to test my videos with it. I rendered new versions yesterday and uploaded them to YouTube. So, if any of you have the Rift devkit or another similar HMD, feel free to try these.

To view these, set YouTube to "Side by Side" and make it full screen, and duplicate the display to the Rift. No warping needed, the warp is baked into the video footage. Head tracking is not used or needed for these - the Rift is only used as a video viewing device. Please use headphones if you can - the binaural sound directionality only works with them. Hope you enjoy!

• Roller coaster ride (Linnanmäki, Helsinki - Head mounted camera)
• Sunny winter day in Stockholm (Handheld camera)
• Running and climbing in a park (Head-mounted camera - even very close-up objects like the monkey bars look OK on the Rift)
• Norrtull street flea market (Handheld camera - lots of ambience)
• Stockholm Marathon 2013 (Head-mounted camera. Rock band, lots of runners)
• Henrik Ehrsson out-of-the-body experiment video - as requested by blazespinnaker
• Julian Layn plays grand piano in Hötorget - YouTube (30 fps) - Internet Archive download (60 fps, 1.1 Gb)

EDIT:

• This is what the head mounted camera rig looks like (plus ear microphones that look like normal earbuds). Used in the roller coaster and monkey bar video.
• This is what the handheld "telehead" camera rig looks like (Built onto a 3Dio Free Space binaural microphone. advantage: the IPD is human). Used in the winter park video.
• Here' s the new Free Space Pro + Sony HDR-AS15 rig - an iteration of the handheld rig, with better picture and sound. Yes, those are ear warmers.

Note: Warping is not needed. The footage itself is already warped due to the fisheye lenses of the action cameras. They seem to match surprisingly well with the required Rift warp: straight lines look more or less straight.

[nateight]

This looks extremely interesting, but I should warn you: Many people feel instant discomfort when the person controlling a camera rig makes head motions they are forced to experience. A recent example of this was Cymatic Bruce viewing this video in his Rift during one of his Twitch.tv hangouts - check out time 13:30 of the archive. As he said, some videos are bearable, but the effect of the player in this video involuntarily moving Bruce's virtual view around was "garbage" and made him dive out of his Rift almost instantly. I strongly suspect mismatched vestibular cues would make some people very uncomfortable when viewing your videos, and even if you yourself are somehow immune to this, you would need to test your setup on a wide range of people to be certain it isn't inherently nauseating to most people.

I really hope I'm wrong about this and everyone can view such things without any discomfort. Please let us know what you learn about this topic, because I consider it one of the single most important unanswered questions hanging over VR development right now. What you're doing has some amazing applications, Skaven252, but only if it doesn't make most of the people viewing such video sick!

[Skaven252]

Thanks for the reply, and good point. I didn't feel any discomfort when I held my head still while watching these videos - but when I turned my head, it felt weird and wrong, as if my head was turning the world around me. It felt pretty dizzy.

So... try to keep your head still while watching. Maybe hold the Rift to your face with your hands (not using the straps), and lean your elbows on your desk to keep still.

Another possibility:
- Read the track motion from the video with something like the VirtualDub Deshaker
- Put the video on a floating virtual screen
- Orbit the virtual screen around the viewer according to the track motion

This way the viewer would have to follow the view/pan with their head to keep up with it, but maybe it feels a bit more natural that way.

But this would reintroduce the problem of decoupling the binaural sound from the picture. You can't really rotate binaural sound in post (if you do a 3D virtual pan it won't work, you'll just get a double binaural effect which sounds wrong).

[geekmaster]

This problem is alleviated greatly when you decouple head tracking from projection, like I do in my GMsphere (PixelBall) app. The perceptual effect is that the video is wrapped around you, projected a large dome (like a planetarium), but you can look around freely under your own control. This program currently displays your desktop, but many people use it to watch a video playing on your desktop.

I have my GMsphere code partially integrated into mplayer, so there is no need to read the (slow) video RAM containing your desktop. This new (unfinished) program renders directly into my PixelBall. You control your own head movement in this virtual environment. I was watching Sintel 3D (SBS-Half) in my PixelBall last night. It was cool, but I still have odd color artifacts due to center-cropping (IPD adjustment) the images while YUV encoded, that I need to resolve when I get time. Also, I only support YUV-encoded video at the moment...

For now, StephaneLX just reported that Rift head tracking was added to the VR Player project:
http://www.mtbs3d.com/phpBB/viewtopic.p ... 82#p123229

The main difference is that my GMsphere PixelBall environment preserves the raw pixels at the expense of geometric accuracy (great for text, like on a Windows desktop), while the VR Player preserves geometric accuracy at the expense of smeared (warped) pixels. Each app has a different niche, but they can both be used for watching video on "the big screen".

[Skaven252]

This problem is alleviated greatly when you decouple head tracking from projection, like I do in my GMsphere (PixelBall) app.

Certainly, but it will decouple the picture from the sound. The binaural soundtrack has strong directionality, but it can't be rotated with the picture. There aren't currently any ways to rotate a binaural recording in post. If you put the stereo channels on virtual speakers in the scene, you'll get a double pinnae effect from the HRTF, which won't sound right.

This is why I more or less insist that the picture should stay locked in front of the view. There's a lot of talk about 360 video and that's a good point as it would allow the user to look around, but unfortunately the binaural sound becomes a strumbling block here.

However, if the audio was recorde in 4-channel surround, maybe it could be interpreted (via a HRTF algorithm) to something that can be rotated with the picture.

[geekmaster]

This problem is alleviated greatly when you decouple head tracking from projection, like I do in my GMsphere (PixelBall) app.

Certainly, but it will decouple the picture from the sound. The binaural soundtrack has strong directionality, but it can't be rotated with the picture. There aren't currently any ways to rotate a binaural recording in post. If you put the stereo channels on virtual speakers in the scene, you'll get a double pinnae effect from the HRTF, which won't sound right.

This is why I more or less insist that the picture should stay locked in front of the view. There's a lot of talk about 360 video and that's a good point as it would allow the user to look around, but unfortunately the binaural sound becomes a strumbling block here.

However, if the audio was recorde in 4-channel surround, maybe it could be interpreted (via a HRTF algorithm) to something that can be rotated with the picture.

My hearing is not good, so I do not often think about audio (other than "good enough for me"). I can barely even use the telephone sometimes (symptom variability). But for those who can enjoy good hearing, you have a point. I greatly enjoyed binaural audio recordings as a child (back when stereo recordings were hard to get, and only audio geeks had stereophonic record players), and most FM stations (with receivers rare and expensive) only broadcast mono content. It was a pure joy when you could find something to make the "stereo subcarrier" neon indicator light glow. Stereo headphones were not easy to obtain (or affordable) so I built my own, just for our binaural audio recordings... But these days, meh! I would not know what to do with positional audio.

Enjoy your hearing (and your health) while you can!

[nateight]

Skaven252, I asked Bruce about this during last night's Twitch.tv hangout (~1:15:20), and he told me your videos weren't quite as bad as I feared. He did said he had to bail out during the monkey bar section, but as long as he wasn't turning his own head, everything was otherwise tolerable. The vestibular mismatch is definitely still a problem with dual cameras, but for gentle walking-around motions with little head turning, such a rig may only be trouble for people highly sensitive to simulator sickness. I'm with geekmaster, though - the recording rig that is likely to ultimately "win" is a 360 camera that allows the end viewer complete control of view rotation. Consider this thread started by rfurlan, and this 360 mountain biking video. It might still make people "feel weird" to have no control over where their avatar was moving, but I suspect the addition of low-latency head tracking to a 360 scene would greatly alleviate a large subset of simulator sickness symptoms.

Solving the binaural audio problem could indeed be tricky, and I'm not sure how portable you could ever make a 4-channel recorder, but it sounds like you at least understand the situation well enough to know what solutions to pursue. Please give us more details whenever possible - I'm utterly fascinated by the possibilities presented by real-world VR recording, and I strongly believe it will be a major component in the widespread adoption of HMD technology.

[Leon3d]

Hey Skaven252, thanks for sharing this.

I have to say, I watched all 3 and did start to feel motion sick. But watched them anyway because the 3D depth is fantastic and I loved the sound. There are moments stood on the hill with the kids sledging and man, it puts you right there.
I would love to jump into cameras positioned all over the world, concerts, tourist spots, big events and enjoy the people watching and the audio ambiance. But so long as the camera is static, or I have a little more control.

The image quality is really nice and looks great on the rift. Nice to just play a youTube movie full screen and get such great quality.

How about hiring a stunt man to act out different stunts. That would be trippy, or those crazy russian guys and girls that climb the bridges and steeples with no harnesses. That would be crazy POVs.

Very cool stuff. Another fine demo for friends to check out.

[Skaven252]

Thanks for the feedback, great answers here!

Cool, did my videos get a mention in Cymatic Bruce's hangout? But I couldn't see it in the hangout video at ~1:15:20, they were talking about Portal 2 and Museum of Microsfar there... are you sure that was the right time code?

Yes, I also noticed that if I turn my own head while viewing those videos, it feels like the world was spinning. Quite dizzy. Being able to look around would solve this, but how to do the sound...

As for 360 + stereoscopic + binaural, maybe the only way to achieve that would be a "plenoptic sphere array" camera. As in, a sphere covered with a spherical plenoptic lens array, and microphone capsules pointing in every direction. Then you could record plenoptic 360 video footage, and interpret that into a stereoscopic look-around view. And you could interpret the 4 channel surround sound to binaural sound with a HRTF algorithm, and make it rotate together with the view. The advantage of this kind of a "plenosphere" video is that you could use a single camera, and stream it to as many viewers as you want - they could all look in the direction they want. They would not be fighting over the controls of a "telehead on a servo" that only looks forward. It would also eliminate the lag issue between the servo and the viewer, as you would be looking around in the 360 video stream locally.

As for the cameras I've used (also edited to first post):

• This is what the head mounted camera rig looks like (plus ear microphones that look like normal earbuds). Used in the roller coaster and monkey bar video.
• This is what the handheld "telehead" camera rig looks like (Built onto a 3Dio Free Space binaural microphone. Advantage: the IPD is human). Used in the winter park video.

As you can see, the head mount has much wider than human IPD, which gets distracting up close (the handheld rig solves this problem, but then you lose the "body experience"). Ideally the cameras would be transparent and right in front of your eyes (or maybe right above them, so you could see past them), but such cameras don't quite yet exist as a commercial product. There's this flat button camera, a pair of which could maybe be mounted on goggles above the eyes, but it has a somewhat narrow FOV and poor image quality.

[Skaven252]

How about hiring a stunt man to act out different stunts. That would be trippy

Knowing how the world usually operates, I reckon the first people to try this in larger scale will be those who produce, um, adult entertainment.

[geekmaster]

As you can see, the head mount has much wider than human IPD, which gets distracting up close (the handheld rig solves this problem, but then you lose the "body experience"). Ideally the cameras would be transparent and right in front of your eyes (or maybe right above them, so you could see past them), but such cameras don't quite yet exist as a commercial product. There's this flat button camera, a pair of which could maybe be mounted on goggles above the eyes, but it has a somewhat narrow FOV and poor image quality.

What you describe is where the "EyeTap" camera is evolving:
http://en.wikipedia.org/wiki/EyeTap

[Skaven252]

What you describe is where the "EyeTap" camera is evolving:
http://en.wikipedia.org/wiki/EyeTap

Cool, this looks pretty advanced! (I didn'tknow about this device) Now all it needs is stereoscopy (how to get two CMOS camera sensors in sync?), and binaural sound.

Or maybe you could cycloptically divert the two images side by side, in anamorphic 3D, onto one sensor. Then you wouldn't have the sync issue.

[Skaven252]

New video available! I used the handheld camera again. There was a flea market in a nearby street and I shot some footage with lots of ambience and crowds.

YouTube: Norrtull street flea market

As with the other videos, play it in full screen, side by side, on the Rift.

Instead of strapping the Rift to your head, try just holding it to your face with your hands, and leaning your elbows on your desk or armrests. This keeps the head still and may reduce nausea a bit, if you have that problem.

[BOLL]

Nice work Skaven My kit has not arrived yet, but I will definitely check out your videos again when I have got it!

As for skewing the images, is that to simulate skewed frustums? Wouldn't the amount of skewing depend on your IPD? Not sure if I got that right from mr Doc-Ok's article(s) but I wonder...

[Skaven252]

As for skewing the images, is that to simulate skewed frustums?

Frankly, I have no idea. It just felt more natural that way, when previewed on the Rift. It's the "Keystone" correction in the Vegas stereoscopic adjust plugin.

Could be I'm still doing it wrong, because some people complain about cross-eyedness and dual image.

[nateight]

I couldn't see it in the hangout video at ~1:15:20, they were talking about Portal 2 and Museum of Microsfar there... are you sure that was the right time code?

Sorry, try 1:51:20. I'm not even dsylexic, I swaer.

[Skaven252]

Sorry, try 1:51:20. I'm not even dsylexic, I swaer.

Found it! Cool, I didn't know someone posted my video on Reddit.

[geekmaster]

... The idea is to record video with head-mounted bullet cameras and binaural ear microphones to get a strong sense of first person immersion. This should in theory produce pretty immersive "virtual tourist" footage. You won't be able to look around, but you get stereoscopy. Also, the binaural sound greatly enhances the immersion. ...

... Solving the binaural audio problem could indeed be tricky...

For full (claimed) 360-degree binaural recording, check out the alien-looking "Beck Binaural Head":
http://www.legacyefx.com/blog/beck-binaural-head/

http://www.youtube.com/watch?v=AjJJ0cA2gqc

... Apparently this is top-secret, sensitive technology...

EDIT: The uploader has not made this video available in your country.

That video is also available on vevo, but only YouTube can be embedded here (AFAIK). Here is a vevo link:
http://www.vevo.com/watch/beck/say-hell ... VEV1394170

Same result:

Sorry! The page you are looking for is not available in your region.

Bummer... Why do they do that?

For those who are not allowed to watch that cool "behind the scenes" video, here is a picture of the head:

[Skaven252]

^ Thanks! I've been emailing with the guy who consulted the ones who made that head, and who makes the microphones I've used for my handheld 'telehead' camera.

It records 8 channels (4 stereo tracks), and the rotation is done by simply crossfading between the tracks. He said it sounds "convincing enough" even though it's not pure binaural rotation, just blending between 4 directions.

Which reminds me... I have a Zoom H-2 recorder that can record in 4 channel surround. I should try how that sounds in my HeaDSpeaker system, if I turn my head around. Could give me an idea how a "plenosphere" recording could sound like.

[Marulu]

EDIT: The uploader has not made this video available in your country.

Same result: Sorry! The page you are looking for is not available in your region.

I know how you guys can watch this video in your region, you can watch it using ProxTube.

There is a free Mozzila Firefox and Google Chrome plugin/addon available called ProxTube which allows you to watch videos on YouTube when they are blocked in your country. This plugin uses a high speed proxy which can load HD videos in a couple of seconds. You don't have to fear that this makes your regular YouTube browsing slower, because it only activates itself only if the video you want to watch is not available in your region. I hope this allows you to watch this video.

[Mark2036]

this site appears to have 3d panoramas (stills only).

http://www.3d-360.com/

The panoramas are output in side by side and are viewed in the browser with mouse used to pan. It would be great if they could use whatever camera setup used to capture these and film panoramic 3d videos, then riftify them with head tracking.

[Skaven252]

A new video is online! I watched the Stockholm Marathon 2013 pass by, and shot some footage with my head mounted camera rig. Features binaural sound, of course!

On YouTube: Stockholm Marathon 2013

[blazespinnaker]

The videos are fine expect when the viewer turns the cameras in odd ways.

When my brain can predict what's going to be viewed next, I have no problem. It's only jarring when an unexpected movement or head turn occurs.

For example, the roller coaster was fine except when the camera holder turned their head to check out the view.

I found myself turning my head to try to keep up with what they're doing, that helped somewhat. But I had to be quick.

[Skaven252]

You could also try the VRPlayer (with the full HD desktop 3D display version of the videos). It places the video on a virtual screen and you can look around, which may make it an easier experience.

I've gotten them to work pretty well with Plane projection, ratio 1.777, FOV 110, Horizontal offset -160 (for the non-Rift aligned videos), no distortion.

[blazespinnaker]

Hey Skaven252,

Check this out:

http://www.nature.com/news/out-of-body- ... ion-1.9569

"A year after removing his subjects from their own bodies, Ehrsson learned how to trick them into acquiring new ones. This time, the volunteers' goggles showed them the view from a camera on the head of a mannequin looking at its own plastic torso. Simultaneously poking the arm or stomach of the mannequin and the volunteer a few times was enough to convince the subjects that they were the dummy. They could even stare at their old bodies from their new ones and shake hands with their old self, all without breaking the spell1. “It really is very intense and incredibly fast,” says Mark Hallett, a neurologist from the National Institutes of Health in Bethesda, Maryland, who experienced it first hand."

Do you think you could do a you tube video where you're looking down at yourself while sitting in a chair and then tapping on your chest with your fist? Maybe do it in a reasonable tap - tap - tap rhythm so someone can follow along?

You could also try putting the gear on a big doll or stuffed teddy bear if you have one. The key would be to have it looking down at itself.

Let me know what you think!

Cheers,

Blaze.

[Skaven252]

Seems like an interesting experiment! Should the video only contain the tapping at a steady rhythm and no head movement, so that the viewers can replicate it, or should it be followed by getting up and looking around?

I don't have any mannequin torsos or teddy bears alas, but I live quite close to Karolinska (Solna). Maybe I could meet with the guy, even though I'm not a qualified neurologist.

[blazespinnaker]

I think the problem with getting up and moving around is that you might break the illusion because of the loss of control.

Once we get proper 360 degree cameras working, doing the substitution reality experiments should be interesting.

http://www.diginfo.tv/v/12-0197-n-en.php

[Skaven252]

I'm also a bit worried that the over-wide IPD of my head worn setup causes problems up close (too much divergence) - and tapping my own chest with my hand is exactly that. The handheld human IPD camera would work better, only that I don't have a mannequin torso, and can't decapitate myself to correctly place the camera.

Flat button gameras, worn like eyeglasses above the eyes would give human IPD and still be wearable - but I haven't been able to find suitable cameras for that. The closest one I could find was this wide angle button camera, but it's not built for the purpose and is not wide angle enough.

[Skaven252]

I figured a way to get my Sixzero cameras to a head mount rig:



Compared to the VIOs, these have better battery life, and they're self contained (the VIOs were tethered to heavy recording units). They also have rock solid famerate and emit a loud beep when started, which makes syncing the footage easier.

Pressed against the temples, the cameras converge (point cross-eyedly towards the center) quite a lot. I need to chain four (!) Stereoscopic Adjust plugins to get the convergence comfortable for the Rift, and there's only overlap in the center.

Do you think you could do a you tube video where you're looking down at yourself while sitting in a chair and then tapping on your chest with your fist? Maybe do it in a reasonable tap - tap - tap rhythm so someone can follow along?

Just tried recording that today, with this rig. I'll render and post it soon. I tried it myself, tapping my own chest, but didn't get a strong illusion... yet. Maybe it has to be realtime to work. But I'll post the video so those interested can try and see if it works for you.

[Skaven252]

Do you think you could do a you tube video where you're looking down at yourself while sitting in a chair and then tapping on your chest with your fist? Maybe do it in a reasonable tap - tap - tap rhythm so someone can follow along?

Here's the video! I edited it to repeat 3 times so you can try getting into sync with it.

http://youtu.be/f37fxwzZjRA

[strangedays]

Is there anyway to film the 3d and have decoupled tracking by a device.... bear with me here.

its two eyeballs (cameras filming in every direction) its shoots a frame and drops 5 inches, and the other one bounces up 5 inches and shoots a frame, and they just keep going back and forth. each bouncing up to take a shot so the other isn't in the view.

no that wont work ...

head turn

how about if the globes rotate in a halo, taking 360 frames per second, one for each degree, and the headtracking chooses the correct 2 frames to display?

yeah that should do it ....

AHHH F IT! Tilt!

[Skaven252]

AHHH F IT! Tilt!

I've been thinking of a plenoptic sphere array. With surround microphone capsules pointing to every axis. Pretty sci fi, but maybe it's possible. It would be 360, stereoscopic and re-focusable (to get look-at eye tracking too, when that's technically possible).

Maybe a bouquet of optic fibers that converge onto an image sensor?

[strangedays]

bro. i have no idea what you just said, but if you make it, I'll buy it

Then I'll film awesome experiences and provide my local hospital with rifts and let the sick kids leave the misery of the hospital in their heads for a while....

and THEN i'll go home and watch porn filmed on it

[blazespinnaker]

I felt it a bit. I found making sure nothing was on my lap and making sure my position was the same as yours helped (and laying my arm over my leg). It's a bit hard because the FOV isn't all that great at the top/bottom empty spaces.

What's more interesting actually, is that when I try to 'remember' some of the other videos I watched that you filmed (say, the next day or the day after), the memory of them felt like it was actually me that was there. It was also impressive how well I remembered them. In particularly, walking through the city flea market.

That probably seems peculiar, but I think what's happening is my brain is registering those POV movies as something I am doing rather than just something I watched.

Not sure if anyone is registering them that way. Maybe it's just my imagination.

Anyways, can't wait to see what else you do with the POV stuff. I think a huge market for it might be virtual tourism. If there is any interesting landmarks you have in your area, might be worthwhile visiting and videoing. My only suggestion is remember that you're like a film maker - try to observe things in an interesting way that people will want to watch

[Skaven252]

Thanks for watching, and thanks for the feedback, blazespinnaker!

I think one part of the problem with my POV stuff is that the cameras are too far apart (on the temples), which makes the IPD wrong, and hard to watch especially close-up stuff. That's why the handheld camera, which has human IPD, is likely easier and thus more immersive to watch.

How closely the video FOV is matched with the natural human FOV can also be a factor. Not sure how to get this right, so it neither feels like "I'm a fish" nor "I'm viewing the world through binoculars".

Sound may be an important part of it. The handheld camera, which was used in the flea market clip, uses the 3Dio Free Space microphone, which likely provides a much more universally compatible binaural sound model than the ear-worn microphones I use in my POV footage. Ear microhpone audio is likely mostly compatible with people who have a similar head/pinnae shape to mine (ie it's less universal).

If there were wide angle, good quality cameras I could mount above my eyes (and some kind of a frame I could use for mounting), I could try doing POV with human IPD.

Some kind of a prismatic wave guide (from temples to front of eyes, like EyeTap) could also be an option, but I lack the engineering skills and facilities to build something like that. Also, I reckon it's hard to do that with wide angle cameras. And you need the wide angle fisheye to cover as much of the FOV as possible.

[aikakana]

About head tracking & binaural sound problem. Could you track the head for just a bit, making the image move say one degree at maximum? This would eliminate sickness from small unconscious movements and possibly even look okay with larger movements. The screen would drag along with your movements if they exceed the 1 degree limit. What do you think?

Edit: additionally, the screen could be like a spring that re-centers as you stop. Adjusting the spring effect would affect the limit the screen moves so there would be no hardcoded 1 degree limit but it would look soft.

[Skaven252]

About head tracking & binaural sound problem. Could you track the head for just a bit, making the image move say one degree at maximum?

You can try and see if this works with VRPlayer - it supports the Rift head tracking, and allows you to look around, while the video plays back on a virtual screen.

Wild idea:
- perform an analysis pass on the video with VirtualDub DeShaker (this creates a vector data file)
- Cleverly stream this data to VRPlayer so that the virtual screen shakes accordingly to compensate. You'll see the frame waggle around, but the picture stays more still.

[3dpmaster]

I've been working with a POV video project similar to the 2001 MindFrog project, but without realtime streaming. The idea is to record video with head-mounted bullet cameras and binaural ear microphones to get a strong sense of first person immersion. This should in theory produce pretty immersive "virtual tourist" footage. You won't be able to look around, but you get stereoscopy. Also, the binaural sound greatly enhances the immersion.

My earlier viewing tests were done with a standard desktop display + headphones. The probem was, that the viewers often turned their heads while viewing, which decouples the direction of the binaural sound from the picture. Also, you had to lean very close to the display to get a wide FOV. I thought that even without head tracking, the Rift would be the ideal device for viewing these videos due to its wide FOV, and the fact it keeps the picture locked in front.

My Oculus Rift devkit arrived yesterday, so I immediately proceeded to test my videos with it. I rendered new versions yesterday and uploaded them to YouTube. So, if any of you have the Rift devkit or another similar HMD, feel free to try these.

To view these, set YouTube to "Side by Side" and make it full screen, and duplicate the display to the Rift. No warping needed, the warp is baked into the video footage. Head tracking is not used or needed for these - the Rift is only used as a video viewing device. Please use headphones if you can - the binaural sound directionality only works with them. Hope you enjoy!

• Roller coaster ride (Linnanmäki, Helsinki - Head mounted camera)
• Sunny winter day in Stockholm (Handheld camera)
• Running and climbing in a park (Head-mounted camera - even very close-up objects like the monkey bars look OK on the Rift)
• Norrtull street flea market (Handheld camera - lots of ambience)
• Stockholm Marathon 2013 (Head-mounted camera. Rock band, lots of runners)
• Henrik Ehrsson out-of-the-body experiment video - as requested by blazespinnaker

EDIT:

• This is what the head mounted camera rig looks like (plus ear microphones that look like normal earbuds). Used in the roller coaster and monkey bar video.
• This is what the handheld "telehead" camera rig looks like (Built onto a 3Dio Free Space binaural microphone. advantage: the IPD is human). Used in the winter park video.

Note: Warping is not needed. The footage itself is already warped due to the fisheye lenses of the action cameras. They seem to match surprisingly well with the required Rift warp: straight lines look more or less straight.

The sound quality is outstanding!

[Skaven252]

Glad you like it, sir!

I recently got a pair of new Sony HDR-AS15 action cameras. With their newest 3.0 firmware update, they can shoot full HD 1080p at 60 fps! They also have super wide angle lenses (170 deg), great low light performance thanks to its backlit sensor, sharpest lens on the market and built in image stabilization that works really well - however after testing it turned out I should not use it for 3D because the images may occasionally diverge out of alignment due to the stabilization. Also, the stabilizer needs to crop the image, so the FOV drops to 120 degrees.

This is what the rig looks like. The cameras had to be mounted upside down, because of their form factor, and because the microphone input is in the bottom of the camera. The HDR-AS15 actually has surprisingly good, clean onboard audio, both with its built in microphones and with external microphones. But it compresses the dynamics quite heavily, so I still prefer recording the audio with my Zoom H1.

I've already done some short test shoots at 60 fps, an it's just uncannily smooth. Definitely adds to the immersion, and to the feeling of being there.

YouTube doesn't support 60 fps though (except through a hacky HTML5 workaround), so I guess these will be for future distribution then....

[geekmaster]

... however after testing it turned out I should not use it for 3D because the images may occasionally diverge out of alignment due to the stabilization. Also, the stabilizer needs to crop the image, so the FOV drops to 120 degrees.

Perhaps you can film 3D with image stabilization OFF, then dewarp them and pack them into an SBS pair, then use the VirtualDub Deshaker plugin to stabilize them together in post processing. Then you can split them into L+R streams and rewarp them.

The key is that both images must use the same stabilization motion vectors. Another method may be to use Deshaker to extract motion vectors from ONE video (perhaps left) then use those motion vectors to deshake BOTH videos on the second Deshaker pass. I have not used Deshaker for stereoscopic video yet, but the methods I suggested sound like they should work. Again, it may provide better results to dewarp the images (no fisheye) before deshaking. You can reapply Rift lens prewarp (fisheye) again after deshaking.

Of course, post-processing image stabilization will add some motion blur, but the only way around that is if the cameras can have their stabilization motion-vectors "genlocked" between the two cameras. Without custom camera firmware, that may not be an option.