It is currently Sat Dec 07, 2019 7:57 pm



 [ 49 posts ]  Go to page 1, 2  Next
 Enhanced immersion 3D POV video concept 
Author Message
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Greetings.

For a while already I've been working on and experimenting with a way to record POV video that would provide heightened immersion when watched. In a nutshell, the idea is this:

  • Head-mount the cameras as close to the eyes as possible
  • Record the sound using binaural ear microphones

Because both the image and the sound stay locked together and both are head relative, their combined effect provides an enhanced sense of being there.

Cases in point (YouTube; use headphones to get the full effect):


Ideally these would be viewed on super wide angle video goggles with headphones. And this makes the Oculus Rift very interesting for this. Even without head tracking, just the FOV-filling view should work very well with these videos. In theory. I'd like to try and see how it works in practice, so I decided to back the kickstarter. I'm not much of a programmer, but I trust the developer community will sooner or later come up with the means to support video on the Rift.

In the meantime, hope you find those videos interesting. :)


Last edited by Skaven252 on Thu Aug 09, 2012 6:08 am, edited 3 times in total.



Thu Aug 02, 2012 11:47 am
Cross Eyed!

Joined: Wed Sep 22, 2010 3:47 am
Posts: 105
Location: Toulouse, France
Hello,

Is there a way you could upload the videos somewhere I can download them to try on my HMD with bino3D ?

Thanks

_________________
InfinitEye - 210° HFOV HMD


Thu Aug 02, 2012 1:00 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
If you're using Chrome, you can use an add-on called "YouTube Downloader" to download them as side-by-side .mp4 files.

If that doesn't work out, we'll figure out something else. :) Any suggestions for a site that would allow large files like these to be shared without hassle or hitting a limit?


Thu Aug 02, 2012 1:02 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Very cool. Although video might be slightly warped on the Rift, I still think it would look pretty cool.

_________________
check my blog - cybereality.com


Thu Aug 02, 2012 10:08 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Thanks for the feedback!

How did you find the directionality of the sound? Binaural sound is quite individual. It relies on the shape of the listener's head and ears, and if the recording person's head and ear shapes differ too much from the listener's, the directionality may not work. But binaural sound is a (if not the) key element in this video concept.

Does the video actually need to be spherically fisheye warped to look correct on the Rift? Because, in fact, untreated footage from those cameras (VIO POV.HD) is fisheye curved. I treated those youtube videos with AviSynth -> Defish to make them rectilinear. If curved footage actually looks more correct on the Rift, it's a win-win. I get a wider field of view in the videos themselves (the rectilinear correction loses some), and the intermediate slow processing step is not required. :)

I should try and provide you with a video that has not been defished to try on your HMDs. Would you like to test that?

Overall, videos like these would be an interesting application for Rift and other super wide angle HMDs, not to mention, a camera to shoot this kind of "first person" footage in one pass would be a good accessory. My setup is a bit complicated: 2 bullet action cameras and their recording units, one audio recorder and its ear mikes, all worn in a chest harness. The footage then needs to be combined and synced in post.

Imagine a wearable stereoscopic POV camera with the bino mikes that you could put on easily. A "rich experience recorder", if you will, which would record everything in perfect sync, in one pass, into one side-by-side file. Should be technically possible, no? Then the Rift would be the device to watching those shared experiences with, in total immersion. I'd back a product like that on Kickstarter.


Thu Aug 02, 2012 10:31 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Love the videos, the roller coaster looked great on my projector, can't wait to see it on the RIFT! I'm not a programmer myself and better men than me have all that rendering stuff covered so I'm going to start working on some video ideas instead. btw, did you notice John Carmack's thread on video here: viewtopic.php?f=138&t=15047

It would certainly be very interesting to test the un-corrected videos, they could be a more natural fit anyway. I have used avisynth for de-anaglyph before and was looking at Defish just the other day, it looks pretty versatile and could be very useful for Rift videos, so I'm really glad we don't have to start from scratch. I have not tested it yet though, how fast is it? Maybe we could get it working in DGdecNV (CUDA) for real-time conversion? I'm a member at Doom9 and David Horman is still very active over there so I'm sure he would be glad to help out.

The sound stuff is very cool to, BBC R&D are doing some work on Binaural /Ambisonics at the moment, they have a few binaural format podcasts to. I'm not a camera guy myself but maybe the GoPro 3D kit would be worth looking at? Initially I am planning to work on getting standard 3D video watchable on the RIFT, hopefully with tracking integration at some stage. Keep us updated on your project, and more videos please! :)


Fri Aug 03, 2012 10:32 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Thanks for the tip on the video thread, I might post something there as well.

All right! I'll render an un-defished version of the roller coaster video for starters, let's see how it works with Rift or another super-wide HMD. I'll post and link it here when it's done.

Defish is actually a pretty heavy plugin - far from realtime even on my i7 (maybe 5 fps or so for full HD video). But it's the only plugin I could find that actually does the defishing correctly, with a third degree polynome (or something). All other lens correctors I tried left some odd warpage behind, no matter how I set the parameters. I actually sent David Horman an email some months ago, asking if he'd be willing to port Defish to Sony Vegas Movie Studio... but there was no response. I guess my request was outrageous. :o

I rejected the GoPro 3D kit for two reasons: 1) the IOD is kinda small, and 2) due to the configuration, the only way to mount it "as close to your eyes as possible" is to put it right in front of your eyes. And then you won't see where you're going. I use bullet cameras mounted on the temples (right outside the eyes) instead. I can see what I'm doing and it's fairly discreet for a weird camera head contraption. The IOD is wider than human, but it gives a nicely strong stereoscopic effect (albeit disturbingly double-imaged up close). He disagrees.

Welp, I'm off to render a fisheye version of the roller coaster edit now.


Fri Aug 03, 2012 1:00 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
All right, here it is!


At the time of posting this message it had just uploaded and is still processing - but it should be live shortly!


Fri Aug 03, 2012 1:54 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Cool, thanks for doing this. I don't think anyone here aside from Palmer actually has a Rift yet but this will be very cool when we do.

But based on the specs I'm working with, the warping looks like it would be perfect on the Rift.

_________________
check my blog - cybereality.com


Fri Aug 03, 2012 7:07 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
It's interesting actually. VIO claim their POV.HD camera has 142° FOV in full HD. However, Defish gave rectilinear results when its FOV parameter was set to 102°. And the Rift FOV is about 100° isn't it?

Yeah, I can hardly contain myself to wait to see these on the Rift. Whatever hurdles there are in the way to get video playback to work. :)


Fri Aug 03, 2012 7:17 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Thanks for the upload. Getting the Rift to display standard 3D content "comfortably" should be pretty do-able with avisynth or an MPC shader, mapping the controls of a mediaplayer to integrate tracking for zoom / pan & scan etc should also be pretty straight forward, working out the best "native" format to shoot in for the rift is the real challenge here, but you have made a great start. I'm also really interested to see if we can use a Kinect or Lytro camera based system to achieve sudo-holographic effects, to try and emulate a light-field viewing experience in some way, if we can simulate "peek-around" parallax for video (or even static image scenes) that would be mind blowing:

http://www.youtube.com/watch?v=4OBIN6RZUd0

http://www.3dfocus.co.uk/glasses-free-3 ... 3d-tv/8626

http://www.3dfocus.co.uk/3d-news-2/3d-t ... ature/6695


Sat Aug 04, 2012 10:03 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Light field / plenoptic / array cameras are fascinating indeed! Has anyone shot any plenoptic video yet, though? All I've seen are single images.

Imagine a plenoptic video combined with eye tracking. The video would actually focus, in realtime, on whatever you're looking at. Wouldn't that be something. Especially if you could shoot that in a 360 panorama. Then you could look around with your head AND focus with your eyes. I wonder what kind of a microphone array would work with that... perhaps the a four-microphone DirAC array (that's a new universal, rotateable surround sound technology, not to be confused with DIRAC) combined with a HeaDSPeaker head tracking surround.

I have seen some "ball cameras" that have touched the subject, but the thing requires astounding processing power and internal data rate so it's still a bit SciFi at the moment.

Anyhow, as you can see, total immersion is possible, but it requires that quite a few separate and expensive technologies collide. This "tech collision" has already happened with cell phones - just look at how many sensors and devices are in a basic smartphone nowadays. I can't see why it couldn't happen with VR/AR too.


Sun Aug 05, 2012 2:33 am
Certif-Eyed!

Joined: Tue Jan 19, 2010 6:38 pm
Posts: 529
Unless I have missed something it seems to me that the fisheye roller coaster vid won't look right. On the youtube vid, if I choose full width the aspect ratio is correct, but this will mess up the fisheye warping. If I choose half width, the fisheye is probably closer to being right, but now everything is super skinny.

_________________
"If you have a diabolical mind, the first thing that probably came to mind is that it will make an excellent trap: how do you get off a functional omni-directional treadmill?"


Wed Aug 08, 2012 1:07 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Thanks for the feedback! In what way is the fisheye warping incorrect? It's the raw, unprocessed fisheye effect straight from the VIO POV.HD cameras. Is the aspect of the fisheye warp itself not right?

The downloaded YouTube video is in side-by-side half width format, so in Bino the source/input format needs to be set to that, then the output to something else (i use red-cyan anaglyph myself).

Did you actually test this on a stereoscopic HMD, or is this based on comparison with the shots from the Rift? It may actually be a matter of cropping the video, rather than changing the aspect.


Wed Aug 08, 2012 2:58 pm
Certif-Eyed!

Joined: Tue Jan 19, 2010 6:38 pm
Posts: 529
The Rift displays video at 8:9. I suspect you were filming at 16:9. I know very little about such things really, so I can't answer your questions. I belive if you mess about with the aspect ratio, the fisheye will be wrong. (Yes 8:9 is right, the view is taller than it is wide.)

I did not test this with a HMD. I might be wrong. I just wanted to point out a potential problem.

I really like the idea of what you are doing. I think you could make Rift compatibe video by turning your cameras 90 degrees.

_________________
"If you have a diabolical mind, the first thing that probably came to mind is that it will make an excellent trap: how do you get off a functional omni-directional treadmill?"


Wed Aug 08, 2012 4:37 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Great, thanks for the feedback. Yeah.. either turn the cameras 90 degrees, or just crop the videos in half horizontally. The former would utilize maximum resolution from the cameras but will require a post processing pass to rotate and combine the videos. AVISynth should be handy for this.

Or: play the whole 16:9 video, but display only a cropped portion of it, and use the Rift head tracking to allow the viewer to look around sideways. The caveat is that this will decouple the binaural sound from the picture (the head rotates, but the sound doesn't). Also I am not sure how the fisheye would work then, because the warping is centered on the middle of the full picture and won't move with the head pan. It may be needed to convert the video to rectilinear, and do the warping of the cropped portion in software.

I'm looking forward to the release of the devkits, so that people can start working on their solutions. :)


Wed Aug 08, 2012 11:43 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Now I think about it, the difference in aspect ratio is critical, and means we are going to have to perform multiple corrections anyway, but this should allow us crop/zoom to leave some room for panning around the shot and still have it look ok. My initial thought was just to put Rift optics in front of the cameras, that might be the key to making sure the video is Orthostereoscopically correct, I'm still reading up on that but it seems to be pretty fundamental to a natural experience, there are some good links here:

http://nzphoto.tripod.com/3d/310orthostereo.htm
http://www.leepvr.com/spie1990.php
http://www.cyclopital3d.com/The_Ortho-s ... ealism.pdf
http://www.vrtifacts.com/hmds/leep-on-the-cheap/

For standard 3D movie content I think we are always going to have to push the image back onto a virtual screen (but not necessarily a flat screen) then use tracking for panning, it might be more natural just to create a "virtual cinema" (as that is what the video was created for), but we could try and do something in software to fill the remaining FOV and remove the hard edges of the screen, a bit like Philips Ambilight for TVs.

We are limited in what we can do at the moment, in developing both solutions there is no substitute experimentation - actually seeing it with your own eyes! 8-)


Thu Aug 09, 2012 9:01 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
I'm trying a "handheld" (telehead) approach to the issue now. Just for added versatility, and also to get the IPD right. When the cameras are on the temples, the IPD is too wide and it causes a nasty double image up close.

There's a new interesting binaural microphone available called the 3Dio Free Space, and I mounted my VIOs on that. Here's the result:

Image


Wed Oct 24, 2012 3:03 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Mon Aug 06, 2012 9:26 pm
Posts: 295
Location: Sweden
That rig looks mighty interesting! Just listened to the sample video on the 3dio site, sounds awesome. Looking forward to see what you will produce :)


Wed Oct 24, 2012 4:50 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
That rig looks epic!!! Nice find on the binaural mic.

_________________
check my blog - cybereality.com


Wed Oct 24, 2012 7:14 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Thanks guys! I look forward to having enough time and daylight to shoot some interesting video too. :) (Scandinavian winters are very dark and my workdays are long... :( ).

Stuff recorded with this rig will probably not sound very different from my earlier head-mounted binaural+3D recordings, but it will look different. My problem earlier was the too wide IPD, which made a distracting double image up close. These should be more pleasant to watch. Though, they'll probably be more shaky too, until I get myself a steadicam handle...


Wed Oct 24, 2012 10:56 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Here's a quick first test shoot I took yesterday and edited last night:
http://www.youtube.com/watch?v=8hf-gd1td-U

Sound recorded with the 3Dio Free Space -> Zoom H-2. The two VIO POV.HDs were mounted on top, like you see in the picture. The mic also has wind muffs, which were on. It was a fairly windy day, and the muffs handle it really well.

I'm going to shoot more footage this weekend.


Sat Oct 27, 2012 1:42 am
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
@Skaven252: Nice one!

_________________
check my blog - cybereality.com


Sat Oct 27, 2012 2:35 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Here's a new video I shot today with the 2 x VIO + 3Dio Free Space rig. Held the camera high up on a monopod for a "just a rather very tall person" effect:
http://www.youtube.com/watch?v=g3f1Ae8gdkI

The tall pole reduces shakes, but not rotation, so there's plenty of side pan jitter. Sorry about that. I guess I should add a horizontal weight bar as well to reduce rotational shaking....


Sun Oct 28, 2012 3:00 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Ok, cool.

Next up you should film some pretty Swedish ladies.

_________________
check my blog - cybereality.com


Sun Oct 28, 2012 3:13 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
I'll have to do that in the summer. :]

Anyhow, here's my latest configuration:
http://imgur.com/a/TBMWR

I'm using two L-shaped camera mount bars I got from DealExtreme. The cameras are now very rigidly held together with a dual pipe holder (a plumbing accessory from a local hardware store). That's the advantage: they stay rigidly aligned. The caveat is that now the IPD is only 4 cm, so further away there's no stereopsis. It could be good for closeups though.

The vertical bar behind the setup also doubles as a handle - but with that alone it's wobbly. I can use this with a shotgun shoulder mount. I found my old Yukon telescope mount that works both as a desk tripod and as a shotgun shoulder support.


Sat Nov 03, 2012 10:50 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Intermission. Old news, but related. There was an academic project called the "Mindfrog" which experimented with this in 2001 - with the exception that their goal was to stream the audio and video, in realtime, from a POV+Binaural camera to another person't HMD over the net.

http://www.mindfrog.net/new/

I can imagine the technical limitations were more strict back then than they are now. I can't see why this couldn't be pursued again, with more bandwidth, higher fidelity camera sensors and HMDs.


Sun Nov 11, 2012 7:38 am
Binocular Vision CONFIRMED!
User avatar

Joined: Mon Aug 06, 2012 9:26 pm
Posts: 295
Location: Sweden
That is very interesting Skaven252! I really like the description of Flow. I'm not installing the Real Media plugin, long since I got sick of that crap :D might be better now but I don't care.

Here is the video as mp4, if the link is still live: http://dw4.convertfiles.com/files/01804 ... erview.mp4
If not, use the site to convert this url: http://www.mindfrog.net/new/movies/overview.rm


Sun Nov 11, 2012 3:37 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Great to see you are making progress with the POV project, but whilst we are having an intermission... :)

I just stumbled across what could be a great video player for standard & 3D content on the Rift, details here:

viewtopic.php?f=138&t=15047&p=86825#p86825 (moved)

;)


Last edited by Nick3DvB on Wed Nov 14, 2012 2:49 am, edited 1 time in total.



Tue Nov 13, 2012 4:15 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
BOLL wrote:
That is very interesting Skaven252! I really like the description of Flow. I'm not installing the Real Media plugin, long since I got sick of that crap :D might be better now but I don't care.

Here is the video as mp4, if the link is still live: http://dw4.convertfiles.com/files/01804 ... erview.mp4
If not, use the site to convert this url: http://www.mindfrog.net/new/movies/overview.rm

Thanks Boll! I tried the link today, but it wasn't there any more - nor the site worked any more - but thank goodness I already downloaded the overview video the day you posted this link. I have a copy of it, but haven't yet gotten a moment of peace and attentiveness to give it a thorough watch.

Streaming POV+binaural in real time from one person to another would have lots of cool uses. You could have "virtual experience surrogates" who travel to concerts, exotic places, etc and others could experience it as if they were there themselves. It could work two-way up to an extent, at least you could relay the spectators' speech to the camera user so they could communicate with each other.

It could turn into a "no, don't look that way, look this way! No, don't stop looking at that yet, I wanted to look at it longer!" nag-fest however... :)

If it was a remote controllable robot telepresence head with more than one people watching, I guess you could use "mouse voting" and get a democratic average of where people want to look at. Just to ensure that absolutely no-one is pleased. Chortle.


Wed Nov 14, 2012 2:21 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Nick3DvB, that AmbiLight solution looks interesting in a sense it could be used to extend the video's field of view, in a way. To give a blurry peripheral vision to the outskirts of the viewer's field of view. This would be helpful if the video was shot with a narrower field-of-view stereo camera (which means that the video alone is not enough to cover the viewer's FOV).

However, what it says in that quote: "For standard 3D movie content I think we are always going to have to push the image back onto a virtual screen (but not necessarily a flat screen) then use tracking for panning", it kind of defeats the purpose of this immersion POV video concept. This kind of video must remain locked to the head, use the full FOV, and no head tracking to crop/pan the picture should be used. This is because the sound is locked to the head too, and I have no idea how binaural sound could be rotated in sync with the picture. Allowing the viewer to pan&scan around while the sound does not, would decouple from each other.

The idea is that the POV video covers the entire field of view, and the viewer uses their eyes to look at different parts of the picture. The viewer will be at the mercy of the camera user's whims what comes to head movement. Yes, Oculus Rift has awesome head tracking, but unfortunately this kinf of video can't really make use of it. :(

Quote:
we just need to integrate the de-fish avisynth script

The de-fishing step is an intermediate step that makes the video rectilinear (just like 3D video game graphics usually are). This can be done when the video is being combined and edited.

What would be needed, is the realtime "Carmack Warp" applied over this rectilinear side-by-side FOV video, to make it compatible with Oculus Rift. I haven't looked into the specifics, but I got the impression the "Carmack Warp" is a software equivalent of the LEEP compression.


Wed Nov 14, 2012 3:32 am
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Skaven252 wrote:
...it kind of defeats the purpose of this immersion POV video concept. This kind of video must remain locked to the head, use the full FOV, and no head tracking to crop/pan the picture should be used.

Yes, I realise it is not suitable for POV videos, that's why I moved the reply to another thread, I was thinking about regular 2D and 3D content. To make it comfortable it will have to be scaled right down into the center of the screen, which leaves a lot of black to fill up, more than just peripheral vision, so the ambilight effect might be a good solution.

Skaven252 wrote:
This is because the sound is locked to the head too, and I have no idea how binaural sound could be rotated in sync with the picture. Allowing the viewer to pan&scan around while the sound does not, would decouple from each other.
Maybe the tracking could still be used for audio only, to pan a "stereo pair" around a 7.1 surround speaker system somehow?

Skaven252 wrote:
The de-fishing step is an intermediate step that makes the video rectilinear... What would be needed, is the realtime "Carmack Warp" applied over this rectilinear side-by-side FOV video...
Yes, I was hoping the de-fish avisynth plugin could be persuaded to do the Carmack warp by entering the magic numbers (in the negative), but if it is not flexible enough to do that I'm sure we can find something else, MPC-HC has post-proc pixel shader support so I'm sure someone will put something together soon.

Keep us updated on the POV / binaural stuff, I look forward to trying it out on the Rift soon! 8-)


Wed Nov 14, 2012 3:59 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Nick3DvB wrote:
Maybe the tracking could still be used for audio only, to pan a "stereo pair" around a 7.1 surround speaker system somehow?

If the original sound was recorded in surround and not binaural, this would be the solution. You could use a simulated binaural HRTF, such as MyEars to create the binaural effect for headpones, and you could use Rift's head tracking that way.

However, actual binaural recordings, recorded with ear microphones or dummy heads, are just basic stereo audio files with the "earlobes and head occlusion" effect baked into the sound itself. There is no way to rotate that afterwards.

One possible solution could be to use a head-mounted surround microphone array (4 microphones) instead of binaural ear microphones to record the sound into 4 channels when recording these videos. Maybe that could be rotated, then translated to HRTF in real time.

Quote:
Yes, I was hoping the de-fish avisynth plugin could be persuaded to do the Carmack warp by entering the magic numbers (in the negative), but if it is not flexible enough to do that I'm sure we can find something else, MPC-HC has post-proc pixel shader support so I'm sure someone will put something together soon.

Do you mean the footage actually has to look fish-eyed (spherically distorted) to work with Rift? I thought LEEP compression was a bit more complex than just a simple fish-eye spherical distortion.

In fact, the VIO POV.HD footage is fish-eyed by default. But is it fish-eyed the right way? If someone has a Rift or another similarly configured HMD, feel free to view this test clip to see if it looks correct. That clip contains un-defished footage from the VIO POV.HDs. This is what the footage looks by default: wide angle fisheye.

Actually the forum member foisi already tested that video with his homemade HMD, but said it looked too distorted.

Quote:
Keep us updated on the POV / binaural stuff, I look forward to trying it out on the Rift soon! 8-)

Wish I had a life, and enough free time (and daylight - Scandinavian winters are dark!) to shoot something more interesting than just street walks in my neighborhood. :)


Wed Nov 14, 2012 5:18 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Hmm... reading that LEEP article again made me wonder: Is peering actually possible with the rift? As in: can you use your eyes to look around all the way to the edges of your field of view - or do you have to keep your eyes forward and rely solely on the head tracking? What's the eye relief of these Oculus Rift style HMDs?


Wed Nov 14, 2012 5:43 am
Binocular Vision CONFIRMED!
User avatar

Joined: Mon Aug 06, 2012 9:26 pm
Posts: 295
Location: Sweden
I'll skip quoting and just speak my mind, haha.

Live Streaming: I have a few projects where I want to stream to the Rift, but like a direct video feed as the latency has to be low. The only solutions I found to merge two HDMI-sources into a SBS signal costed an arm and a leg though so I put that on ice.

Online Streaming: Otherwise whenever it hits the web, which should be easy with all the streaming services now available, it will be very interesting if one could stream live from wherever :) I bet the best way to handle it, if you want user input, is to have one person reading stream chat and giving instructions to the cameraman via audio link. I know how messy crowded group conversations can be (raided 40-man WoW in the past, ugh) as well as text chat channels (IRC). Question again is what kind of box you would need to bake the image to be streamed, especially if it needs defishing and then leep-warping.

Video Format: That brings me to that subject. As people most certainly will use different kind of cameras to create videos for the Rift, I pondered if it would be best to publish them as rectilinear and then have the video player warp the image during playback. This way the only information required from the video is what FoV it displays, to scale the image to fit the Rift. It feels like it would be a good idea to figure this out before people start making videos, then with someone making a video player, but then perhaps we are not that many who plan to produce Rift media :x And I just realized we're not in the Rift forum now, oh well.

Video Playback: As for displaying the video in the Rift, many thoughts.
  1. Display the video directly on the display, no tracking.
  2. Display the video as a warp around screen which is static when you rotate your head.
  3. Just stabilization or sensors to record camera movement, move the image around the virtual space.
For option 1, I'm wondering in what position you should be when watching. Lying down? Sitting in a comfortable chair? I guess a static head at least, as to not have more conflicting motions than those present in the video. In option 2 it would basically work as using a dome display in real life, I'm fairly sure it will be less immersive, but I don't know. Option 3 just seems annoying as you'd have to track the video window manually. In this case 360° video might be better, but then you'd lose stereo vision instead, doesn't sound optimal.

Sound Playback: As for rotating binaural sound, one simple way would be to just play back the sound from two virtual speakers matching the video. If you look some other way, the sound will come from other directions even if they are binaural. This might totally break the audio immersion though, so probably a bad idea haha. Myself I will probably start out just recording normal stereo, if I can get even that to work.

There, my brainpoop for now ;)


Wed Nov 14, 2012 7:23 am
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
BOLL wrote:
Live Streaming: I have a few projects where I want to stream to the Rift, but like a direct video feed as the latency has to be low. The only solutions I found to merge two HDMI-sources into a SBS signal costed an arm and a leg though so I put that on ice.

... and if you add portability to the mix (ie the head camera person could move around without lugging a heavy piece of gear), I guess it becomes a case of impossibru. I tried looking around at some point too. There are stereoscopic HDMI video mixers, but not only they cost a fortune, they're also meant for studio rack use.

Two HDMI capture cards on a PC, and Stereoscopic Multiplexer... nah. :|

The MindFrog project dudes built their own multiplexer, but that was for PAL/NTSC video signal only.

Quote:
Question again is what kind of box you would need to bake the image to be streamed, especially if it needs defishing and then leep-warping

It's all technically, theoretically possible. My Nokia 808 PureView phone handles its 41 megapixel sensor output to resample it and to do a bit of lens correction too. With a dedicated chip. Which Average Joe or even Engineer Joe cannot build just like that.

Quote:
I pondered if it would be best to publish them as rectilinear and then have the video player warp the image during playback.

Agreed, rectilinear would act as a good intermediate / universal format. It should already be Rift compatible (FOV aside) if there was video support, because Rift was originally designed for polygonal game graphics which are rectilinear by default.

Quote:
For option 1, I'm wondering in what position you should be when watching. Lying down? Sitting in a comfortable chair? I guess a static head at least, as to not have more conflicting motions than those present in the video.

People have usually watched my videos on a desktop display / laptop, and I've adviced them to keep their heads still. Some complained about motion sickness.

Quote:
As for rotating binaural sound, one simple way would be to just play back the sound from two virtual speakers matching the video. If you look some other way, the sound will come from other directions even if they are binaural.

This might totally break the audio immersion though, so probably a bad idea haha.

Right you are. The thing is, you can't rotate binaural sound even with virtual speaker. Or yes you can, but it won't rotate the binaural directional effect. You'll lose the binaurality, and just hear normal stereo sound. Binaural sound has to be listened to with headphones, preferably earbuds so they bypass your own earlobes (so you avoid the "double earlobes" effect).

Quote:
Myself I will probably start out just recording normal stereo, if I can get even that to work.

That may actually work better than a binaural recording - if the sound is played through a game engine (or such) which supports virtual speakers and uses HRTF to simulate it.

No... wait... it won't. If you record just basic stereo, there is no directional information. You'll at least need a Schneider disk to reduce crosstalk between the mikes. Or use a stereo mike that has clear separation with the cardioid patterns pointing outwards. That will give you left and right, but no forward or backward.

You could try recording in surround with a Zoom H-2 (or H2S) and use 4 virtual speakers maybe?


Wed Nov 14, 2012 8:27 am
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Skaven252 wrote:
Do you mean the footage actually has to look fish-eyed (spherically distorted) to work with Rift? I thought LEEP compression was a bit more complex than just a simple fish-eye spherical distortion.
No, sorry I think we are just getting confused over the name, Dave Horman's avisynth plugin is called "DeFish"

BUT this filter can be used to correct (or induce) BARREL and pincushion distortion.

http://forum.doom9.org/showthread.php?t=152860

I just made a very quick 2D input script:
Code:
DirectShowSource("file.avi")
converttoRGB32
BilinearResize(640,800)
defish(fov=-100)
TurnLeft
Video1=TurnRight
StackHorizontal(Video1,Video1)

You can see the warp and aspect ratio are not really correct for the Rift:

Image

But there area a lot of variables to play with, I've not even looked at scaling, polynomial values etc. :?

It will be much easier when we have the official warp data from the SDK and an actual Rift to test it out with.

Hopefully that should be any day now... 8-)


Wed Nov 14, 2012 6:29 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Nick3DvB wrote:
No, sorry I think we are just getting confused over the name, Dave Horman's avisynth plugin is called "DeFish"
BUT this filter can be used to correct (or induce) BARREL and pincushion distortion.

I wasn't confused, I understood what you were getting at, I was just wondering if the default (already distorted) footage from VO POV.HD would be distorted the right way. :)

If I'm not wrong, barrel distortion is a simpler 2-degree version of spherical distortion (in DeFish). That's why I was unable to make my VIO POV.HD footage rectilinear with Emiliano Ferrari's Barrel Distortion filter. Which was a pity, because it's much faster than DeFish (likely because it's simpler).

You could try that Barrel Distortion filter for this too. It's a VirtualDub filter however... here's another barrel distortion plugin for AviSynth:
http://avisynth.org/vcmohan/DeBarrel/DeBarrel.html

... BUT anyway, I'm sure, or at least hoping that the Rift devkit comes with the official Carmack Warp in its SDK/API. :)


Wed Nov 14, 2012 11:18 pm
Binocular Vision CONFIRMED!
User avatar

Joined: Wed Oct 06, 2010 10:51 am
Posts: 311
Location: UK
Thanks for the links, I found BeBarrel a while back but have not tried it out yet. I'm pretty sure AVIsynth can load some VirtualDub plugins (in RGB32 mode) but you might need to use a wrapper of some kind? I will do some tests with Emiliano's plugin soon. Hopefully the SDK will have the warp info, but if not there are lots of ways to work it out, depending on how accurate you want it to be.


Thu Nov 15, 2012 8:39 pm
Cross Eyed!

Joined: Thu Aug 02, 2012 2:07 am
Posts: 120
Location: Sweden
Just a note to self: If you shoot stereo with two VIO POV.HDs, make sure both units' batteries are fully charged.

Because if one unit's battery gets low, it does keep recording, sure - but it starts dropping more frames than the other, and you're going to have a bad time trying to get the two in sync. :/

Two reasons:
1) Vegas Movie Studio refuses to pair two video clips into a stereo pair if their average framerates differ too much
2) if you force them both to the same framerate, they will drift apart. Oh, woe.

I guess I could still get them in sync by using some other video software, such as After Effects, but that baby costs like €1200. >_<


Sun Nov 18, 2012 3:04 am
Display posts from previous:  Sort by  
   [ 49 posts ]  Go to page 1, 2  Next

Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.