Stereo 360 spherical image pair

This is for discussion and development of non-commercial open source VR/AR projects (e.g. Kickstarter applicable, etc). Contact MTBS admins at customerservice@mtbs3d.com if you are unsure if your efforts qualify.
Post Reply
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Stereo 360 spherical image pair

Post by mediavr »

Hi
here are a left/right spherical image pair which people can use if they like for stereo environment maps - for Rift etc
.. these will map seamlessly on a sphere background
http://www.mediavr.com/belmorepark1left.jpg
http://www.mediavr.com/belmorepark1right.jpg

If you want to preview this in 3d (anaglyph, shutter glasses etc) you can use Stereophoto Maker
http://stereo.jpn.org/eng/stphmkr/ (Open/Left-Right images then Stereo/Grey Anaglyph (say) then View/Panorama Mode etc

If you would prefer to use cubic environment maps you could convert them say with the demo version of Pano2VR
http://gardengnomesoftware.com/pano2vr.php

I have notes on stereo panorama creation, viewing etc on my blog
http://www.stereopanoramas.com/blog/

If people are interested in using realworld 3d panoramas for Rift etc I will make some more available

Note these panoramas are aligned at infinity. For most purposes you would want to have a software feature so you can push the scene back in 3d (via lateral alignment)

If someone can write me a 3dVision compatible interactive viewer (ie. dewarping) for a sequence of stereo panoramas (for a regular 3d display or the Sony HMD) I would like to hear from you

PeterM
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: Stereo 360 spherical image pair

Post by geekmaster »

That looks like an interesting application for the Rift. Sort of a VR equivalent of the old ViewMaster us "old-timers" had as children.

If you compute a depth map from those stereo pairs, you should be able to compensate for sideways head tilt in the Rift:
Stereo Matching - Depth Maps from Stereo Pairs
http://3dstereophoto.blogspot.com/2011/ ... pairs.html
There's a paper published in 2002 titled "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms" by Daniel Scharstein and Richard Szeliski that does a thorough review of stereo matching methods. "Taxonomy" means that they have identified "building blocks" that are common in most algorithms. Doing that always makes things easier since it shows that a lot of algorithms are actually quite similar in the way they are designed. "Dense" means that the matching is done at the pixel level. "Dense Two-Frame Stereo Correspondence" is basically another way of saying "depth map". The authors are kind enough to provide the source code that was used to do the testing (which you can use on your own stereo pairs).
It would be fun to look around with full pan and zoom, while browsing through a VR album of these panoramic 3D stereo image pairs in the Rift.
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Re: Stereo 360 spherical image pair

Post by mediavr »

Stereo 360 panoramas have been used in virtual reality applications for a long time. Twenty years ago, pre-digital, there was the Spaceshot, a film stereo panorama camera/viewer system which had a lot of resources thrown at it.

" As I recall the device was on a plinth and you walked
around it while peering into a stereo viewer which had a geared mechanism which
propelled the film at the speed appropriate to your movement. The immersive
impression was supposed to be very strong and it was expensive!
>
>
> You are describing the "Spaceshot" (initially called "Pan 3D") - an immersive
3D viewing system designed by Jerry Kew of Leicester England. This however is
much more recent - dating from 1993. Although there was a display model built
as you describe, a handheld version was also available that had a built-in
gyroscope.
>
> A lot of money was put into this project and Seitz even built a stereo
panoramic camera for it. Unfortunately, the system ended up being way too
costly with cameras at $20,000. and viewers at $5000. each. In fact, film
cartridge modules cost about $1800. each !
>
> Even though the backers were hoping to sell the system to a large US
corporation involved in the real estate industry, the deal never happened and
the project was abandoned.
>
> Francois

http://groups.yahoo.com/group/photo-3d/message/85081

Ten years ago, when the VFX HMD was popular M.Husak wrote code for viewing stereo panoramas in the VFX (which had tracking) and the source is still
available online and the viewer executable works very well still on XP with Quadro/FireGL cards (it uses cylindrical format 360 images)
http://www.gali-3d.com/archive/articles ... acking.php

A tracked HMD project I made the stereo panoramas for about then was called Conversations from a research group in Sydney -- it had video characters in the panoramas, and collaborative "game" problem solving (between two HMD equipped participants)
http://www.icinema.unsw.edu.au/projects/conversations/

Stereo panoramas can be rendered from CG programs using a simple process eg.
http://paulbourke.net/stereographics/stereopanoramic/
and analogous concepts can be used for single camera stereo panorama capture of real world scenes but for action scenes (which is mainly what I am interested in)
twin camera (or twin lens) systems are necessary (or depth map 2D-3d conversion).

Twin camera spherical panorama capture of action scenes is non-trivial currently --- mainly because of calibration problems due to the off-axis cameras and blending issues in the stitching process viz. the left and right views are not aligned laterally except at one scene depth -- and manual guidance (eg. masks) needs to be given to the blending algorithms to ensure identical L/R stitching of action elements in the overlapping images in the L/R sequences.

Re depth maps from stereo panoramas. Some stereo video editing programs have stereo disparity detection and adjustment features now, eg. Nuke and Stereoid
http://www.pretendllc.com/ -- which can be used for creating depth/difference maps - but they are expensive. Scene reconstruction and depth maps from that are another approach but stereo footage is not ideal for scene reconstruction approaches. Kolor have a prototype program Stereopano for depth map creation from vertically separated panoramas
http://www.kolor.com/blog-en/2012/09/07 ... dre-jenny/

Vizard btw, from Worldviz -- has a feature in its virtual reality software package that enables occlusion effects from depth maps with stereo panorama backgrounds. A very efficient method, eg. for having CG birds etc, flying between the branches of CG rendered forest background stereo panoramas/depth maps.

PeterM
User avatar
Fredz
Petrif-Eyed
Posts: 2255
Joined: Sat Jan 09, 2010 2:06 pm
Location: Perpignan, France
Contact:

Re: Stereo 360 spherical image pair

Post by Fredz »

mediavr wrote:here are a left/right spherical image pair which people can use if they like for stereo environment maps - for Rift etc
Thanks for the images Peter.
mediavr wrote:If people are interested in using realworld 3d panoramas for Rift etc I will make some more available
Yeah, don't hesitate to post more of them. If you've got videos filmed in the same way all the better. :)
mediavr wrote:If someone can write me a 3dVision compatible interactive viewer (ie. dewarping) for a sequence of stereo panoramas (for a regular 3d display or the Sony HMD) I would like to hear from you
StephaneLX is working on such an app for the Rift. It's called VR Player and supports spherical 360x180° stereoscopic videos. Have a look here for more information :
http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=15927

He also released a Chrome extension for streaming videos in VR Player :
http://www.mtbs3d.com/phpBB/viewtopic.p ... 84#p101484
geekmaster wrote:Stereo Matching - Depth Maps from Stereo Pairs
http://3dstereophoto.blogspot.com/2011/ ... pairs.html
For state of the art algorithms in this domain, have a look here :
http://vision.middlebury.edu/stereo/eval/
mrYoda
One Eyed Hopeful
Posts: 2
Joined: Wed Apr 10, 2013 10:00 am

Re: Stereo 360 spherical image pair

Post by mrYoda »

Hi guys,

I've found the following link recently and have been thinking about its possible applications with Oculus:
http://immersivemedia.com/demos/

It's basically a panoramic video which you can control with a controller (mouse or anything) to have the sensation of looking around in the video. So I went ahead and downloaded one of the videos there which is in f4v format. Then I started looking for a video player which could possibly play this in a stereoscopic (side-by-side) format. My thought was if we can have video converters to convert regular 2D videos into side-by-side format, and if we have interactive controllers to watch these panoramic videos there surely could be player that could play this in a stereoscopic format. Apparently not at all!

After a bit of Googling I found this video detailing a project for a stereoscopic panoramic "image" viewer. And watching the video the process seemed incredibly complex.
http://www.youtube.com/watch?v=5dTpLCXRCfA

So I decided to take a simpler approach and just found a flash player to view the downloaded panoramic video locally. The player allows you to view the video in a browser, move the camera around with mouse inputs but unfortunately is not open source. Then I thought, what if there's a way to play the same video side-by-side to create a pseudo stereoscopic image. And the result is crude but gives you the idea:
Image

Obviously when you have two browsers playing side-by-side you can't easily synchronize the video playback or the mouse controls for both. But considering the simplicity of the flash player for panoramic videos how difficult would it be to modify the code to play two views side-by-side? You also need to apply the Oculus' warping transformation (the player already has transformations for things like Fisheye, Stereographic) but it should be fairly simple to upgrade such a player to a full-blown 3D Panoramic Viewer for Oculus.

Is there any app developments going on for something like this that any of you are aware of?
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Re: Stereo 360 spherical image pair

Post by mediavr »

Hi
here is another stereoscopic panorama image set (a street scene at an intersection) - that I hope might be useful for those playing with stereo panoramas with
Rift etc. The L R panoramas are 360 by 180 images converted to cubic image sets -- also there is a 360 by 180 anaglyph equirectangular
version of the panorama. This panorama has more depth and has higher resolution than the previous one I posted (the link will only work for 30 days - 12 Mb)
http://www.yousendit.com/download/UVJpT ... MHVFQk1UQw

for interconverting panorama formats Pano2VR and PTGui are very good tools

PeterM




mediavr wrote:Hi
here are a left/right spherical image pair which people can use if they like for stereo environment maps - for Rift etc
.. these will map seamlessly on a sphere background
http://www.mediavr.com/belmorepark1left.jpg
http://www.mediavr.com/belmorepark1right.jpg

If you want to preview this in 3d (anaglyph, shutter glasses etc) you can use Stereophoto Maker
http://stereo.jpn.org/eng/stphmkr/ (Open/Left-Right images then Stereo/Grey Anaglyph (say) then View/Panorama Mode etc

If you would prefer to use cubic environment maps you could convert them say with the demo version of Pano2VR
http://gardengnomesoftware.com/pano2vr.php

I have notes on stereo panorama creation, viewing etc on my blog
http://www.stereopanoramas.com/blog/

If people are interested in using realworld 3d panoramas for Rift etc I will make some more available

Note these panoramas are aligned at infinity. For most purposes you would want to have a software feature so you can push the scene back in 3d (via lateral alignment)

If someone can write me a 3dVision compatible interactive viewer (ie. dewarping) for a sequence of stereo panoramas (for a regular 3d display or the Sony HMD) I would like to hear from you

PeterM
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: Stereo 360 spherical image pair

Post by geekmaster »

Thanks for the new image. I will need to learn how to process these. Last time I stitched a bunch of photos together I used some free tool from Microsoft Research, and it only new how to deal with a collection of "normal" photos.

Which reminds me, I should probably make my Canada panoramas Rift-ready. But my panoramas were not stereoscopic. Perhaps I can extract stereoscopic pairs from the overlapping Left/Rift portions of different camera rotations (for the parallax offset) to get a pair of stereoscopic panoramas. I have read how to do that, and I do realize that using images from a single camera rotation for stereoscopic panoramas can cause subtle parallax errors -- but I want to do it anyway...

Here are many free non-stereoscopic cube maps, and a cubemap viewer:
http://www.humus.name/index.php?page=Textures

EDIT: That cubemap viewer want the images to be 512x512, and with specific file names. I resized and renamed your Belmore images, and packaged them with the cubemap viewer (as allowed by included freeware license). Here is the combined set:
CubeMapViewer.zip
I ran that program twice, once with each of left and right Belmore 512x512 cubemap sets, and placed them side-by-side to take this Cross-Eye stereoscopic image pair, after moving the mouse around to find a good view and lining them both up:

Belmore Cross-Eye View
Belmore.jpg
The link above includes full source code for this freeware CubeMap viewer.

EDIT2: Hmm... For some reason, the cubemap viewer blew up with the original 2024x2024 Belmore images after renaming them. I upscaled them to 2048x2048 and it still did not work. Then I downscaled to 1024x1024, and still bad. Then 512x512 worked fine (as included in the above attachment). What is strange is that OTHER cubemap images you can download at the above link (and there are MANY) are 2048x2048 and they work with this cubemap program... Hmm... The mousewheel lets you zoom into them, but not much use on the 512x512 images.[/color]
You do not have the required permissions to view the files attached to this post.
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Re: Stereo 360 spherical image pair

Post by mediavr »

Krpano which is popular software for displaying 360 panoramas (in Flash or HTML5) can show synced interactive panoramas usng Javascript
http://krpano.com/examples/10812/exampl ... creen.html

Krpano also has options for perspective type for viewer geometry -- rectilinear, fisheye etc
eg. http://krpano.com/krpano.html?pano=pano ... ruecke.xml
.. right click to see the perspective options -- you can zoom in and out with shift and control keys

.. in this example the wide angle zoom is restricted but with the latest version of krpano the fisheye view can be as wide as you like

I will have to see if I can get an example working for Rift

PeterM







geekmaster wrote:Thanks for the new image. I will need to learn how to process these. Last time I stitched a bunch of photos together I used some free tool from Microsoft Research, and it only new how to deal with a collection of "normal" photos.

Which reminds me, I should probably make my Canada panoramas Rift-ready. But my panoramas were not stereoscopic. Perhaps I can extract stereoscopic pairs from the overlapping Left/Rift portions of different camera rotations (for the parallax offset) to get a pair of stereoscopic panoramas. I have read how to do that, and I do realize that using images from a single camera rotation for stereoscopic panoramas can cause subtle parallax errors -- but I want to do it anyway...

Here are many free non-stereoscopic cube maps, and a cubemap viewer:
http://www.humus.name/index.php?page=Textures

EDIT: That cubemap viewer want the images to be 512x512, and with specific file names. I resized and renamed your Belmore images, and packaged them with the cubemap viewer (as allowed by included freeware license). Here is the combined set:
CubeMapViewer.zip
I ran that program twice, once with each of left and right Belmore 512x512 cubemap sets, and placed them side-by-side to take this Cross-Eye stereoscopic image pair, after moving the mouse around to find a good view and lining them both up:

Belmore Cross-Eye View
Belmore.jpg
The link above includes full source code for this freeware CubeMap viewer.

EDIT2: Hmm... For some reason, the cubemap viewer blew up with the original 2024x2024 Belmore images after renaming them. I upscaled them to 2048x2048 and it still did not work. Then I downscaled to 1024x1024, and still bad. Then 512x512 worked fine (as included in the above attachment). What is strange is that OTHER cubemap images you can download at the above link (and there are MANY) are 2048x2048 and they work with this cubemap program... Hmm... The mousewheel lets you zoom into them, but not much use on the 512x512 images.[/color]
mrYoda
One Eyed Hopeful
Posts: 2
Joined: Wed Apr 10, 2013 10:00 am

Re: Stereo 360 spherical image pair

Post by mrYoda »

Hi,

Yes, krpano was the player I used but it was the only one with flv support. Since it's not open source I couldn't really play with it further to work towards a Rift compatible function but wanted to throw the idea out here thinking someone more resourceful could pick it up.

Edit: I mentioned flv support because I think the end goal shouldn't be limited to viewing static pictures but also videos.
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Re: Stereo 360 spherical image pair

Post by mediavr »

here are high resolution L/R versions of the new panorama (intersection) in equirectangular (spherical) 360*180 perspective
http://www.yousendit.com/download/UVJqV ... NEw1SE1UQw

PeterM






[quote="mediavr"]Krpano which is popular software for displaying 360 panoramas (in Flash or HTML5) can show synced interactive panoramas usng Javascript
http://krpano.com/examples/10812/exampl ... creen.html

Krpano also has options for perspective type for viewer geometry -- rectilinear, fisheye etc
eg. http://krpano.com/krpano.html?pano=pano ... ruecke.xml
.. right click to see the perspective options -- you can zoom in and out with shift and control keys

.. in this example the wide angle zoom is restricted but with the latest version of krpano the fisheye view can be as wide as you like

I will have to see if I can get an example working for Rift

PeterM
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: Stereo 360 spherical image pair

Post by geekmaster »

Thanks. I like the spherical image pairs better.
mediavr
Two Eyed Hopeful
Posts: 58
Joined: Sat Aug 08, 2009 9:53 pm

Re: Stereo 360 spherical image pair

Post by mediavr »

I played with VRPlayer and I think this is a correct source file for it-


http://www.mediavr.com/lrintersection.jpg
To use it you need to set the following settings:
Media/Format to Side by Side
/Projection to Sphere
/Effect to None

Display/Layout to Side by Side
/Distortion to Barrel Distortion

To make this source file from the equirectangular pair I posted I did this:
resize both images so they are squeezed laterally 50% -- ie. now they are square
place them side by side in a new image ... done

About the VRPlayer -- the images that it produces currently - even from high resolution source material - seem to be a lot
more low res than they should be. They seem very aliased also.

PeterM
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: Stereo 360 spherical image pair

Post by geekmaster »

Thanks!
StephaneLX
One Eyed Hopeful
Posts: 39
Joined: Mon Sep 03, 2012 10:38 am
Location: Montreal

Re: Stereo 360 spherical image pair

Post by StephaneLX »

Hello PeterM,
mediavr wrote:About the VRPlayer -- the images that it produces currently - even from high resolution source material - seem to be a lot more low res than they should be. They seem very aliased also.
In the next version (0.1.6.0), I added a settings to adjust the polygon count of the sphere/cylinder. This should improve the aliasing but It will not be perfect cause the image has to be deformed by the graphic engine I use (WPF).

I will release the next version very soon, probably this week. I will be interested to know if this is helping!
Frank1000
One Eyed Hopeful
Posts: 1
Joined: Tue Mar 18, 2014 6:28 am

Re: Stereo 360 spherical image pair

Post by Frank1000 »

Hi,
had seen this post when i was researching for the topic of spherical stereo filming for texturing a sphere object and looking around in it via VR (Unity3D each left and right cam sees separate textured sphere). for my understanding distant placing of two cameras for left and right eye is not enough since depending on the view direction the distance of the cams gets broken. as a result, when looking at the portion of the spherical images that represent the left side of the scenery, the viewpoint of the right eye will be behind of the viewpoint of the left eye ... thus increasingly losing the distance between the left and right eye when watching the left side of the scenery.

Any opinions ?

Regards,
Frank
Post Reply

Return to “VR/AR Research & Development”