Interactive panoramic video 360 + Head tracking (HMD)

[urospeed]

Hi everyone!

First of all, I'm apologize about my bad english speaking! I come from Quebec City!

I read often this forum and it's the first time that I write here!

I'm presently working on a personal project and I have a problem.

I have videos likes this :
http://www.nimmobay.com/media/360-video ... ter-resort
http://gardengnomesoftware.com/pano2vr_ ... demo=video

But i wnat to use my Vuzix 920VR as head tracking and video glasses (HMD)
I want to control this interactive panoramic video with my HMD.

Like this example:
http://www.youtube.com/watch?v=ilXUqXPn ... ce=message

Do you have a alternative for Imove Geoviewer?

I want to run it on Raspberry model B 512 mb Ram, under debian Linux.

I fund a little driver for my 920VR under linux:
http://www.mygnu.de/index.php/2009/02/v ... 3d-stereo/

Can you help me with my project, I know it's probably simple, but I'm a little bit newbee in programamtion.

Thanks alot for yours futures answers!

Have a nice day

P.S. Very apologize about my english speaking!

[urospeed]

Nothing??

[brantlew]

I can only give you guidance on using the Vuzix tracker in Windows. You can find sample C# code for interfacing that device in the FreePIE project.

https://github.com/AndersMalmgren/FreeP ... rPlugin.cs

This is probably not very helpful though if you are creating a Linux project. Sorry.

[cybereality]

That helicopter video was cool!

Sorry I can't be more helpful, not sure how you would get that working in linux. I mean, you would probably need to build your own video player. If I were to do something like that, I would probably use OGRE, I believe there is open source code out there that will play video on a texture. Then you would just need to handle the panoramic aspect of it maybe that could be done with cube mapping or something similar. Never really looked too deep into it, but I bet its not that hard.

[urospeed]

Thanks a lot Brantlew and cyberreality!

All informations is good for my project!
If you have an sample of C code, javascript, flash on windows to integrate head tracking in an immersive 360 video it's helpful!

An other example 360 video ! Imagine with the head tracking!

http://www.vrwebdesign.co.uk/360-degree-video/

[rmcclelland]

Very cool. I think this has a lot of potential for "virtual tourism".

It would also be a good way to give people a tour of a lab or something remotely.

I guess the limitation is you can only get head rotations, not translations, which could induce motion sickness.

[urospeed]

For the translation, Why not modify the speed a little bit? you want to stop at a place make a fix point 360 video, you want to be faster so, speed up the video! more lower? speed down!

I will contact programmer friends to help me in my project! But if you have video, code, explaination, tips or example! Feel free to talk to me!


I hope that you understand a little bit what I try say!

Thank you!

Have a nice day!

[MSat]

I was just about to start a new thread on the subject when I discovering this one. I think the implications of this technology could be immense! I really hope we can get this discussion going again.

The ideal panorama experience for an HMD would be to support stereoscopic images (still or moving) + tracking of orientation and translation. On the surface, it sounds extremely complicated, but I don't think it has to be in every case.

Lets start with computer generated images because we have full control of the scene and camera placement. One way to do it is to make like a gazillion stereo renders from all the the various vantage points the user might be able to see from. While this is easily possible to do in CG, it's terribly inefficient for "stills", not to mention dynamic panoramas. But it is an option that could yield really good results if you have enough renders. Another way is to render various "elements" of the scene separately and from relevant vantage points (the further the element, the less renders you need of it) and then generate the final scene with all the pre-rendered elements in real time using some scene descriptor. This method could probably yield as good results as the first one, but with a smaller data set (still a large amount @ high quality). In order to make it more efficient still, we would have to reduce the number of renders per element, and hope we can make something that doesn't look like crap. Can we mend together a smaller amount of renders using interpolation and other types of image processing like stretching, rotating, skewing, etc. and still get good results?

[StephaneLX]

The ideal panorama experience for an HMD would be to support stereoscopic images (still or moving) + tracking of orientation and translation. On the surface, it sounds extremely complicated, but I don't think it has to be in every case.

Hello MSat, I am currently working on a player that support sterepscopic images and videos, orientation tracking but not translation.
Maybe I will try to add translation when the player will support depth map for creating a parallax effect.

In the meanwhile, you can check out this thread: http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=15927

I am at a point where I would like to create content to test the player so feel free to experiment with the software, (it's an early alpha). If you manage to create content for it, I would be very interested in trying it.

[MSat]

Interesting project, StephaneLX!

I'm trying to imagine how stereoscopic cube mapping would work.. Wouldn't stereoscopy break when an object is projected partly on one cube surface, and partly on another?

Also, how difficult would it be to add image layering and alpha compositing support?

[StephaneLX]

I'm trying to imagine how stereoscopic cube mapping would work.. Wouldn't stereoscopy break when an object is projected partly on one cube surface, and partly on another?

Yes. The stereoscopic effect is optimal when camera is facing a surface. When the camera is facing an edge where 2 images are "stitched", the effect is diminished.

Also, how difficult would it be to add image layering and alpha compositing support?

To create image layering with the actual plugin system, you would have to create a new type of geometry (Wrapper).
ex: A cube inside a cube, inside a cube, etc...
The problem is that for each additional layer, the media file size increase a lot.
Also, I don't think that video with alpha channel are supported in WPF so you would have to use a blue screen type of pixel shader directly on the geometry to see through each layers..

Anyway, it's not difficult but it is not possible at the moment without modifying the code. Still, the idea is interesting.

[MSat]

When I think about it, maybe cube/spherical mapping is less ideal for this than something like 6DOF image/sprite placement. That way you're not binding images of the environment to a limited number of surfaces, but rather an indefinite amount of surfaces at any location, and any orientation - but all still subject to (at least some) rotations and transforms. This does sort of make generating a scene a PITA, as essentially you would be turning every object in a scene to multiple sprites (rendered from various perspectives due to parallax). It would require a scene descriptor which would dictate where the "engine" places the sprites/surfaces in 3D space.

I'm looking into giving this a shot using SDL and OpenGL. I think if I can describe the scene and do the math correctly, it might be a viable option.


Another option might be to map ray vectors passing through certain surfaces. Lets say you put a hypothetical transparent cube at some location in a scene you wish to render. As the renderer is ray tracing, some of those rays will pass from the outside through one of the 6 surfaces of said cube. If the color and vector data could be recorded, it could also be "played back" correctly to any view restricted within the cube. The detail being limited by the number of vectors per surface area and the resolution of the HMD. Using a moving aperture (relative to the imaging sensor) is might even be possible to record ray data from a real scene!


Also of interest to this topic: http://www.vision.huji.ac.il/stereo/

[BOLL]

An other example 360 video ! Imagine with the head tracking!
http://www.vrwebdesign.co.uk/360-degree-video/

Watched this again. Nice with no apparent stitching and probably post stabilization. I'm guessing it looks so good as there was an overcast, reducing contrast in the scene. Also, there are different modes in the player which I didn't notice last time, arcitectual and little planet are interesting.

Something I found quite fascinating which I wanted to share. When the video looped it was looking straight up into the sky and as I was watching it I let it spin slowly. It was a very liberating sensation, like lying on the cart without a care in the world, just watching the sky. And this is just from watching a video on a webpage?

I'm trying to imagine how stereoscopic cube mapping would work.. Wouldn't stereoscopy break when an object is projected partly on one cube surface, and partly on another?

Cubemapping itself is invisible if you have properly generated textures in a high enough resolution so you cannot see perspective distorted pixels and the cube placed with the center on the camera origin. To have a stereo pair of cubemaps you would simply render separate cubes for both eyes, with the camera always in the center of the cube. The problem then being that you'd need different cubemaps for different stereo pairs, meaning every direction you could turn your head in, including head/neck model then. As long as it's CG I'd still say it seems like a better idea to just run something in engine As for video, a different problem.

[MSat]

Cubemapping itself is invisible if you have properly generated textures in a high enough resolution so you cannot see perspective distorted pixels and the cube placed with the center on the camera origin. To have a stereo pair of cubemaps you would simply render separate cubes for both eyes, with the camera always in the center of the cube. The problem then being that you'd need different cubemaps for different stereo pairs, meaning every direction you could turn your head in, including head/neck model then. As long as it's CG I'd still say it seems like a better idea to just run something in engine As for video, a different problem.

That's why I don't think typical cube mapping mapping is the way to go, but instead an engine that can manage 2D "sprites" in 3D space, or what I refer to as a "lightfield box" - a sort of virtual container where the entire lighfield within it is mapped, so your head is free to float around within the box's limits, and generate an appropriate image for each eye. It's essentially a simple ray tracer for prerendered (or real) content. As long as you develop interactive media within its constraints, you can make it very immersive and have life-like visuals.

[mediavr]

On my blog about stereo panoramas http://www.stereopanoramas.com/blog/ there is a post with some links
to stereo panorama viewer software -- some with Hmd support
http://www.stereopanoramas.com/blog/ste ... ama-links/

Bitmanagement's BSContact Stereo is the only one with support for stereo video textures in skyboxes I think (maybe Vizard's Worldviz or Virtools too)

PeterM