Kinect Inadvertently Demonstrates 3D, U-Decide Update

[PressBot]

We have a proof of concept 3D lesson for everyone today!  Unless you have a medical limitation, you are probably able to perceive 3D, and this is how it works:  if you close your eyes one at a time, you will see that each eye sees a slightly offset view from the other.  In addition to being offset, each eye sees more of one view than the other.  Your brain takes the unique images provided by your left and right eyes, and combines them into a single picture. This picture includes the depth we all take for granted.



In the S-3D gaming market, there are two dominant schools of thought.  The first is complete left and right camera view rendering that could require double the processing power (in some cases).  Most PC stereoscopic 3D drivers and a selection of console games do this 100% of the time.  The second option is 2D+Depth which only renders a single camera, and based on the game's inherent Z-buffer information, it places the objects at different depths.  This is advantageous because there is little to no loss of performance, but you lose that extra information that a second camera would normally provide.  So how important is this extra information?
Read full article...

[Likay]

I'm very curious if crytecs "2d+depth" (what they now call it...) option will have a somewhat convincing 3d-image. Since they get the z-depth directly from the gameengine there shouldn't be a better way to use 2d+depth. So: If crysis 2 2d+depth isn't giving a convincing effect this technique won't cut it, ever.
According to my own experience no converted movie has so far been looking good enough for me to choose artificial 3d instead of 2d. Converting photos 2d+depth works amazingly well if some effort is put into it.

[Neil]

Check the article again. I think there is room for some healthy debate because we made a new discovery about how Kinect does its magic.

Regards,
Neil

[Chiefwinston]

2D + depth is probably just a bridge technique until consoles are powerful enough to render 2 seperate images. I'm okay with that. I'm betting its better than analygraph any day of the week. I'm willing to give it a far chance and play games that use this. It may be good enough to fool some of you into thinking the game is rendering to seperate images. Any one know of a game that uses this type of 3D. I'd like to take a close look and compare right now if its possible.

cheers everyone

[Fredz]

It determines your body's positioning with the help of a stereoscopic 3D camera. Oliver Kreylos' video (inadvertently) demonstrates the importance and quantity of extra information provided by a second camera view

It's not a stereoscopic camera, there are 3 cameras in Kinect, the first one is RGB and captures colored pixels, the two others are monochrome and are used to evaluate depth based on the image produced by a laser projecting interference patterns.

If 2D+depth was shown on camera, we theorize that straight on, it would look the same as this video. However, when the camera is rotated, the objects would be on different depth plains and appear as paper thin elements - like a deck of cards standing up on a table. It's this lack of visual information that makes it difficult for 2D+Depth to have convincing out of screen effects. Is it worth the trade off in performance? Share your thoughts!

I really don't see what you're wanting to say here. Are you saying thar Kinect should produce a cardboard effect or not ? And why did you talk about the cardboard effect in the first place, what does it have to do with the subject ?

UPDATE! On closer inspection, while there are two cameras with Kinect, they aren't equal resolution. One is 640X480, and one is 320X240. The 320X240 unit is used for the depth information capture, and it is offset from the first. So let's leave it to the MTBS membership to decide. Is Kinect in fact 2D+depth and is this article's theory wrong? Or is it more a true stereoscopic 3D capture device? How do you differentiate?

From Wikipedia, the three (and not two) cameras have all a 640x480 resolution. The technique used is 2D + depth, the pixel colors only come from the RGB camera, the monochrome cameras are used to record depth only.

And btw, this has nothing to do with stereo game drivers, both are called 2+depth but they only have that in common. Kinect is a recording solution, the technology used in Crysis 2 is a rendering solution.

[Likay]

Check the article again. I think there is room for some healthy debate because we made a new discovery about how Kinect does its magic.

Regards,
Neil

Lol, sorry. Got carried away by the 2d+depth mention.
The algorithm as explained (due to bad hearing i can't hear much but i have a very vague idea) is optimized to find objects and their positions in the volumetric room but the way doesn't match the imagination of 2d+depth that i have (of course feel free to widen a narrow mind).
Imo 2d+depth is technically a texture combined with a bumpmap, no matter how it's done or called. 2d+depth is not limited to show the "carbon cutout" some mentions in combination with 2d-3d conversions because the bumpmap isn't limited to the shape of the texture. If a great deal of effort is put into the conversion of a photo it can look really great.
2d+depth is imo technically a image which has an added topography. It can be said it looks like an image-explanation in physic of how gravity bends the actual space. The texture is the rubbercloth and the bumpmap is the gravity.
So if you have this rubberclothimage in mind and watching it from top or bottomside you're very close. So i don't really see any similarities between Kinect algorithm and 2d+depth.

Is Kinect about 3d? Absolutely, since the system is made to recognize objects in a 3d-room even if not presenting them in 3d. It's a pretty cool gadget actually!

[Dom]

Hi this could be stereo similar in the fact that it has two cameras or three, but does not produce a stereoscopic image. Just a floating pan and zoom that mixed two different camera angles together. The same was done with the Matrix and how they got to move the video view of Neo bending over to dodge bullets. They placed like 30 cameras in a oval going from high to low to the ground. Each camera takes a different view and they are pieced together with software and can be used in slow motion ect.. I think it only looks 3d cause it can move around in 3d space with differnt views.

With a 2d+depth is'nt it that they use the first renderer to make the second view so that the information for the second view is in a different space in volume and with added geometry or objects details. For a video game that has all the objects already on file all thats needed is a match to where you are in game and the driver can calculate where the extra volume, detail is needed.

The Kinect I think would need to render a whole image instead of the cardboard cut out that it does show somewhat. When zooming past the breaking point the system needs to make a congruent replication and use ultra sonic sensors to sense the objects that are around the room.

[PalmerTech]

Kinect almost used a 2d+depth system. They acquired 3DV, which had developed a tech called Z-Cam. Would have been much cooler, IMO.

[cybereality]

Yeah, I saw this video the other day. Pretty cool stuff. However it shows the limitations of 2D+depth: mainly that you have a lot of missing information you need to fill in. I had this problem when I was working on my 3D image conversion software. I actually abandoned the project because I wasn't happy with the results I was getting for this very reason. Although this could technically be done, for example the "content aware fill" in the new version of Photoshop, but its not feasible to do in real-time at the moment. I'm sure there will be progress in the field though.

[BlackShark]

Hi this could be stereo similar in the fact that it has two cameras or three, but does not produce a stereoscopic image. Just a floating pan and zoom that mixed two different camera angles together. The same was done with the Matrix and how they got to move the video view of Neo bending over to dodge bullets. They placed like 30 cameras in a oval going from high to low to the ground. Each camera takes a different view and they are pieced together with software and can be used in slow motion ect.. I think it only looks 3d cause it can move around in 3d space with differnt views.

Not exactly. The Matrix "Bullet Time" sequences are not 3D captures of the characters because the cameras do not shoot simultaneously. Each camera takes only one still shot and they shoot in sequence one after an other, emulating a single 2D camera travelling at very high speed.
The software used to assemble the shots is there to produce frame interpolation and create motion blur to create the smooth motion you see in the final shot.
http://www.youtube.com/watch?v=_KtghA0rkDY" onclick="window.open(this.href);return false;

[Chiefwinston]

I see this discussion has branched into two paths. I'm going to comment on the 2d + depth stereoscopic 3d standard. I read the white papers from the patent holders last night. 2D + depth is also one of the supported formats of the HDMI 1.4 spec. It does not have a lose of information. It is equal to page flipped , side by side, over under, and the many other formats out there. I believe this because 2D + depth can be converted to page flipped, side by side, over under, or the many other formats out there without a change in the 3D information. 2D + depth is not fake 3D. It was developed from the efforts to do 2D to 3D conversions. Any one peeing on it just isn't grasping it's significance. I also reviewed actual examples- quite interesting.

cheers

[Neil]

Hi Chiefwinston,

2D+Depth definitely loses information because when you have two cameras, you have two unique perspectives. You can't see around the corner without a second eye, right?

Regards,
Neil

[Chiefwinston]

Think of it this way. Start with a 3D image in front of you. lets say its in perfect 3D. This perfect 3D image can be reproduced with many formats. one of them is page flipped, side by side, ect... but also 2D + depth map. All these formats can also be converted to one another without making up info. In other words 2D + depth can be converted into 2 seperate side by side images that are equal to 2 native side by side images. neat trick eh.

cheers

[Chiefwinston]

Game makers may use this 2D + depth as there 3D format base. And simply convert into the various other fomats( side by side, over/under) for shutter glass use. 2D + depth in its raw format is more for autostereoscopic displays. Interesting very interesting. It may revolutionize console 3D gaming. Consoles need all the help they can get. And this maybe the ticket.

we'll see tho
cheers

[Chiefwinston]

I'm not sure if it helps any. but if I could rename 2D + depth. It would be 2D with perspective + depth. Just plan 2D seems to make people think of it like a flat playing card. It should be thought of as a perspective image.

cheers

[Neil]

I think I'm a bit foggy on the process of 2D+depth. In the gaming world, I understood it as capturing a single camera view, and placing the objects at different layers according to the Z-buffer (depth placement) information. So straight on, it looks ok, but if you were try to look around the object, the information wouldn't be there. This is why the tech is so limited by hairline separation levels and little to no out of screen effects. This has nothing to do with side by side, formats, etc. 2D+depth is the process to create the images for side by side, etc.

However, my explanation could be wayyyyyyyy off. Could someone share a 2D+depth explanation and link to a source?

Regards,
Neil

[Likay]

In other words 2D + depth can be converted into 2 seperate side by side images that are equal to 2 native side by side images. neat trick eh.

This is definitely not true. 2d+depth is not the same as true stereoscopic 3d, nor a transfermethod even if hdmi 1.4 support it (2d+depth is two different images transfered though).
See the post i made above which somehow describes 2d+depth. A 2d+depth image is exactly as it sounds like: One 2d-image and a greyscale topographic map (bumpmap). I'll give an example below why informations defeintely is lost using this format:
Look at a telephonepole with both eyes. The pole will cover one part of the background for the right eye while this part will be visible in the left eye and vice versa. There is information one eye has which isn't available in the other. By using 2d+depth this kind of data is obviously missing. One question is if this really will be noticeable if games uses 2d+depth on gameenginelevel and presents it in 3d. It remains to be seen.
Movieconversions using 2d+depth as is today are pure crap though but it's not because those who makes conversions are doing are doing a bad job. Instead it's an insanely hard and effortdemanding procedure with nowaday tech and still only yields poor results.
@Neil: I might haven't got everything straight but i think so. A great example for you of 2d+depth principle whould be how photoforms work. In this app you load a 2d-image and then manipulates the depthmap (topography) which gives depth. Sources have yet to be found though.

Edit: I just found out that topographic isn't the best word... help me....

[Fredz]

2D + depth is not fake 3D

It's not fake 3D, but as Neil said you definitely loose something compared to images taken with two cameras. You loose the depth and colour information that is visible on one eye only (edges mostly) and you won't be able to recreate the correct illumination for the second eye (metallic surfaces for example). You also can't capture the depth behind transparent objects (glass, smoke, etc.).

In other words 2D + depth can be converted into 2 seperate side by side images that are equal to 2 native side by side images. neat trick eh.

You'll have a somewhat acceptable approximation, but it won't ever be as good as images taken from two points of view.

2D + depth in its raw format is more for autostereoscopic displays.

Only for a certain class of stereoscopic displays, those like the Philips WOWvx. The other and most used types of stereoscopic displays (parallax barrier, lenticular lenses) need a different encoding.

but if I could rename 2D + depth. It would be 2D with perspective + depth. Just plan 2D seems to make people think of it like a flat playing card. It should be thought of as a perspective image.

It would be an incorrect term, the first image is 2D and not 2D with perspective. The term 2D + depth is perfectly adequate.

[Chiefwinston]

hmm, Its interesting that all of you seem to be stuck with the single 2D image. I'm not going to be able to show you how or why its equal to true 3D. It is. And its quite impressive if you can get your head around it. and understand its ramifications. but hey, I tried.

cheers

[Likay]

Found a wiki. It feels a little old (does not explain 2d+depth presentation using z-depth within a gameengine though) but get some info here: http://en.wikipedia.org/wiki/2D-plus-depth" onclick="window.open(this.href);return false;

hmm, Its interesting that all of you seem to be stuck with the single 2D image. I'm not going to be able to show you how or why its equal to true 3D. It is. And its quite impressive if you can get your head around it. and understand its ramifications. but hey, I tried.

cheers

I'm sorry but this makes no sense.

[Fredz]

In the gaming world, I understood it as capturing a single camera view, and placing the objects at different layers according to the Z-buffer (depth placement) information.

The 2D + depth technique doesn't deal with objects but with pixels. Initially two render targets are created, one for each eye. Then for each pixel in the 2D image, the depth is extracted from the depth map. The position of the pixel on the two render target is then calculated, by casting a ray from the eye to the plane of the render view that pass by the position of the pixel in the 3D space. The pixel is then rendered for each render target at the calculated position.

Then there are specific techniques to deal with depth discontinuities and transparent objects, like depth buffer resampling and bilinear filtering.

So straight on, it looks ok, but if you were try to look around the object, the information wouldn't be there.

Since the rendering is calculated for each movement of the camera, this is not a problem. In fact it's exactly the same than for brute force rendering.

This is why the tech is so limited by hairline separation levels and little to no out of screen effects.

The out of the screen effects are perfectly handled with this technique, it corresponds to the case when the casted ray encounters the pixel in 3D space before the 2D plane of the render target.

[Likay]

The 2D + depth technique doesn't deal with objects but with pixels. Initially two render targets are created, one for each eye. Then for each pixel in the 2D image, the depth is extracted from the depth map. The position of the pixel on the two render target is then calculated, by casting a ray from the eye to the plane of the render view that pass by the position of the pixel in the 3D space. The pixel is then rendered for each render target at the calculated position.

Then there are specific techniques to deal with depth discontinuities and transparent objects, like depth buffer resampling and bilinear filtering.

The wiki-link doesn't explain what happens on gameenginelevel but this is a good one. ^^

[Neil]

Hi Fredz,

Your remarks have a lot of valuable information in them. However, 2D+Depth definitely has limitations with out of screen effects and levels of separation. While this is proprietary in nature:

1. DDD's Tridef Virtual 3D (2D+Depth) offers very limited flexibility with separation and convergence.
2. Crytek has gone on record at GDC Online that out of screen effects are problematic with their solution - though this isn't 100% 2D+Depth.
3. At GDC Online, Trioviz confirmed that there are limitations to separation and convergence with their technique (via Unreal Engine on console)

To be clear, these are all proprietary solutions with their own checks and balances - but there is a consistent trend here. I'm interested to learn why this is.

Regards,
Neil

[BlackShark]

It seems logic to me that it's a visual limitation, not a technical one.
The weakness of 2D + depth is interpolating the occluded pixels behind the objects.

As long as you are in depth the occluded area is small and hidden in the background where the player hardly looks, and the amount of separation is limited by the camera parameters, no matter what the player does. Basically where there are bugs, you don't really look or you don't see them.
When you come out of screen, everything is reversed : the closer the object the bigger it becomes, the more it separates, and also the more it attracts the attention of the viewer. Basically the bugs are obvious and you're looking right at them. Also since this is a video game, the developers cannot control how close the player will move the camera to objects. As a developer responsible of setting up convergence, you cannot allow the player to hurt himself, so you have to limit the amount of out-of-screen effects dramatically to make sure the player won't hurt his eyes

[Fredz]

However, 2D+Depth definitely has limitations with out of screen effects and levels of separation.

It's not specifically related to out of the screen effects, it's only because depth discontinuities are more and more present when objects come closer to the viewer. But that's the case for an object in any position in respect to the projection plane, it only depends on how far the background is.

I made this crude drawing to illustrate the retroprojection based on 2D + depth compared to brute force rendering :



In the last picture you can see that the retroprojection can't render around some edges (drawed in black) because it is missing depth information. That's the case for both out and in the screen effects.

[Chiefwinston]

It seams to me one of the great things about this true stereo 3D format. Is also, its ability to do suedo 3D through mathmatics. Its a young format. It has a lot of potential. Out of screen 3D, better seperation all can be addressed mathematically. And those with great solutions will capitalize on it.

cheers

[Fredz]

2D + depth is surely a good thing for stereo 3D, even if it's far from perfect for now. Any technique that can avoid a brute-force calculation is very welcome, but it'll take some time for this technique to improve. That's what I hoped the CryTek guys did, but unfortunately it turned out that it wasn't the case.

One of the solutions I can see for the depth discontinuities problem is that instead of doing interpolation for missing pixels, the engine could render with brute-force for only those missing pixels. Then you could have perfect rendering at a not too expensive cost.

[Neil]

Hi Blackshark,

I think you explained it as I explained it - but in a different way. When looking through a depth-only situation, things look ok. When things get closer and eventually out of the screen, the data is missing or flawed. I pictured it as empty space, but I'm gathering this is incorrect. Instead, this data is warped or unnatural because it is missing the second view a second camera would normally provide.

Close?

Regards,
Neil

[BlackShark]

No Neil, in depth is not ok, the only place that is ok is at screen depth.
As soon as you leave screen depth, be it in depth or out of screen, data will be missing and problems will appear.

The difference is that in depth the damage can be limited and kept under control at an acceptable level, while in out-of screen it becomes very quickly extremely bad and cannot be limited by any gameplay mechanic that isn't obvious or unfriendly to the player.

[Chiefwinston]

Sorry guys, but the sketches and explanations are off the mark. Your using traditional double imaging to explain 2D + depth 3D. And well you've missing the boat. 2D +depth will yield perfect 3D results. If you see it differently your not fully understanding the format.

cheers

[BlackShark]

Chiefwinston, I think you are confusing 2D + depth with 2D+depth+occlusion

2D + depth cannot reproduce full stereo information, it is mathematically impossible : 2D + depth creates holes with missing information when transformed into stereo.

2D + depth is a 3D information format, but not a complete stereo information format

[Neil]

So was my original explanation of playing cards standing up at different plains correct?

I have an idea. BlackShark - if memory serves - do you have the DDD drivers? Can you do a comparison of a Virtual 3D and S-3D screenshot? Maybe point out the flaws, etc. so we can see the examples.

Regards,
Neil

[BlackShark]

No, I don't have the DDD drivers, only iZ3D drivers.

[Fredz]

So was my original explanation of playing cards standing up at different plains correct?

No, it's still this cardboard theory that has nothing to do with 2D + depth. You must understand that there it's a pixel centered technique, not an object one.

Can you do a comparison of a Virtual 3D and S-3D screenshot? Maybe point out the flaws, etc. so we can see the examples.

Yep, that'll be a lot better than my crude drawings. It could be quite difficult to see the difference with brute-force depending on the scene though, I think the problems will be most visible in moving scenes.

[Neil]

DDD works with FRAPS. Would that solve the problem?

Regards,
Neil

[Fredz]

Chiefwinston, maybe you could capture a video with the TriDef virtual 3D driver ?

[BlackShark]

Video probably won't be necessary.
Screenshots should be enough to show the difference, you just need to use the same amount of separation in both modes.

[Chiefwinston]

So was my original explanation of playing cards standing up at different plains correct?

I have an idea. BlackShark - if memory serves - do you have the DDD drivers? Can you do a comparison of a Virtual 3D and S-3D screenshot? Maybe point out the flaws, etc. so we can see the examples.

Regards,
Neil

Sorry Neil, I believe this is incorrect. But most of the guys here have the same explanation. 2D + depth will yield true 3D imaging that can then be shown in what ever format someone wants. It's more like a master 3D formula that can be used to generate side by side, over under ect.... and it won't be fake. Tho it can be used to make fake stereo, also.

cheers

[BlackShark]

Chiefwinston, the problem is not whether or not 2D + depth is a genuine 3D format, the problem is that Stereo images contains some information that 2D + depth cannot provide. Thus you cannot produce perfect stereo side by side or whatever other stereo picture format.

The opposite is also true, you cannot produce a perfect 2D + depth picture with a stereo pair of images : there will be missing information.

[Chiefwinston]

So whats missing? That pevents you from moving perfectly from 2d + depth to side by side?

cheers