Optimization for Making Stereoscopic 3D Games on PS3

[Fredz]

Optimization for Making Stereoscopic 3D Games on PlayStation® (PS3™)

I stumbled upon this presentation from Sony talking about the creation of stereoscopic 3D games for the PS3. There are a lot of subjects addressed in depth and they even talk about the reprojection techniques used by Trioviz or Crytek in good details. Very interesting reading indeed.

Outline :
• Stereoscopic 3D (S3D) Games Overview
• How Stereoscopic 3D Works
• PS3™ Implementation of S3D Games
• Reprojection
• SPU Optimisation
• Case Studies

The PDF of the presentation : http://www.technology.scee.net/files/pr ... ionPS3.pdf" onclick="window.open(this.href);return false;

[Chiefwinston]

Wow. That is a good read fredz. I'm gonna go over it a second and third time.
Does anyone know if sony's gt5 uses the reprojection method? I've seen the Trivioz method. Its crap 3d. If gt5 is using this reprojection method. then its settled in my mind. Cause i thought gtr5 was using dual rendering. I sense that crow is on the plate for some. hahaaa

cheers everyone

[Fredz]

I think GT5 is using brute-force as seems to be confirmed by the interview of the US producer Taku Imasaki :
http://www.connectedconsoles.com/ps3-im ... ammers.cfm" onclick="window.open(this.href);return false;

[Chiefwinston]

Well fredz, I got a good taste of the ps3's reprojection technique. I was very disappointed with a trivoz game I played. It had horrible 3d. I just played several hours of killzone 3 with a move controller. The 3d could use a slight bit of a bump in separation. But I feel it works remarkably well. The immersion is superb. I did not see any strange anomalies. I walked around staring at everything for hours. A bump in separation doesn't seem all that difficult from here. I think we'll have to rename it really really good fake 3d.

Cheers everyone

[Chiefwinston]

Okay I've studied this document a few times fredz. Did you notice page 73 they have some head tracking technique with the playstation eye camera that gives a hologram effect when the viewer moves about a room. Fascinating.

Cheers

[Chiefwinston]

Looks like they call it ortho viewing head tracking VR.

Cheers

[Fredz]

Yes, I wonder why they didn't use it yet in a game, that would give a really great immersion.

First because it would add motion parallax which is absent in current stereo 3D games and is one of the strongest cues for depth perception, even more for people that can't see in stereo.

Second because as you said it would allow orthoscopic viewing, ie. what you see on the screen is at the same size than it would be in reality, so you would really feel like you're part of the scene. It would also add a very useful depth cue, namely "familiar size".

With this technique, 11 of the 14 depth cues would be reproduced instead of the 9 available in current stereo 3D games and movies. The missing depth cues would only be accomodation, convergence and blurring, which would be a real progress. I really hope they'll try to implement this soon.

[cybereality]

Ok, that was an interesting read. But there was nothing earth-shattering there, just a confirmation of what I expected. At the very least it shows that Sony knows exactly what they are doing and are able to explain that to 3rd party game developers. Just by reading that PDF any game developer can easily avoid 90% of the problems with 3D games currently. So this is great stuff. Combined with the effort Nintendo is doing with the 3DS, I imagine every game studio worth its salt will be a pro at 3d game production by the end of this year.

The ortho-viewing stuff is really cool and would certainly add a lot to 3D gaming. I actually have experimented with this a little myself ( http://www.youtube.com/watch?v=hcOdFAGXMTk" onclick="window.open(this.href);return false; ). Unfortunately because of the narrow viewing angles on the Zalman I was not able to produce a 3D video of that project. However I did test it with 3D glasses and it did look pretty cool. I have plans to complete that project at some point but I was having trouble getting the correct projection matrix and just had to fake it a little. When I get some more free time I will research this further. Its definitely the next level of immersion after stereo 3d. Optical head-tracking is one option (ie Playstation Eye or a standard webcam like in my video), but I think using a 6dof physical tracker would give much better results (like the tracker on the upcoming VR1200). DDD tried to implement this on their 3D driver but its was really buggy and barely worked at all. Maybe one day we as a community can create our own 3D driver and we can experiment with all kinds of stuff like this.

Also, I liked the parts when they talked about the reprojection technique, but I wish they had some more details. The part where they explain what to do with invalid pixels puzzles me, though. They say to just fill the invalid pixel with the last valid color scanned. This seems like an extremely naive implementation. That was the first thing I tried when I was working on my 3D image conversion application, and the results were not that great. I mean, if I tested with solid colored shapes that might work, but with a real image it looked pretty bad. The main problem with just doing one pass like this, aside from the loss of texture detail, is that the "back-filled" pixels are biased in one direction. For example if the background is mostly flat and then there is a character close to the screen, then there is going to be invalid pixels surrounding that character. When something is close it is going to shift in the opposite direction of your eye. So the person will be offset to the left in the right eye, and to the right in the left eye. Since it will leave invalid pixels in its wake, that means in the left eye there is a "shadow" to the left of the character of invalid pixels. In the right eye there is a "shadow" on the right. It will look something like this:



The green pixels are the invalid pixels cause by the shifting of the character in 3D space. As you can see, this is a problem. If we do what Sony has suggested in the PDF, simply iterate through the lines and replace invalid pixels with the last valid pixel we get something like this, which has a bias in one direction:




Now this actually looks OK in the left eye, since the sky background is very uniform in color. However it looks absolutely horrible in the right eye, where you see a stretched doubled shadow of the character trailing off his edge. Completely unacceptable. I managed to get around this in my program by always using the further back pixel. So instead of using just the last color scanned, the algorithm would continue scanning until it found another valid pixel, then fill with the color of the one that was further back. In this case it would use the sky of the invalid pixels on both sides, and thus actually look pretty good. This isn't a prefect system and it can still look pretty bad when the background is more detailed than just a sky or other solid areas like a grass or dirt ground. For example, if there are vertical bars that are occluded (like the edge of a door, for example) then this will not be recreated and it will look pretty bad. Other problems are areas with detailed textures, for example if an advertisement with text on it is occluded, then no algorithm can recreate the missing text (or even fake something decent). Lots of problems with this technique. That is why they must keep depth at a minimum to mask these issues. It can probably provide acceptable results, but it will never come close to a true stereoscopic render.

Reprojection_01.jpg

Reprojection_02.jpg

[Chiefwinston]

I'm sorry Cybereality but your mis-understanding a fundamental part of sony's reprojection technique. It's doesn't really matter though because there are a few games playable right now and coming up that do not suffer from that horrible stretched pixel disease. And have 3D that is indistinguishable from dual render/brute force methods right now. I've seen them.

Cheers everyone

[Fredz]

No Chiefwinston, Cybereality is right and knows perfectly what he's talking about.

Artefacts of reprojection are barely visible on games like Crysis 2 because of the low separation that is used, but they are still present. And Killzone 3 can't be used as a proof of the good results of the reprojection technique because nobody except the programmers of the game know if brute-force or reprojection was used.

Crytek and Sony even said in their respective papers that the reprojection technique is no panacea and that it forces a compromise on separation, don't be more royalist than the king...

[cybereality]

Well I was coding 2D+depth algorithms before Sony even announced they were making games in 3D, so I think I might know what I am talking about. The PDF linked to in the OP describes exactly the technique I illustrated in the pictures I posted. Now I am not saying that is what game developers are actually doing. I am sure the programmers at Crytek are smart enough not to use such a naive algorithm. So what you have experienced in games like Crysis 2 are probably a lot more advanced than that. All I was saying is that Sony was going on record suggesting a really bad method of back-filling invalid pixels. And its possible some developers may be lazy and/or inexperienced with stereo 3d and not realize how bad an algorithm that is. Hopefully most developers would be smarter than that, which is why I was surprised Sony would even suggest it.

[Chiefwinston]

Yes fredz, you are right. Mr cybereality I'm sorry. I didn't mean to come across so brash (an ass). I hope you'll accept my apologies.

But back to the debate on hand. The part in the sony document about their second rendering pass is where they cure the stretched pixel disease. There is no need for a second pass if your going to stretch,copy or guess what the missing pixels are. That can be done with one pass.

Last night I fired up my VT25 and threw in my new 3D blu-ray of Avatar. I watched the whole thing. Its a masterpiece of stereoscopy. After the first jaw dropping viewing. I set up my game machines. First I would watch Avatar my reference quality material then switched quickly and played cysis 2. I did this back and forth for several hours. I also, did this with Killzone 3. I must say the stereoscopic settings for the games are at the same pleasant settings as the movie. Convergence and seperation are within a guestimated 5%. All look fantastic. The 3D with these new upgraded game engines will fool everyone into thinking they are the brute force method. If you have Avatar the 3d blu-ray try it. Its a great way to judge and calibrate what is good stereoscopy and what is bad.
we will have an amazing time with this new technology. It is quite good/great.

cheers everyone

[android78]

Last night I fired up my VT25 and threw in my new 3D blu-ray of Avatar. I watched the whole thing. Its a masterpiece of stereoscopy.

Avatar may be a masterpiece (I watched it 4 times at the cinemas), but it is a film and I felt that it could have done with a bump up in depth myself. Mind you, I haven't got it at home, so maybe they bumped it up for home release? (I doubt it though)
The other point is that avatar is a film, so I accept less then a realistic representation of depth since it has to be acceptable to a wide range of viewers. When the content is generate real time, I don't see why we shouldn't be able to have as much depth as we want... except of course, the anomalies caused by faking the 3D.

[Chiefwinston]

Well I don't think it will replace the brute force method. It's a another complementary method of stereoscopy. It does represent a huge efficiency gain that software engineers are already well aware of. I've invested in so much PC hardware over the years getting games to run in stereo that I cringe at the thought of how much more I'll be buying. I'll do it because there's nothing comparable to 3d gaming. And yes I like high separation and convergence settings too. So I welcome advancements that will bend the curve of hardware costs. Software efficiency is badly needed.

Cheers everyone

[cybereality]

I actually found Avatar to be pretty mild as a 3D experience, even though I did see it in theaters 3 times and liked it very much. I mean, even some Disney stuff like Tangled probably had better 3D effects than Avatar (however I would watch Avatar over that Disney stuff any day of the week if given the choice). And, in general, I find the 3D gaming experience to be a lot more satisfying than cinema. So if all Sony can do with PS3 games is match 3D movies, I would consider that a fail. They have the potential to do so much more.

[Chiefwinston]

It looks like there will be 2 tennis games and 1 baseball game released in the near future. I hope they impliment the head tracking VR tech. I've played some of the move tennis demos. I like them. hologram type tech with this particular game type would be cool. It has natural head movement. the game is also simple enough that a ps3 could brute force the graphics (if thats how it would be done).

cheers everyone