GG3D Evaluation Criteria Need Update...

[whyme466]

I believe that GG3D criteria need to be updated to reflect actual operational 3D rendering layer (Nvidia, DDD, iZ3D) performance - that is, how users actually play the games. This is how any system is evaluated by end users. I believe GG3D assessments should emphasize NORMAL (operational) rendering layer settings, not unique settings, unused in typical game play.

The current asssessment criteria lack any evaluation of the dynamic performance of the rendering layer - how much interaction is required as a Dragon Age 2 facial closeup jumps to a panoramic scene, for example. Is manual interaction required? If so, how should this interaction be scored (penalized)? Is the rendering layer prone to severe (large penalty) or small, occasional (small penalty) autofocus adjustment, if typically used in normal game play?

The use of some effects in the 3D realm need reassessment, as well. For example, I agree with the author of this review about DOF: http://www.behardware.com/articles/807- ... ef-3d.html" onclick="window.open(this.href);return false;. I find that disabling DOF improves 3D (depth) resolution, a key attribute in 3D immersion. So, if a game cannot disable DOF, it should be penalized (not the current scoring approach in GG3D, however).

Any thoughts? I have reached an impass with my GG3D submissions (for example, see http://www.mtbs3d.com/gg3d/index.php?sh ... d&val=1601" onclick="window.open(this.href);return false;).

[Neil]

Hi Whyme466,

Thank you for sharing your thoughts. I know we have had an exchange of emails over our differing opinions, and I think it would be helpful if we did some digging as to what it is you are after.

Let me quote some of the statements listed, and I'll share my view, and I'm hoping the community will share their ideas as well.

I believe that GG3D criteria need to be updated to reflect actual operational 3D rendering layer (Nvidia, DDD, iZ3D) performance - that is, how users actually play the games. This is how any system is evaluated by end users. I believe GG3D assessments should emphasize NORMAL (operational) rendering layer settings, not unique settings, unused in typical game play.

I think there is room to be more specific here. The way to think about GG3D is it's not designed to tell you how you should play your game in 3D. Everyone's eyes are different, everyone has a preference on what they like and what they don't like, and the nature of the game has a role in this as well. DDD, Nvidia, and iZ3D all have their own way of rating games, and there is no meaning to how they rate their games. Excellent or Good or Lousy could all be talking about the same game and same game experience.

The idea behind GG3D is it rates games according to the ability to have maximum eye candy and being able to game without having visual anomalies. All games start with a score of 100%, and get deductions when eye candy has to be turned off, and/or there are visual flaws. Each of these components has a score value. What's left after the penalties are taken away is categorized into the Bronze to Platinum grades. There is a secondary indicator which tells us whether or not the game is capable of out of screen effects, but that isn't handled as a penalty - just an indication of visual flexibility.

When members are making submissions for their games, we ask that they accurately submit according to the GG3D specifications. HOWEVER, members are always welcome to share their personal recommendations for best results in the comments section.

The current asssessment criteria lack any evaluation of the dynamic performance of the rendering layer - how much interaction is required as a Dragon Age 2 facial closeup jumps to a panoramic scene, for example. Is manual interaction required? If so, how should this interaction be scored (penalized)? Is the rendering layer prone to severe (large penalty) or small, occasional (small penalty) autofocus adjustment, if typically used in normal game play?

Can you elaborate more on what you mean here?

If you are talking about having great 3D settings in one scene, and then suddenly having crazy separation or convergence levels in another where your eyes want to jump out of their head, that's called a camera angle problem, and that's listed as a "secondary anomaly" choice in GG3D.

We acknowledge that there are auto-convergence features in some drivers, and we ask that they be turned off for testing purposes. They prevent measurable S-3D flexibility, they cut down on game performance, and more often than not, they aren't even needed. If a game is well rendered, it shouldn't have camera angle problems in S-3D, and gamers shouldn't have to use an auto-convergence feature to get a positive result. Yes, there are instances when auto-convergence can benefit a game, and we ask that this commentary be left to the comments section as a personal recommendation. Auto-Convergence is a symptom of a problem that makes things difficult for driver developers - and the problem should be acknowledged in a measurable way.

The use of some effects in the 3D realm need reassessment, as well. For example, I agree with the author of this review about DOF: http://www.behardware.com/articles/807-" onclick="window.open(this.href);return false; ... ef-3d.html. I find that disabling DOF improves 3D (depth) resolution, a key attribute in 3D immersion. So, if a game cannot disable DOF, it should be penalized (not the current scoring approach in GG3D, however).

Let me ask the question a different way. If you don't turn the DOF off, will the game have visual problems and anomalies that shouldn't be there because of a driver incompatibility? If it's a case of compatibility, then it should still be handled as a penalty. Remember, we aren't telling gamers they should or shouldn't turn DOF off, we are saying they have the option to. If the driver can't support DOF, then it's a penalty because they aren't giving maximum flexibility to the gamer - it's not a real choice.

Any thoughts? I have reached an impass with my GG3D submissions (for example, see http://www.mtbs3d.com/gg3d/index.php?sh" onclick="window.open(this.href);return false; ... d&val=1601).

I didn't see settings reductions scored in your Skyrim profile - just pictures. Are your settings being reduced because your GPU isn't fast enough, or is it because there is an incompatibility that is harming performance?

A good example is AA with some DDD profiles. Turn on anti-aliasing, and a 55FPS game gets cut down to 5. It's not a problem with the GPU, it's an incompatibility with AA with this particular game. So, we have to turn AA off (which is a minor penalty). If you are turning settings down for natural speed loss, that's incorrect because users with much better GPUs won't know the flexibility that the game can potentially offer them. That said, you should still include your personal settings recommendations in the comments section (as you have) so the score of the game doesn't get falsely reduced. However, don't hesitate to list required score reductions for compatibility reasons.

The scoring isn't about artistic choice, it's about visual flexibility and measurable compatibility.

Regards,
Neil

[whyme466]

As a systems engineer with more than 30 years experience, we always evaluate systems the way they are used operationally. Yes, we can perform other specialized tests (as GG3D currently suggests), but operational behavior has always been critically important to assess as part of any system characterization. If an important part of operational behavior is simply ignored (like autofocus behavior or need to use manual interaction during active gameplay), the system evaluation becomes inaccurate and does not serve the end user (the gamer).

Have the GG3D criteria been vetted by human factors/cognitive experts or studies, in consideration of 3D versus 2D perception? For example, relevance of DOF?

In my scoring, I made sure criteria like DOF worked correctly (so no penalty), then I disabled for my actual gameplay. Any discussion about the concept for need of DOF in 3D? In 2D, DOF helps guide the eye to areas of interest/focus. In 3D, this process blurs depth resolution.

My settings are actual operational settings, derived from considerable experimentation to provide best visual quality while still maintaining adequate performance (please refer to Skyrim example text again). Note that I did not score any aspect of performance; rather, I documented the results of hours of tuning/experimentation (system dependent, but I describe my overall approach to tuning process, for application to other configurations). I am trying to help others by providing a documented setup, with a candid assessment of its issues, key references, and a characterization of 3D operational performance - my actual gaming experience.

[Neil]

As a systems engineer with more than 30 years experience, we always evaluate systems the way they are used operationally. Yes, we can perform other specialized tests (as GG3D currently suggests), but operational behavior has always been critically important to assess as part of any system characterization. If an important part of operational behavior is simply ignored (like autofocus behavior or need to use manual interaction during active gameplay), the system evaluation becomes inaccurate and does not serve the end user (the gamer).

Have the GG3D criteria been vetted by human factors/cognitive experts or studies, in consideration of 3D versus 2D perception? For example, relevance of DOF?

In my scoring, I made sure criteria like DOF worked correctly (so no penalty), then I disabled for my actual gameplay. Any discussion about the concept for need of DOF in 3D? In 2D, DOF helps guide the eye to areas of interest/focus. In 3D, this process blurs depth resolution.

My settings are actual operational settings, derived from considerable experimentation to provide best visual quality while still maintaining adequate performance (please refer to Skyrim example text again). Note that I did not score any aspect of performance; rather, I documented the results of hours of tuning/experimentation (system dependent, but I describe my overall approach to tuning process, for application to other configurations). I am trying to help others by providing a documented setup, with a candid assessment of its issues, key references, and a characterization of 3D operational performance - my actual gaming experience.

GG3D was actually founded as a proof of concept over a year ago under the name MTBS' 3D Game Analyzer, and is largely based on the input shared in these forums (and I'm not exaggerating when I proudly say that some of these guys predate a lot of bigwigs who are current 3D experts). That said, modern academic research in the realm of stereoscopic 3D gaming is very much preliminary, and a great deal is still being learned. I know this because there was such a necessity for continued research, I helped found a government funded initiative to help determine best practices in stereoscopic 3D gaming called iGO3D: http://news.uoit.ca/archives/2011/03/20110303_2.php" onclick="window.open(this.href);return false;

It's hoped that what is learned can contribute to an industry backed quality expectations standard. They aren't credited yet, but DDD is one of the big suppliers for this effort.

GG3D is a very different animal from this. GameGrade3D is a "wisdom of crowds" technology, and to maximize accuracy, we have to have reasonable confidence that people all over the world with varying levels of 3D experience can understand the questions in a measurable way, and give defined answers. If GG3D works properly, game developers, driver developers, and gamers should all be able to rate games with the same measuring stick and accurately match results. Subjective quality expectations are too nebulous a thing, and we are purposely avoiding that. GG3D currently gives gamers the information they need to know so they can make their own choices on how to play their games by telling them what settings they are absolutely required to reduce if any, and what visual flexibility they have to choose from. It's a protective measure too. For example, if there are DDD or Nvidia zealots submitting false scores to promote their favorite brand, it's difficult to debate if everything is subjective. GG3D puts everyone on an equal and transparent playing field, and the less wiggle room for debate, the better.

It sounds like you are doing your submissions both well and correctly. In particular, you are completing the score sheet according to the defined Q&A so there is a measured score, and you are sharing your own recommendations and ideas as separate comments and subjective scoring for interpretation. This is how GG3D is supposed to work. Just a reminder though that if there are camera angle problems when auto-convergence is turned off, you need to report that as a camera angle problem in the scoring - but you can discuss auto-convergence as a potential solution in your subjective score and comments at the end.

Regards,
Neil

[Fredz]

Concerning DOF, I'm not sure there is a consensus on its effect on 3D depth perception as Neil said.

It should also be noted that DOF is widely used by filmmakers in 3D movies, a notable example of this being Avatar. It forces spectators to focus on a specific part of the scene, mostly for two reasons :
- it tries to simulate the DOF people would experience in the same situation in real life ;
- it allows for more separation in the background that better corresponds to reality.

Without using this, the depth of a scene would have to be reduced to minimize the vergence-accommodation conflict.

I can't say if it's better or worse to use DOF for movies and neither for games. But if it does work this way in 2D in both movies and games there is probably a reason, and it's difficult to look at blurred zones even in 2D.

[BlackShark]

The use of DOF varies from game to game.
Some of them use DOF for artistic reasons (focus on some event, the character speaking, etC...), some use it for gameplay reasons (reloading a weapon, your character is supposed to be busy with the weapon and is not supposed to be looking elsewhere), some use it as a replacement for atmospheric effects, etc...

Sometimes it works great in conjunction with 3D, some other times it just does not.
I found the most annoying times where DOF gets in the way is when you stop actively playing the game and want to take a minute to look at the scenery, and some blur effect is right there in your face preventing you from looking where you want to look at. Usually that kind of effect is annoying both in 2D and 3D, so I turn it off under any circumstance.

[Guig2000]

Strangely some games (i.e.DIRT1 and 2) have not DOF but when you activate S3D some kind of DOF appear with the iZ3d driver: more objects are far from the 0 parallax depth, more they are blurry. Technically it's a post-processing issue but in fact I feel it's like an enhancement.

[cybereality]

Personally, I like depth of field. But I find in many games it can reduce the depth range, or completely kill it. A lot of times it is implemented as a full screen post-process effect, and those are notoriously problematic for 3D drivers. But, in this case, I guess it would be considered an anomaly so in GG3D you could say you had to turn it off and let the game get the penalty. But if its just a personal taste thing, I don't think the rating should be penalized. A lot of times I will turn AA off in order to get better framerates in 3D, but this is not necessarily because it was a problem. I just don't think it is necessary, and find the performance boost more important. Totally a personal choice.

[Neil]

Personally, I like depth of field. But I find in many games it can reduce the depth range, or completely kill it. A lot of times it is implemented as a full screen post-process effect, and those are notoriously problematic for 3D drivers. But, in this case, I guess it would be considered an anomaly so in GG3D you could say you had to turn it off and let the game get the penalty. But if its just a personal taste thing, I don't think the rating should be penalized. A lot of times I will turn AA off in order to get better framerates in 3D, but this is not necessarily because it was a problem. I just don't think it is necessary, and find the performance boost more important. Totally a personal choice.

This is where we make the distinction. In the comments section, you can recommend that DOF be turned off because you think it makes the 3D look better. However, you would not list it as a required settings reduction in the score list (i.e. actually tick it in the list) because it's not a compatibility issue where you get funky screen anomalies or problems, etc. when DOF is on. Scoring and personal taste have no relationship in GG3D (with the exception of subjective scoring and the comments section).

Regards,
Neil