Scoring Adjustment: Should We Do It?

[Neil]

Hello everyone,

In our review of Battlefield 3, we managed to get a far superior result with the DDD stereoscopic 3D drivers over what was included in the game for AMD's HD3D and Nvidia's 3D Vision platforms. The reason is that the DDD option gave full stereoscopic 3D depth to nearly all aspects of the game, whereas the EA native version had a lot more scenery in 2D or bordering on 2D (e.g. vehicle rides, certain scopes, transitions between rounds, etc.). Despite this clear improvement, the EA native version still ranked higher because DDD had some minor visual flaws and bugs that had to be acknowledged in GG3D - minor tradeoffs for a stellar result.

Should we add a score deduction for games that don't fully utilize stereoscopic 3D support excluding menus and pre-rendered cinematics? This would give DDD a legitimately higher score than their counterparts with this game. The only caveat is it's unclear if this flatness in game areas was an artistic choice - would this scoring conflict with the intentions of the game? Remember, GameGrade3D is not there to infringe on artistic choice - it's there for Quality Assurance testing.

Please answer the attached poll ASAP.

Thanks in advance,
Neil

[Guig2000]

I voted "yes" but there is a "but".
There a case where I agree that the stereo should be disabled: It's on a realistic game when the avatar of the player have a monoscopic vision. Example: a soldier wearing monoscopic night vision goggles.
The between two case example: when aiming with the scope of a rifle: A real soldier see into the scope with one eye, but he can maintain the other eye open in order to have a more global vision: the perfect realistic case here should be to have one eye which see threw the scope, the other have a black picture or or non zoomed view.

[Neil]

I voted "yes" but there is a "but".
There a case where I agree that the stereo should be disabled: It's on a realistic game when the avatar of the player have a monoscopic vision. Example: a soldier wearing monoscopic night vision goggles.
The between two case example: when aiming with the scope of a rifle: A real soldier see into the scope with one eye, but he can maintain the other eye open in order to have a more global vision: the perfect realistic case here should be to have one eye which see threw the scope, the other have a black picture or or non zoomed view.

I think that would be too complicated. What made BF3 a unique case is nearly all the vehicles were rendered in near mono when the DirectX information was readily available to do a dual camera view. Cinematics or pre-rendered material, I understand, but why would it be acceptable to have regular blasts of 2D during game play?

Regards,
Neil

[cybereality]

I think games should be penalized if they don't give players the option to adjust the 3D to their liking. Of course, there is some merit to the argument of artistic control, but with PC games usually allowing a fair amount of customization, 3D should be part of this. I am thinking more about games like Resident Evil 5 that locked the convergence controls with Nvidia. I also can see the point of restricting 3D in certain cases like looking through a sniper scope. But if the 3D is reduced (or shut off) arbitrarily and its not enjoyable to the players, then it should lose points.

The grading on GG3D shouldn't be limited to bugs or defects. Developers should be rewarded when they produce a superior 3D experience (for example with Trine 2) which has greater 3D effects and more customization options for players. Otherwise games like Deus Ex could get great scores while the 3D experience is, in reality, lackluster.

[Earmack]

My vote for "YES". Absolutely yes!

[ceashure]

I can see both sides of the issue, but there will be times were the game developer may have a legitimate reason to provide 2d. For example, lets say a cut-scene is rendered in 3D, but the camera pans into a TV screen. The view becomes the TV screen. In this case a 2D rendering should be shown. However, at the same time, if a game developer clearly disables 3D in cockpit views as you mentioned Battlefield 3 has done..they should be penalized.

[andysonofbob]

I think games should be penalized if they don't give players the option to adjust the 3D to their liking. Of course, there is some merit to the argument of artistic control, but with PC games usually allowing a fair amount of customization, 3D should be part of this. I am thinking more about games like Resident Evil 5 that locked the convergence controls with Nvidia. I also can see the point of restricting 3D in certain cases like looking through a sniper scope. But if the 3D is reduced (or shut off) arbitrarily and its not enjoyable to the players, then it should lose points.

Yup especially if this effect wouldn't be noticable in 2D (like the 2Dish vehicles in B3).

The grading on GG3D shouldn't be limited to bugs or defects. Developers should be rewarded when they produce a superior 3D experience (for example with Trine 2) which has greater 3D effects and more customization options for players. Otherwise games like Deus Ex could get great scores while the 3D experience is, in reality, lackluster.

Completely agree and this is what gamers want to see in a grading systems

BUT

as Neil has said: "GameGrade3D is [snip] there for Quality Assurance testing."

I believe this is why some gamers have had issues with GG3D. They want a 3D experience grade but GG3D looks like it has been designed to be used as an analysis tool for devs, which is fair enough. Unfortunately QA and experience ratings are different purposes and have very different audiences.

I would like 2 levels of GG3D; a GamerzGrade3D level, for experience ratings only and the GG3D

GamerzGrade3D might have the following:
the GzG3D rating;
a "My drivers are up to date" tickbox;
a relative quality rating compared to best 2D quality e.g. for best 3D experience I needed to reduce the overall graphics quality to medium;
some tick boxes for any remaining issues including, 2D crosshair, shadows, 2D unit icons etc;
and perhaps a comments box

2 - GG3D as it is.

I realise as scoring adjustment go, this is a biggy. However I think gamers would be far more inclined to contirubute to GzG3D and the hardcores would still be able to contribute to the GG3D. There is no reason why the end user's grade on GG3D couldn't be automatically added to the GzG3D database as well.

What do you think?

[Neil]

I would like 2 levels of GG3D; a GamerzGrade3D level, for experience ratings only and the GG3D

GamerzGrade3D might have the following:
the GzG3D rating;
a "My drivers are up to date" tickbox;
a relative quality rating compared to best 2D quality e.g. for best 3D experience I needed to reduce the overall graphics quality to medium;
some tick boxes for any remaining issues including, 2D crosshair, shadows, 2D unit icons etc;
and perhaps a comments box

2 - GG3D as it is.

I realise as scoring adjustment go, this is a biggy. However I think gamers would be far more inclined to contirubute to GzG3D and the hardcores would still be able to contribute to the GG3D. There is no reason why the end user's grade on GG3D couldn't be automatically added to the GzG3D database as well.

What do you think?

I think half the battle is already won with this. When making submissions, there is a subjective score portion as well as a comments section. The only problem is that to see this subjective score, you need to go into the individual submissions to see the details.

If we changed "Subjective Score" to "Personal 3D Score", and had this number included in the GG3D table outputs - would this satisfy this requirement?

I think it's important that the GG3D elements are maintained though - we can't have one without the other.

Regards,
Neil

[Neil]

Reading your message a second time, I think I should make it clear why the GG3D mechanism is important for all gamers, and not just the "hardcore" groups. There are plenty of subjective opinions out there already. Nvidia has their opinions, DDD has theirs, and I assure you - personal brand loyalty often determines what is acceptable by 3D gamers as well. A database shaped only by 3D feelings would ultimately come down to how many gamers can vote for one solution over another, and not have measurable points to debate over. I wouldn't put it past the technology makers to throw a few self promotional votes in too - I've seen it all!

The "3D Experience" grade is very much part of GG3D in the subjective score, though we can re-title and highlight it more. However, if gamers are hesitant to point out bugs and problems in a measurable way because they are scared of tarnishing their favorite brand, I don't know what value their opinion would really bring in any scoring platform.

Regards,
Neil

[andysonofbob]

Sure, I hear what you are saying but I don't know...

I still think having two levels of ratings at MTBS3D would be hugely beneficial to its mission of producing a standard. As long as you have a hard core of enthusiasts who continue to contribute objectively to GG3D an area for subjective discussion and rating is healthy. There has always been format wars and you will never be able to stop it. You could encourage positive discussion by brandishing the motto, 'let 3D gamers game' at key places of the site. I have found that has worked very nicely at school with bickering students.

Have you heard of Metacritic? It is a hugely successful site which reviews games, films, music etc. What has made it stand out is that it provides two clearly distinct catagories - Critic reviews from the professional market and User reviews. People can look at the two, often conflicting scores to gain a more complete picture of the game. Now I appreciate there are many differences between metacritic and MTBS3D but your goals are similar.

GG3D is simply not accessible to many 3D gamers; this is because it takes too long to complete. (I have used it over a period of time so I understand how to use it). If a 3D gamer is like me they have probably spent the first hour or two (probably more) tweaking their game to get the best performance/visual settings and the last thing they want to do is complete the form. I haven't made that up BTW I know (not believe) I speak for others.

If I were to redesign the GG3D section of MTBS3D. I would design the front page look exactly like metacritic. 'GzG3D' result against the GG3D result.

The data entry forms?

GG3D - no change but add a current driver tick box and use an action query to generate the driver for goodness sake!

GzG3D - one page data entry form. Score 0 - 100.
** For reference purposes, at the top, list examples of games which fit in each catagory (in order) 0-25, 26-50, 51-70, 71-85, 86-95, 96+; at the side a similar list but best at the top worst at the bottom. I would use aggregate scores for this.
** Name of game
** Their score
** Tick boxes for 3D provider and one next to each saying 'I am using latest driver' - use an action query to generate the driver.
** Overall graphics settings needed to get best 3D result. Minimum - max
** A panel of eight tick boxes on the left for details of effects needing to be turned off including: shadows, crosshair, bloom, HDR, quality shaders
** A panel of eight identical tick boxes on the right which details the remaining issues which cannot be turned off.
** At the bottom of the screen a box for comments

NOTHING ELSE apart from a link to GG3D to provide a contribution for a thorough analytical breakdown of the games 3D.

You say there are too many independant subjective reviews for 3D. So what? As long as you are the biggest. Providing a quick outlet for a gamer who wants to give a rating, gush, rant, flame etc their game AND BE HEARD on a site like MTBS3D is a powerful tool and will bring in other punters.

Think of the impact. I am convinced people would be more inclined to contribute to 'GzG3D' and if 95% of the people contributed to GzG3D only, it would still vastly increase your profile. And this will help you promote your mission. This would be a stonking amount of work I know, but I really believe in your mission. I think you have confused your audience and purpose though, they don't match. I say that with the greatest of respect.

What do you think? - again

ps You can't see how many times I have edited this can you? That would be quite embarassing the amount of times I have!

[Neil]

Hi Andy,

Thank you for investing the time in this. You have some very good ideas here on how to present the information. Maybe there are ways to get the same data in a more user friendly way?

It would be helpful if you could clarify this statement more:

GG3D is simply not accessible to many 3D gamers; this is because it takes too long to complete. (I have used it over a period of time so I understand how to use it). If a 3D gamer is like me they have probably spent the first hour or two (probably more) tweaking their game to get the best performance/visual settings and the last thing they want to do is complete the form. I haven't made that up BTW I know (not believe) I speak for others.

Regards,
Neil

[andysonofbob]

Morning!

It would be helpful if you could clarify this statement more:

GG3D is simply not accessible to many 3D gamers; this is because it takes too long to complete. (I have used it over a period of time so I understand how to use it). If a 3D gamer is like me they have probably spent the first hour or two (probably more) tweaking their game to get the best performance/visual settings and the last thing they want to do is complete the form. I haven't made that up BTW I know (not believe) I speak for others.

I will do my best

GG3D is simply not accessible to many 3D gamers

A few things:

1 - To complete the form you need to read a lengthy set of instructions and review the anomaly guide. If you do not read this carefully enough you might make mistakes or contribute a currently incompatible game like Crysis 2.

2 - On the first page of GG3D, once you have filled in your details, there is an text box for 'Stereo Driver Profile'. What is that (rhetorical)? It sounds important and I can't fill it in! This bit alone puts me off as a 3D Vision user.

3 - It is very complex (I havent a photographic memory so I cannot remember all the anomaly guide) and you have to read through each of the pages carefully to find and select the best fit description which is never easy or enjoyable.

this is because it takes too long to complete. If a 3D gamer is like me they have probably spent the first hour or two (probably more) tweaking their game to get the best performance/visual settings...

I like high FPS and I like quality graphics. Unfortunately I don't have a top end system to achieve both. For me high FPS takes precedent so I will spend time tweaking the graphics to find the best visuals for the highest FPS. This is how I do it:

1 - Find a hardware intensive early section of the game (might use a forum to help me find this) along with its benchmarking facility if present as long as it is a good one (GTA4's benchmark was rubbish, Batman AC's was good)

2 - Look for a tweak guide with benchmarks

3a - Tweak those settings using FRAPs to my FPS standard.

3b - Taking the occasional screenshot to compare quality difference to see if a drop is worth it (I never take the time to catalogue these images so I bin them because they aren't useful beyond tweak setting)

4 - As I continue to play the game i might experiment with some of the settings I wish I could enable (PhysX for example) to see if I could get away with it.

Doing the above takes some time.

and the last thing they want to do is complete the form.

Especially when GG3D scores the game differently to the 3D experience (for better or worse). I find this galling. This is down to the purpose / audience issue I believe GG3D has:

GG3D wants -
gamers to submit a detailed technical analysis of a game's 3D so it can provide a 3D score based on the technical quality of the game's 3D;

Gamers want -
a grade based on 3D experience so they can chose to buy the game or share their thoughts on a purchased game.

Sadly the two often provide conflicting results. Case in point - Prepatch 'Oil Rush' would have achieved a positive result on GG3D (it would have been only penalised for not being able to control convergence) yet arguably the lowest gamer score due to its pitiful 3D depth.

I haven't made that up BTW I know (not believe) I speak for others.

This was about the time entering the data, there are clearly plenty of posts on this forum as well as at least one blog about the last couple of points. A user on the 3D Vision forum has mentioned this to me (its good but its too much), a 3D gamer mate who plays Battlefield (OK, wrong audience admitedly, but he was interested in the scoring system and he didn't get it) and my father (interested in my hobbies and he asked the question about how 3D quality is officially rated) who joked if I any time to play the games!

There you go mate!

You once asked why 3D Vision users contributed less compared to other 3D provider users. I am a 3D Vision user, I would say my tech knowledge is sufficient to get alot out of 3D gaming and the abovementioned statements are my thoughts (viewed from the eyes of a potential 3D Vision user).

Hope that makes sense.

[Neil]

Hi Andy,

Thank you for your input.

A few things:

1 - To complete the form you need to read a lengthy set of instructions and review the anomaly guide. If you do not read this carefully enough you might make mistakes or contribute a currently incompatible game like Crysis 2.

2 - On the first page of GG3D, once you have filled in your details, there is an text box for 'Stereo Driver Profile'. What is that (rhetorical)? It sounds important and I can't fill it in! This bit alone puts me off as a 3D Vision user.

3 - It is very complex (I havent a photographic memory so I cannot remember all the anomaly guide) and you have to read through each of the pages carefully to find and select the best fit description which is never easy or enjoyable.

I don't think it's that complicated. There isn't any jargon or language that only a game developer would understand, and when reading MTBS' forums as well as DDD and Nvidia's site, the level is far more advanced than what GG3D is asking. HMDs, crazy rigs, special settings, etc. - GG3D is child's play in comparison.

The Stereo Driver profile is more applicable to DDD than anyone else. We opened it up to other solutions for instances like REGEDIT files or config files, but I agree - this is adding unnecessary confusion. We can correct the descriptive text so it doesn't create unnecessary pressures on Nvidia 3D Vision users to complete something that doesn't really exist for them.

The guide is just there for people who don't know what an anomaly is and to help them classify them. It's not a tax return.

I like high FPS and I like quality graphics. Unfortunately I don't have a top end system...

What I'm gathering is you are tweaking your game for performance more so than graphics quality. That's fine by me, and GG3D isn't there to interfere with that. If anything, GG3D will help you accomplish this goal because it tells you the maximum visual flexibility you can hope to attain, and the minimum setting reductions needed to accomplish that flexibility. GG3D isn't an artistic score, it's a QA and visual flexibility score.

I haven't read anything here that would prevent gamers from maxing their eye candy settings, and figuring out which are required to remove to get their 3D game to visually work. That's probably the most basic thing you had to do before you started tweaking for performance. There isn't anything unnatural here.

GG3D wants -
gamers to submit a detailed technical analysis of a game's 3D so it can provide a 3D score based on the technical quality of the game's 3D;

Gamers want -
a grade based on 3D experience so they can chose to buy the game or share their thoughts on a purchased game.

I think you are jumping to conclusions as to what gamers want. Look at the forum surveys on MTBS. Countless views in many cases...but how many take the simplest step of clicking a vote button? Seven?!? Limited gamer participation has nothing to do with GG3D or anything else - it's just the nature of the beast.

On the topic of Oil Rush, the choice or level of 3D depth is a measurable criteria that can be added to GameGrade3D (similar to what we did with Battlefield 3). We just didn't get around to it yet. GG3D is a continually developing and evolving service.

This was about the time entering the data, there are clearly plenty of posts on this forum as well as at least one blog about the last couple of points. A user on the 3D Vision forum has mentioned this to me (its good but its too much), a 3D gamer mate who plays Battlefield (OK, wrong audience admitedly, but he was interested in the scoring system and he didn't get it) and my father (interested in my hobbies and he asked the question about how 3D quality is officially rated) who joked if I any time to play the games!

The blogger you are thinking of has been making some of the best and most detailed GG3D submissions we have ever seen. There is still some debate about the scoring values, but the benefits to 3D gaming are clear.

For those that don't understand it, the GG3D scoring is all detailed here: http://www.mtbs3d.com/index.php?option= ... Itemid=122

It's just a Bronze to Platinum grade based on ranges of numeric scoring quality, and the letters indicate the conditions of having depth/pop-out if that's important to you. I don't think it gets any simpler than that. Maybe the descriptions or the text isn't well written enough - we can review.

I agree that the checklist format of GG3D makes it look bigger or longer than it really is. We are exploring ways to get the same data in a more digestible format.

Especially when GG3D scores the game differently to the 3D experience (for better or worse). I find this galling.

Yes, but how did the subjective score fair? Would it be equally galling to you if the person scored a game you really like with a 2 instead of a 10? I hated the Avengers in 3D and I couldn't tolerate another showing of Avatar in 3D if you paid me. Is this equally galling?

If I wasn't allowed to share these opinions, I too would have a problem with GG3D. However, GG3D lets you share these opinions and more, but it has to always maintain the measurable portion because there will be times where DDD does a better QA job, Nvidia will do a better QA job, etc. etc. A service devoted to subjective scoring would only deteriorate into how many voices are loyal to one brand over another - it's inevitable. I think this would do more harm than good.

Regards,
Neil

[andysonofbob]

OK

Hopefully we can agree to disagree.

I would like GG3D to succeed as it is, barr the natural tweaking oc. I would also like to see more hits on this site to help promote MTBS3D's cause. Frankly as long as MTBS's cause of promoting a 3D standard succeeds, I would be happy any how it is achieved. (Coercive persuasion might be a step too far though...)

I do believe providing a quick and simple, subjective rating system to compliment (not replace) GG3D will help to both increase the profile of MTBS3D as well as demonstrate the rigour and quality of GG3D. I know you are concerned with format wars but they aren't that much of a biggy - I work in a school and I run a game club where I have used Day[9]'s moto, 'Let gamer's game'; this has effectively stamped out format wars. I think the same would work for 3D.

All the best!

Andy

[Neil]

Hi Andy,

I don't wish to discount or come across as discounting these ideas. There have been some very positive developments around GG3D recently, and we have to remember that it's a continually growing and developing service. Will this include a more pronounced subjective weight or a separate track? It's too early to say - I will consider it.

Regards,
Neil