Official M3GA Calibration Thread

[sharky]

EDIT: This thread is now M3GA's calibration thread. Share your results here so we can adjust our scoring algorithm if necessary.

-----


is there a way to see the submitted games? i mean, can i check the games and see what i can buy or not? wasnt able to find a page where i can just go trough them..

[Neil]

There is no list...YET!

For now, we would like members to go through the test, and cut & paste the results in the forums (NVIDIA or iZ3D game settings forums).

We need to make sure the scoring system works consistently before we move to phase 2!

Regards,
Neil

[yuriythebest]

perhaps there should be like a single thread to post results- that'd make viewing them much faster.

[Neil]

Ok - start with this thread. Then once we know the scoring looks about right, we will transition to the secondary forums for organized results. I edited the first post in this thread to indicate this.

Regards,
Neil

[yuriythebest]

Ok, Here is my result for Battlefield 2

game : BF2
system :core 2 quad 2.4Ghz, nvidia 8600gts
driverType :iz3d
3d driver version :1.09 , 1.10b3

Recognized Anomalies:
Objects disappear in one or both eyes (with the exception of reasonably high convergence situations). that could not be romoved by reducing graphics settings

Graphics Settings to Reduce/Resolve Anomalies:
none

total score: 0

iz3d forum thread with screenshots:
http://forum.iz3d.com/viewtopic.php?t=1024" onclick="window.open(this.href);return false;

I think that rating it zero is quite unfair, because even through most of the gun disappears in one eye, the game is very immersive and enjoyable in 3d

game : UT3
system :core 2 quad 2.4Ghz, nvidia 8600gts
driverType :iz3d
3d driver version :1.09 , 1.10b3

Recognized Anomalies:
Poorly rendered reflections (water, glass).

Graphics Settings to Reduce/Resolve Anomalies:
none


total score: 9.75

In this game there were several instances of lights being rendered away from where they are supposed to be (very rare) however that option was only available for 95% of the game so the score should be lower

[Neil]

Hi Yuriy,

I think the Battlefield 2 score should remain. Those are terrible anomalies that no game developer should be proud of. Though, maybe we can add a subjective factor to the game. Like, even though the game scored horribly, maybe there is a forgiveness factor like "How willing are you to play the game in S-3D with these anomalies?" and add that as a separate weighted score.

Even then, if I was Electronic Arts, I wouldn't want my game shown like this. I still think the current score still makes sense to me.

For UT3, we can add an occasional disconnected lighting feature for smaller parts of the game. Can you share screen shots of this?

Regards,
Neil

[yuriythebest]

here:
http://forum.iz3d.com/viewtopic.php?t=1 ... tournament" onclick="window.open(this.href);return false;

perhaps users should get a choice of "specify other minor anomaly" (-0.50)
again I should point out that unless you are looking for those anomalies you probably won't notice em (an average use, that is) since they appear in only a few instances
EDIT: I just noticed in blacksharks post that there are also problems with shadows, so the score should be slightly lower.

[Neil]

I tried to duplicate those results, but couldn't. If I had specific maps, I could try.

Regards,
Neil

[yuriythebest]

I tried to duplicate those results, but couldn't. If I had specific maps, I could try.

Regards,
Neil

DM-Sanctuary ships with the game (Second screenshot)

[DickDastardly]

Hi,
Love the idea of the tool (and the name) -it'll be extremely useful for both prospective purchasers of games and game developers themselves to have a quantifiable assessment of a game's performance in 3D. Having said that, some of the questions seem a bit vague and could perhaps do with a bit of clarification. For example:

1. It's not clear to me how the following anomalies should be assessed in relation to the "apply to 95% or more of the game" rule:

-A game which contains a total of 100 levels, 4 of which are completely unplayable in 3D.
-A game which contains a total of 100 models (or textures), 4 of which have glaring anomalies in 3D.
-A game which features 100 weapons, 4 of which are sniper rifles with scopes which can't be used at all in 3d.
-A game which contains 100 levels, 4 of which contain a minor visual anomaly at one particular point within the level.

If I understand the current scoring system correctly then all of the above examples might receive a 10/10 score for 3d. This is imho a very very bad thing. To score 10/10 a game should be perfect in 3d with all graphics options enabled and no anomalies whatsover, so I'd remove the "95%" bit entirely. (I'd also limit a 9/10 score to games which run in perfect 3d when some very minor compromises are made wrt graphics options, and 8/10 to games which run in perfect 3d after more significant reductions in graphics options. I don't think any game which has any unfixable anomalies in 3d should ever score higher than a 7).

2. I also think it would be a good idea to remove completely any subjective terms like "seriously" as in "depth of an object is seriously inconsistent to its actual location". Either an object renders correctly or it doesn't. If it doesn't, then score penalties should apply, whether it's off my a meter or a mile.

3. There are a couple of occasions when "2d" is used in a description where it would be less misleading to say "rendered at the wrong depth". For example, just about all games render fire effects or very distant terrain on a 2d sprite or surface. So long as this 2d effect is drawn at the correct depth in the game relative to other objects this is fine and shouldn't detract from the score. The problem is only when such effects are rendered at the wrong depth (generally at screen depth).

4. "FPS performance is less than 30% of the 2D mode performance (whilst driver is resident)". This is far too forgiving. 50% should be the minimum acceptable frame rate before significant score penalties apply. And what is meant by "whilst the driver is resident"? The 3d performance should be compared to running the game in 2d with the driver totally disabled (and all other settings the same). Comparing 3D to 2D with the 3D driver enabled in the nVidia control panel but not currently turned on with CTRL+T (which I think is what is meant by the "resident" above) is pointless as the 3D driver lowers fps even when not currently running. The only valid comparison is to playing in 2D with the 3D driver completely disabled in the control panel.

Also, why not incorporate the actual percentage reduction in fps in the score instead of having an arbitrary cut-off point like 30%. All other things being equal, a game which runs at 70% of 2d framerate should get a better score than one which runs at 50% of 2d framerate. This isn't currently the case.



There are also a few questions which aren't currently in the utility but could be worth adding. For example:

1. For what percentage of the time you spent playing the game were no anomalies whatsoever visible in 3d?

2. Are there any points in the game where 3D must be turned off to progress past a certain point? (eg a vital hint message which can't be read when playing in 3d or a level which simply crashes unless run in 2d).

3. If some settings must be adjusted to play in 3D without anomalies, which of the following applies:

A. The game offer a "3d mode" option within its menus which makes all the necessary changes for you.
B. Appropriate settings for 3d can be set by adjusting a series of options in the game's menus.
C. Appropriate settings for 3d can only be set by editing configuration files with a text editor or typing commands into an in-game console.

Cheers,
DD

P.S. It would be useful to show the weighting that has been applied to every question (at least during the testing phase). Some of them curently don't tell you how they impact the score so we can't give feedback on them without running the utility dozens of times changing one answer each time.

P.P.S. It might also be interesting to add a final question just before the overall score is revealed: "How would you rate the 3D in the game overall out of 10?" This could be useful in callibrating the utility.

[Neil]

This is excellent feedback.

I want to see more of this.

Regards,
Neil

[Neil]

1. It's not clear to me how the following anomalies should be assessed in relation to the "apply to 95% or more of the game" rule:

-A game which contains a total of 100 levels, 4 of which are completely unplayable in 3D.
-A game which contains a total of 100 models (or textures), 4 of which have glaring anomalies in 3D.
-A game which features 100 weapons, 4 of which are sniper rifles with scopes which can't be used at all in 3d.
-A game which contains 100 levels, 4 of which contain a minor visual anomaly at one particular point within the level.

If I understand the current scoring system correctly then all of the above examples might receive a 10/10 score for 3d. This is imho a very very bad thing. To score 10/10 a game should be perfect in 3d with all graphics options enabled and no anomalies whatsover, so I'd remove the "95%" bit entirely. (I'd also limit a 9/10 score to games which run in perfect 3d when some very minor compromises are made wrt graphics options, and 8/10 to games which run in perfect 3d after more significant reductions in graphics options. I don't think any game which has any unfixable anomalies in 3d should ever score higher than a 7).

There are two types of anomalies. Anomalies that are so bad, you can't play a game from beginning to end (the 95% rule), and the occasional anomaly which doesn't seriously hinder the game, but exists nonetheless.

Can you think of a way to differentiate the two? Remember that most people could rate a game without playing it from beginning to end. They may play 10 hours of it, and not realize there is a single problem on the last level.

How about:

1. Please tick all the following anomalies that consistently appear from one level to the next in an estimated 95% or more of this game when creating a COMBINED DEPTH AND POP-OUT experience. Do not select anomalies that are successfully...

OR:

Look at the critical anomalies list. These are problems that should not appear on one screen and disappear on another. The HUD will always split, the doubled image will always be there, the blur effect in one eye will be consistent. I know this from experience. Are there anomalies in this list that are more likely to be scene specific? Maybe the solution is to take certain problems off the list, and move them to the tier 2 or individually scored list.

2. I also think it would be a good idea to remove completely any subjective terms like "seriously" as in "depth of an object is seriously inconsistent to its actual location". Either an object renders correctly or it doesn't. If it doesn't, then score penalties should apply, whether it's off my a meter or a mile.

We can look at this. If we have image samples, that may explain this enough.

3. There are a couple of occasions when "2d" is used in a description where it would be less misleading to say "rendered at the wrong depth". For example, just about all games render fire effects or very distant terrain on a 2d sprite or surface. So long as this 2d effect is drawn at the correct depth in the game relative to other objects this is fine and shouldn't detract from the score. The problem is only when such effects are rendered at the wrong depth (generally at screen depth).

I agree with this. An image sample will help, but I agree with this.

4. "FPS performance is less than 30% of the 2D mode performance (whilst driver is resident)". This is far too forgiving. 50% should be the minimum acceptable frame rate before significant score penalties apply. And what is meant by "whilst the driver is resident"? The 3d performance should be compared to running the game in 2d with the driver totally disabled (and all other settings the same). Comparing 3D to 2D with the 3D driver enabled in the nVidia control panel but not currently turned on with CTRL+T (which I think is what is meant by the "resident" above) is pointless as the 3D driver lowers fps even when not currently running. The only valid comparison is to playing in 2D with the 3D driver completely disabled in the control panel.

Also, why not incorporate the actual percentage reduction in fps in the score instead of having an arbitrary cut-off point like 30%. All other things being equal, a game which runs at 70% of 2d framerate should get a better score than one which runs at 50% of 2d framerate. This isn't currently the case.

I see where you are coming from. Let me explain my logic:

When some stereoscopic 3D drivers are resident, some performance is lost even in 2D mode because memory is reserved for S-3D mode. So, I'm saying resident meaning S-3D capability is on standby with the tap of a button.

The 50% expectation you have shared is based on a zero overhead from the driver. Nothing for the extra work needed to calculate the left and right images let alone the display of them. Even NVIDIA with their deep GPU access has had performance drops to just 40% of the 2D equivalent.

Game to game performance is also very different.

I went with 30% because it's low enough to take into account all the possible handicaps, and high enough that it demonstrates the performance is hardware limited rather than programming limited. I don't want it to be necessary for people to do too much benchmarking because that could hold back participation if it is a requirement.

There are also a few questions which aren't currently in the utility but could be worth adding. For example:

1. For what percentage of the time you spent playing the game were no anomalies whatsoever visible in 3d?

We have something in mind for this, but it will be a little different. Remember, what if the game hasn't been completed when this data is entered?

2. Are there any points in the game where 3D must be turned off to progress past a certain point? (eg a vital hint message which can't be read when playing in 3d or a level which simply crashes unless run in 2d).

I have to think about this one. We don't want a hardware bug or an individual's specific computer setup to reflect on the game's rating when it could work properly on 100 other machines.

3. If some settings must be adjusted to play in 3D without anomalies, which of the following applies:

A. The game offer a "3d mode" option within its menus which makes all the necessary changes for you.
B. Appropriate settings for 3d can be set by adjusting a series of options in the game's menus.
C. Appropriate settings for 3d can only be set by editing configuration files with a text editor or typing commands into an in-game console.

I think this is more of a native S-3D support type scenario. We can think about this.

P.S. It would be useful to show the weighting that has been applied to every question (at least during the testing phase). Some of them curently don't tell you how they impact the score so we can't give feedback on them without running the utility dozens of times changing one answer each time.

We can implement this. Though, I think it indicates a score count at the top.

P.P.S. It might also be interesting to add a final question just before the overall score is revealed: "How would you rate the 3D in the game overall out of 10?" This could be useful in callibrating the utility.

Perhaps a subjective portion of the score. Have to think about this. The more fact based it is, the better.

Regards,
Neil

[DickDastardly]

There are two types of anomalies. Anomalies that are so bad, you can't play a game from beginning to end (the 95% rule), and the occasional anomaly which doesn't seriously hinder the game, but exists nonetheless. Can you think of a way to differentiate the two?

How about:

Please tick any of the following anomalies which you've seen whilst playing the game and weren't able to eliminate by adjusting graphical options:

-Objects or effects which are only visible with one eye (not counting those at the extreme left or right edge of the screen)
-Objects or effects which are rendered at an incorrect depth
-Flashing or flickering textures
-Light effects which are disconnected from their sources
-Shadows which are disconnected from the object casting them
-Distorted models (example pic here).
-HUD severely split or invisible when convergence and seperation are set to give the best 3d in the underlying scene
-Labels or icons rendered at an inappropriate depth eg player tags rendered at screen depth instead of at the depth of the corresponding character model.
-Backgrounds or skyboxes rendered at an incorrect depth (eg distant mountains which appear to be in front of nearby buildings)
-Reflections which don't look correct in 3D.

Under each item you could offer several choices so users could assess the severity of each problem. It's very hard to entirely remove subjective language here, but they might be labelled something like this:

-Extremely rare or subtle (unlikely to be noticed unless you were specifically looking for it).
-Occasional/noticeable (but didn't significantly detract from enjoyment of the 3d experience).
-Frequent/annoying
-So bad that you'd generally prefer to play in 2D
-Impossible to play in 3D

Remember that most people could rate a game without playing it from beginning to end. They may play 10 hours of it, and not realize there is a single problem on the last level.......If we have image samples, that may explain...

Love the idea of having an image sample for each problem -I think that would be a big help. As for the fact that not all reviewers will have played the game all the way through, I don't see this as a major problem. Even if someone had completed the game, they might not have seen every part of it due to branching paths etc so people will just have to rate those parts they've seen (though obviously they should be discouraged in the utility's preamble from rating a game they've not played for long).

One thing I'd really like to stress, though, is that I think it's crucial to keep standards high - a 10/10 score on M3GA should only ever be given to a game which is completely perfect in 3D (no matter what graphical options are selected). If someone sees "Rated 10/10 by M3GA by all reviewers" then they should be able to buy the game in complete confidence that there'll be no problems whatsoever. Already there are several posts on the nVidia forums from people complaining that they bought a game which nVidia gave an "Excellent" 3d rating to, only to discover major anomalies. I wouldn't want the M3GA ratings to become similarly meaningless by being too lax - that wouldn't benefit users or developers.

As for FPS, I still think that some kind of numerical assessment would be better than having an abitrary cut-off point. Perhaps you could ask something like this:

What was the average percentage loss in fps when playing in 3d compared to playing in 2d with exactly the same graphics settings and with the 3d driver disabled in the control panel (not just by pressing CTRL+T)?

Then you could ask how they arrived at the number, giving less weight to the percentage for each successive answer:

-FPS tested using an in-game recording, track or timedemo which gives an repeatable average fps figure over a period of time.
-FPS tested by comparing the value of the frame rate counter in several scenes to the same scenes in 2D.
-There is no frame rate counter but I estimated the difference.

I have to think about this one. We don't want a hardware bug or an individual's specific computer setup to reflect on the game's rating when it could work properly on 100 other machines.

I don't think you'll ever be able to entirely eliminate the problem of games being rated unfairly by users who haven't configured their setup properly or who have an unrelated hardware or software issue causing problems, but hopefully if enough people rate a game then these incorrect ratings will be lost in the overall average. It might also be worthwhile to provide a link for users to follow to some guidelines or a tutorial on how to set up 3D correctly before they rate a game with M3GA.

(Responding to the suggestion of a question about the steps required to setup the game for 3D) I think this is more of a native S-3D support type scenario. We can think about this.

This was to determine the "PITA" factor in getting the 3d working. There are a lot of users who might be happy to adjust a couple of menu options in the game's GUI but would baulk at the prospect of having to manually edit configuration files.

Perhaps a subjective portion of the score. Have to think about this. The more fact based it is, the better.

I wasn't really suggesting that the figure should be incorporated into the score, more that it might help assess how each problem should be weighted. Eg if one game suffers from anomaly A and is consistently given a high subjective rating by users and another suffers from anomaly B but is consistently given a low rating then that would tell you that anomaly B should be given more weight in the scoring algorithm than A.

Cheers,
DD

[Neil]

Hi DickDastardly,

You added an anomaly or two that are very critical - so we will add them to the list (e.g. distorted models).

I don't know about the sub-options. I have to think about it. The problem is some gamers will artificially boost their score or ratings just because they like one solution over another. We need something so iron clad, it greatly reduces their chances of doing this and still hold up to public scrutiny. The original structure may handle this already. The 95% rule differentiates the consistent errors, and the occasional errors that are scored individually.

I think the FPS rating is too complicated. If it's too complicated, it will prevent participation - and there isn't enough to be gained by this question.

I agree that the 10/10 score is for flawless games. If the questions are answered honestly, a true 10/10 should be hard to come by.

I'm still having trouble with the subjective scoring. Let me think about it.

Regards,
Neil