New M3GA Scoring Mechanism - Need Some Feedback

[Neil]

We are pleased to report that some game developers have been sharing positive feedback about M3GA. In particular, they really like the separate scoring between MTBS certification, and strict deduction based numeric scoring now shown in a separate column.

For the numeric side, out of a possible score of 100%, it now follows a weight of 25 points for setting reductions, 25 points for non-critical errors, and a full 50% for critical anomalies. Each section is protected that if the anomalies add up to more than their individual section weight (e.g. 50 points out of 50), their deductions max out to zero, not negative figures. In theory, you can have Gold certified games scoring lower than Silver rated games because the rules are different on the numeric side. This is great for gamers wishing to go by a different quality expectation scale.

There is a remaining challenge, though. Usually games don't have 15 anomalies or setting reductions at one time, so out of 100, the deductions currently have a minimal numeric impact. Should we increase the value of each deduction choice with an understanding that it will be possible to hit the section's zero mark before all the options are selected? For example, let's say a section is valued at 25 points. Should a deduction be valued at 10 points, with an understanding that even if three or four points are selected, the maximum reduction will be 25 points for that given section?

See what you think.

Regards,
Neil

[Likay]

As is now: Do the analyzer simply add the percentages of anomalies? For instance: If a game has two major anomalies it gives 100-50-50=0%. However you can make the anomalies make a percentage impact on the final score. For instance: A game has two major (50%)and two minor (25%) issues. This gives the total sum: 50%x50%x75%(because it minor it has to be 100-25%)x75%=appr 14% which is pretty bad (unplayable)
If a game has three minor issues @25% it gives 75%^3=appr 42% (acceptable)
You will never have negative figures but each and every anomaly counts in to the final result as a factor.

A very critical anomaly can be rated 100% and will then give the total point 0 regardless of anything else.

Just an idea but maybe it's already calculated this way?

[Neil]

It works on a subtraction basis. All games start at 100%, and each anomaly that is selected reduces the score. As it is, when you add all the anomalies together, they won't add up to more than 100%.

The critical anomalies are more valuable and carry more weight within the 50%.
The game settings and non-critical anomalies have less value, and each sections adds up to no more than 25% (25% X 2 = 50% total score).

Regards,
Neil

[cybereality]

I think for general cases that system sounds good but in certain cases it falls apart. For example, on Street Fighter 4 with the DDD driver the game starts out looking perfect but when someone does a super move then all the characters disappear in one eye and stay like that. Clearly the game is unplayable with such a serious bug like that. However there is no way to indicate the severity of this unacceptable show-stopper bug. I chose "Objects appear in one eye" and it only took like 5% off giving it a Gold rating which is clearly not right. There should be some way to accommodate issues like this.

[Neil]

Check the scoring again:

http://mtbs3d.com/m3ga/index.php?show=m ... id&val=342" onclick="window.open(this.href);return false;

One of the benefits of M3GA is when we do updates like this, the older entries get refreshed so games don't have to be resubmitted.

Regards,
Neil

[cybereality]

Ok cool, that works then.

[Neil]

So does this mean we are good to go for the next stage?

I'm putting a request through to show the numeric penalty by each entry so it's easy to see how the numeric scores came to be.

Speak now or forever hold your peace...not piece:



Regards,
Neil

[BlackShark]

Well I'm not that sure about that principle of 25-25-50% the scoring that results looks very wrong to me.
At the moment almost every game ranks somewhere between 80% and 100% with a vast majority of them above the 90% mark, including many game that have big issues.

The issue caused by separating the different sections is that no game, even the worst game ever will ever get a 0% ranking. At the moment, the absolute worst game in the m3dga has a ridiculous high 50% due to this scoring method. It's like the sunday kids singing shows, everybody is nice everybody is great, everybody did some good work, everybody gets a present.
If we want to have some credibility, the system should be able to tell the developers that something is very wrong with their game engines, and for that the system should be able to score low if the issues are severe.

I believe that a better scoring system would have a total amount of points much greater than 100%, I recommend to double your points attribution to reach a total possible loosable points of 200%.
Critical anomalies should have a maximum influence of 100%, these are the bad stuff we don't want to see (driver not injecting = no 3D = score 0%), this should make the developers aware that there is a major issue with their games that they absolutely must do something about it
Non critical anomalies should have a maximum influence of 50%, they're annoying issues, we'd really like them away
and settings reduction should have a maximum influence of 50%, low graphics are only half the fun, come on devs ! You can do better than that.

[Neil]

I think we are thinking alike because most games don't have 20 anomalies, so the scoring isn't getting reduced as much as it should.

What we can do is treat it as uniform scoring as you suggest, increase the score values for each selection, and give some factors very high grades. I don't understand the benefit of having double the points, though - I'm not understanding the benefit.

Regards,
Neil

[BlackShark]

Double the loosable points :

Games start a +100%, they can loose up to 200 calculated points(%) if there are issues everywhere (critical + non critical + graphic reductions). But M3DGA will only show scoring between 0% and +100% (all negative scores clamped at 0%)

[Neil]

It's easier to make it out of 100 and work with decimals if necessary. It would work out the same way.

Thank you for your continued contributions. As soon as the scoring is updated, I'll let you know.

Regards,
Neil

[BlackShark]

I think you didn't get exactly the issue I'm taking about or my request.

My issue with the present system isn't about the current score, it's about the system that prevents a game from getting a 0%

The current system splits the 100% in 3 categories like you'd cut a cake with one part for each guest.
Each category has it's own share of % and anomaly in one category only affects it's own share. This means that no matter how many critical issues can happen in one category, the game score can be saved by other categories.
Statistically speaking, this means that very few game (if any) will ever manage to score 0% in all 3 catagories at the same time.

Imagine a game that have absolutely every single issue possible on which you can't improve anything with graphics reductions.
This is the worst game in the world, it should get a 0%, however the share system it would get a miraculous 25% because of the graphics reductions

[Neil]

I completely get it, don't worry. You will have your 0%.

We will lose the weight score between sections, everything will be out of 100, and it will be possible with a single selection to have dramatic score drops (e.g. critical errors that drop the score to 0 right away).

Regards,
Neil

[BlackShark]

Thanks.



(but i'm quite panicked, your smiley misses two eyes )

[Neil]

It's from Hitchhiker's Guide to the Galaxy:

http://en.wikipedia.org/wiki/Hitchhiker ... the_Galaxy" onclick="window.open(this.href);return false;

Required reading!

Regards,
Neil

[Likay]

Off topic but: Definitely worth the reading. The movie doesn't even come close to what it's meant to be.

[Neil]

Ok! The scoring has been updated!

Here is the reference story: http://www.mtbs3d.com/index.php?option= ... 8&catid=35" onclick="window.open(this.href);return false;

Do you like?

Regards,
Neil

[Neil]

Blackshark!

Is the scoring what you were describing? Are we ready for stage 2?

Regards,
Neil

[BlackShark]

Yes this scoring now looks much better.
There might be some minor tweaks later as the database grows and we find some more fine tuning but it looks on the good track.

I noticed that setting reduction looses less points than letting the issue happen 'which is the correct way to do it) it looks great so far.

What's phase 2 ?

[Neil]

Ahah! Praise from our toughest critic! Excellent!

For Phase 2, we need a new thread.

Regards,
Neil

[Guig2000]

When looking to my submissions, the scoring seem adequate.

A last suggestion: (witch will change nothing for the moment):
There is one issue witch is present with all games and drivers: no one is able to cut "popping-out of screen objects" in order to avoid them to be clipped by the screen borders.
A driver/ game with this ability should have a bonus on the score.

[Neil]

Well, we can look into scoring bonuses, or special features that add scores rather than take them away.

What would be good examples of this?

Regards,
Neil

[BlackShark]

I can't think of any bonuses for games through 3D drivers.

But i can think of bonuses if the game has a native mode (which M3DGA needs to add to the database separately from the other drivers)

-Native support for a wide variety of 3D displays without requiring drivers (put list of 3D outputs here)
-Menus taking advantage of 3D
-Cutscenes occasionally taking advantage of pop-out effects (not necessarily often, a game with a serious realistic context wouldn't use much of them)
-Specialized camera management and or automatic convergence according to the scene shown/camera angle/cutscene (prevents objects from uncomfortably too separated when close to the camera)

[Neil]

I knew we forgot something!

Ok, will review.

Regards,
Neil