It is currently Sun Dec 08, 2019 2:19 pm



 [ 24 posts ] 
 New M3GA Scoring Mechanism - Need Some Feedback 
Author Message
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
We are pleased to report that some game developers have been sharing positive feedback about M3GA. In particular, they really like the separate scoring between MTBS certification, and strict deduction based numeric scoring now shown in a separate column.

For the numeric side, out of a possible score of 100%, it now follows a weight of 25 points for setting reductions, 25 points for non-critical errors, and a full 50% for critical anomalies. Each section is protected that if the anomalies add up to more than their individual section weight (e.g. 50 points out of 50), their deductions max out to zero, not negative figures. In theory, you can have Gold certified games scoring lower than Silver rated games because the rules are different on the numeric side. This is great for gamers wishing to go by a different quality expectation scale.

There is a remaining challenge, though. Usually games don't have 15 anomalies or setting reductions at one time, so out of 100, the deductions currently have a minimal numeric impact. Should we increase the value of each deduction choice with an understanding that it will be possible to hit the section's zero mark before all the options are selected? For example, let's say a section is valued at 25 points. Should a deduction be valued at 10 points, with an understanding that even if three or four points are selected, the maximum reduction will be 25 points for that given section?

See what you think.

Regards,
Neil


Fri May 14, 2010 9:48 am
Petrif-Eyed
User avatar

Joined: Sat Apr 07, 2007 4:34 pm
Posts: 2909
Location: Sweden
As is now: Do the analyzer simply add the percentages of anomalies? For instance: If a game has two major anomalies it gives 100-50-50=0%. However you can make the anomalies make a percentage impact on the final score. For instance: A game has two major (50%)and two minor (25%) issues. This gives the total sum: 50%x50%x75%(because it minor it has to be 100-25%)x75%=appr 14% which is pretty bad (unplayable)
If a game has three minor issues @25% it gives 75%^3=appr 42% (acceptable)
You will never have negative figures but each and every anomaly counts in to the final result as a factor.

A very critical anomaly can be rated 100% and will then give the total point 0 regardless of anything else.

Just an idea but maybe it's already calculated this way?

_________________
Mb: Asus P5W DH Deluxe
Cpu: C2D E6600
Gb: Nvidia 7900GT + 8800GTX
3D:100" passive projector polarized setup + 22" IZ3D
Image


Last edited by Likay on Sun May 16, 2010 9:52 am, edited 1 time in total.



Fri May 14, 2010 10:20 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
It works on a subtraction basis. All games start at 100%, and each anomaly that is selected reduces the score. As it is, when you add all the anomalies together, they won't add up to more than 100%.

The critical anomalies are more valuable and carry more weight within the 50%.
The game settings and non-critical anomalies have less value, and each sections adds up to no more than 25% (25% X 2 = 50% total score).

Regards,
Neil


Fri May 14, 2010 10:26 am
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
I think for general cases that system sounds good but in certain cases it falls apart. For example, on Street Fighter 4 with the DDD driver the game starts out looking perfect but when someone does a super move then all the characters disappear in one eye and stay like that. Clearly the game is unplayable with such a serious bug like that. However there is no way to indicate the severity of this unacceptable show-stopper bug. I chose "Objects appear in one eye" and it only took like 5% off giving it a Gold rating which is clearly not right. There should be some way to accommodate issues like this.

_________________
check my blog - cybereality.com


Fri May 14, 2010 8:34 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Check the scoring again:

http://mtbs3d.com/m3ga/index.php?show=m ... id&val=342

One of the benefits of M3GA is when we do updates like this, the older entries get refreshed so games don't have to be resubmitted.

Regards,
Neil


Fri May 14, 2010 8:42 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Ok cool, that works then.

_________________
check my blog - cybereality.com


Fri May 14, 2010 8:57 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
So does this mean we are good to go for the next stage?

I'm putting a request through to show the numeric penalty by each entry so it's easy to see how the numeric scores came to be.

Speak now or forever hold your peace...not piece:

Image

Regards,
Neil


Sat May 15, 2010 2:04 pm
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
Well I'm not that sure about that principle of 25-25-50% the scoring that results looks very wrong to me.
At the moment almost every game ranks somewhere between 80% and 100% with a vast majority of them above the 90% mark, including many game that have big issues.

The issue caused by separating the different sections is that no game, even the worst game ever will ever get a 0% ranking. At the moment, the absolute worst game in the m3dga has a ridiculous high 50% due to this scoring method. It's like the sunday kids singing shows, everybody is nice everybody is great, everybody did some good work, everybody gets a present.
If we want to have some credibility, the system should be able to tell the developers that something is very wrong with their game engines, and for that the system should be able to score low if the issues are severe.

I believe that a better scoring system would have a total amount of points much greater than 100%, I recommend to double your points attribution to reach a total possible loosable points of 200%.
Critical anomalies should have a maximum influence of 100%, these are the bad stuff we don't want to see (driver not injecting = no 3D = score 0%), this should make the developers aware that there is a major issue with their games that they absolutely must do something about it
Non critical anomalies should have a maximum influence of 50%, they're annoying issues, we'd really like them away
and settings reduction should have a maximum influence of 50%, low graphics are only half the fun, come on devs ! You can do better than that.

Image

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Sun May 16, 2010 7:42 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
I think we are thinking alike because most games don't have 20 anomalies, so the scoring isn't getting reduced as much as it should.

What we can do is treat it as uniform scoring as you suggest, increase the score values for each selection, and give some factors very high grades. I don't understand the benefit of having double the points, though - I'm not understanding the benefit.

Regards,
Neil


Sun May 16, 2010 8:17 am
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
Double the loosable points :

Games start a +100%, they can loose up to 200 calculated points(%) if there are issues everywhere (critical + non critical + graphic reductions). But M3DGA will only show scoring between 0% and +100% (all negative scores clamped at 0%)

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Sun May 16, 2010 8:24 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
It's easier to make it out of 100 and work with decimals if necessary. It would work out the same way.

Thank you for your continued contributions. As soon as the scoring is updated, I'll let you know.

Regards,
Neil


Sun May 16, 2010 8:56 am
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
I think you didn't get exactly the issue I'm taking about or my request.

My issue with the present system isn't about the current score, it's about the system that prevents a game from getting a 0%

The current system splits the 100% in 3 categories like you'd cut a cake with one part for each guest.
Each category has it's own share of % and anomaly in one category only affects it's own share. This means that no matter how many critical issues can happen in one category, the game score can be saved by other categories.
Statistically speaking, this means that very few game (if any) will ever manage to score 0% in all 3 catagories at the same time.

Imagine a game that have absolutely every single issue possible on which you can't improve anything with graphics reductions.
This is the worst game in the world, it should get a 0%, however the share system it would get a miraculous 25% because of the graphics reductions

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Sun May 16, 2010 9:22 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Image

I completely get it, don't worry. You will have your 0%.

We will lose the weight score between sections, everything will be out of 100, and it will be possible with a single selection to have dramatic score drops (e.g. critical errors that drop the score to 0 right away).

Regards,
Neil


Sun May 16, 2010 9:36 am
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
Thanks.



(but i'm quite panicked, your smiley misses two eyes :shock: )

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Sun May 16, 2010 10:08 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
It's from Hitchhiker's Guide to the Galaxy:

http://en.wikipedia.org/wiki/Hitchhiker ... the_Galaxy

Required reading!

Regards,
Neil


Sun May 16, 2010 10:40 am
Petrif-Eyed
User avatar

Joined: Sat Apr 07, 2007 4:34 pm
Posts: 2909
Location: Sweden
Off topic but: Definitely worth the reading. The movie doesn't even come close to what it's meant to be.

_________________
Mb: Asus P5W DH Deluxe
Cpu: C2D E6600
Gb: Nvidia 7900GT + 8800GTX
3D:100" passive projector polarized setup + 22" IZ3D
Image


Sun May 16, 2010 2:14 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Ok! The scoring has been updated!

Here is the reference story: http://www.mtbs3d.com/index.php?option= ... 8&catid=35

Do you like?

Regards,
Neil


Tue May 18, 2010 9:11 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Blackshark!

Is the scoring what you were describing? Are we ready for stage 2?

Regards,
Neil


Wed May 19, 2010 7:04 am
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
Yes this scoring now looks much better.
There might be some minor tweaks later as the database grows and we find some more fine tuning but it looks on the good track.

I noticed that setting reduction looses less points than letting the issue happen 'which is the correct way to do it) it looks great so far.

What's phase 2 ?

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Last edited by BlackShark on Wed May 19, 2010 11:53 am, edited 1 time in total.



Wed May 19, 2010 11:52 am
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Ahah! Praise from our toughest critic! Excellent! :mrgreen:

For Phase 2, we need a new thread.

Regards,
Neil


Wed May 19, 2010 11:53 am
Cross Eyed!
User avatar

Joined: Wed Nov 25, 2009 9:47 am
Posts: 158
Location: Bordeaux, France
When looking to my submissions, the scoring seem adequate.

A last suggestion: (witch will change nothing for the moment):
There is one issue witch is present with all games and drivers: no one is able to cut "popping-out of screen objects" in order to avoid them to be clipped by the screen borders.
A driver/ game with this ability should have a bonus on the score.

_________________
Image


Fri May 21, 2010 2:23 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
Well, we can look into scoring bonuses, or special features that add scores rather than take them away.

What would be good examples of this?

Regards,
Neil


Sun May 23, 2010 3:08 pm
Certif-Eyable!
User avatar

Joined: Sat Dec 22, 2007 3:38 am
Posts: 1155
Location: Montpellier, France
I can't think of any bonuses for games through 3D drivers.

But i can think of bonuses if the game has a native mode (which M3DGA needs to add to the database separately from the other drivers)

-Native support for a wide variety of 3D displays without requiring drivers (put list of 3D outputs here)
-Menus taking advantage of 3D
-Cutscenes occasionally taking advantage of pop-out effects (not necessarily often, a game with a serious realistic context wouldn't use much of them)
-Specialized camera management and or automatic convergence according to the scene shown/camera angle/cutscene (prevents objects from uncomfortably too separated when close to the camera)

_________________
Passive 3D forever !
DIY polarised dual-projector setup :
2x Epson EH-TW3500 (2D 1080p)
Xtrem Screen Daylight 2.0, for polarized 3D
3D Vision gaming with signal converter : VNS Geobox 501


Sun May 23, 2010 4:23 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Wed Dec 31, 1969 6:00 pm
Posts: 5721
I knew we forgot something!

Ok, will review.

Regards,
Neil


Sun May 23, 2010 4:52 pm
Display posts from previous:  Sort by  
   [ 24 posts ] 

Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.