Trying to find an objective ghosting measurement

[Fredz]

Continued from this topic : Review of the 23″ Planar SA2311W 3D Vision-ready LCD Monitor

EDIT : updated with values for ghosting in the center.

I wanted to see how one of the worst DDD tests I've ever seen (with the 3D LCD TV Samsung UE46C7700) would fare in this test.



Samsung UE46C7700 scale : 220 - 0 = 220.
Samsung UE46C7700 ghosting (lower left, upper left) : 144-131=13 (5,08%) - 157-140=17 (6,64%)
Samsung UE46C7700 ghosting (center) : 19-0=19 (7,42%)
Samsung UE46C7700 ghosting with corrected scale : 5,90% (lower) - 7,72% (upper) - 8,63% (center)

Surprisingly it's not as bad as I thought, so I tested the 3D Plasma TV Samsung PS50C7790 displaying the DDD ghosting test with the 3D mode disabled to see what max ghosting could be.



Samsung PS50C7790 scale : 217 - 2 = 215.
Samsung PS50C7790 ghosting (lower left, upper left) : 130-90=40 (15,62%) - 179-127=52 (20,31%)
Samsung PS50C7790 ghosting (center) : 162-2=160 (62,5%)
Samsung PS50C7790 ghosting with corrected scale : 18,60% (lower) - 24,19% (upper) - 74,42% (center)

As I said, these tests aren't complete yet, I've to come up with something better to be truly able to compare different 3D displays in an objective way. But quite interesting for now still.

[Fredz]

Time to measure ghosting in the center as it was not really fair to the Planar to test only corners, same test for the Panasonic to compare.



Planar SA2311W scale : 254 - 14 = 240
Planar SA2311W ghosting (lower left, upper left) : 129-116=13 (5,07%) - 76-62=14 (5,47%)
Planar SA2311W ghosting (center) : 21-14=7 (2,73%)
Planar SA2311W ghosting with corrected scale : 5,42% (lower) - 5,83% (upper) - 2,92% (center)



Panasonic TX-P50VT20 scale : 170 - 4 = 166
Panasonic TX-P50VT20 ghosting (lower left, upper left) : 91-89=2 (0,78%) - 98-97=1 (0,39%)
Panasonic TX-P50VT20 ghosting (center) : 21-4=17 (6,64%)
Panasonic TX-P50VT20 ghosting with corrected scale : 1,20% (lower) - 0,60% (upper) - 10,24% (center)

As expected the ghosting in the center is much more pronounced on the Panasonic with 10,24%, the Planar is much better in this regard with only 2,92%. I wonder why there is so much difference between the center and the corners, I would have think this would have been more homogeneous.

It's also pretty weird that the Planar has less ghosting in very high contrast zones (black & white) than in zones with less contrast (black & grey and white & grey). I hope more images of the ghosting test will be available in the future to see if these measurements are really meaningfull.

[cybereality]

I am very much interested in this myself. I don't think the DDD test is necessarily the best empirical test for ghosting, although it does make for an easy image for people to look at and compare. I am not sure exactly how you came up with those numbers, but I have come up with some numbers myself (for the original Samsung image). My method is as follows:

- I turn the image into grayscale, so that I am only concerned with white/black ghosting and not color distortions.

- Using the palette in the center, I sample the pure white and pure black values so its possible to get corrected values. On that image I get 214 for white, and 1 for black (in a perfect world, white would be 255 and black would be zero). So the range is within 213 units starting from 1.

- I sampled the center area with the big L and R. This should give the maximum ghosting values: either black on white or white on black.

- I first sample the curve of the R, white is a white block over a black area. This is the "white on black" ghosting value. I get 198.

- Then I sample the black area of the L, on the bottom where it does not overlap the R. This is the "black on white" ghosting. I get 9.

- Then to get the percentage of "white on black" ghosting I take 198 - 1 = 197. But we want the difference from the ideal (which is 213) so I do 213 - 197 = 16. Then divide that by the total range: 213. So I get 16/213, which equals 0.075, or 7.5% ghosting.

- I do the same process for the "black on white" ghosting: 9-1 = 8; 8/213 = 0.0375; so we get about 3.8% ghosting.

So the total results I get for the center area are 7.5% "white on black" and 3.8% "black on white". From a numbers standpoint, that actually doesn't sound too bad (although taking samples from the sides may be worse). We would need some way to take all those values into account. Especially on displays like Nvidia 3D Vision, which are known to have issues with the tops and bottoms of the screen.

[Fredz]

I don't think the DDD test is necessarily the best empirical test for ghosting, although it does make for an easy image for people to look at and compare.

Yes, but it's the only image that seems to be often photographied on 3D displays. I'd like to see someone create another image which could allow some minimal calibration as I've proposed in another thread, but it seems nobody is interested in this. So I'll continue to use this one for now...

I am not sure exactly how you came up with those numbers, but I have come up with some numbers myself (for the original Samsung image). My method is as follows:

For each image, here is what I've done :
- open it in Gimp ;
- resize it to 793 pixels width (to have the same width than the Planar image) ;
- Image | Mode | Grey levels.

For the measurements, I checked the neighbouring average in the options and I selected a radius of 4.

Then I move the mouse on the whole part of the image I'm interested in and I choose the least favorable value depending on the context :
- WHITE : the highest value for white in the big R ;
- BLACK : the lowest value for the black rectangle in the center ;
- BLACK_GHOST_LOWER : the lowest value for lower right part of the circle in the lower left ;
- GREY_LOWER : the highest value in the grey rectangle just below this circle ;
- WHITE_GHOST_UPPER : the highest value for higher right part of the circle in the top left ;
- GREY_UPPER : the lowest value in the grey rectangle just above this circle ;
- WHITE_GHOST_CENTER : the highest value for the L part in the center.

I didn't test for BLACK_GHOST_CENTER since I found it to be barely visible in most images.

Then I do these calculations :
- SCALE = WHITE - BLACK
- LOWER_GHOST = (GREY_LOWER - BLACK_GHOST_LOWER) * 100 / 256
- LOWER_GHOST_SCALED = (GREY_LOWER - BLACK_GHOST_LOWER) * 100 / SCALE
- UPPER_GHOST = (WHITE_GHOST_UPPER - GREY_UPPER) * 100 / 256
- UPPER_GHOST_SCALED = (WHITE_GHOST_UPPER - GREY_UPPER) * 100 / SCALE
- CENTER_GHOST = (WHITE_GHOST_CENTER - BLACK) * 100 / 256
- CENTER_GHOST_SCALED = (WHITE_GHOST_CENTER - BLACK) * 100 / SCALE

With the image of the Samsung LCD, you have :
- WHITE = 214
- BLACK = 1
- WHITE_GHOST_CENTER = 9

And I have :
- WHITE = 220
- BLACK = 0
- WHITE_GHOST_CENTER = 19

You then calculate the scale and the "black on white" ghosting the same way as me, but your measured values are quite different from mine and we get different results in the end (3,75% for you, 8,63% for me). So the difference comes from our greyscale technique and the way we choose the values (worst-case for me, average for you it seems).

It's not really a problem since our own calculations should be comparable between themselves, but not with each other. We would need to choose the same algorithm for greyscale conversion, which one did you use ? And we also should decide if the measures should be worse-case or average, I choosed the former because I didn't want to choose an arbitrary average value or to calculate the average of all the values measured in a zone. It's justified when doing it by hand, but if I write an application to automate this the average value could be a better choice.

It should also be noted that our calculations can't take into account the fact that the black levels may be burned, which is the case for the Samsung LCD image (the two rightmost little black squares in the horizontal scale have almost the same value). That's why I wanted a better ghosting test, which would have allowed to avoid this.

From a numbers standpoint, that actually doesn't sound too bad (although taking samples from the sides may be worse). We would need some way to take all those values into account.

Yes, that's why I first measured ghosting in the lower and upper left, but more measurements would be needed to account for all the different types of ghosting (other parts of the circles, and doing all the same calculations for the L image).

Especially on displays like Nvidia 3D Vision, which are known to have issues with the tops and bottoms of the screen.

I think we can't do anything about that. When publishing results we should indicate which glasses were used. We can't measure absolute ghosting of a 3D display, only ghosting of a couple (3D display + glasses).

[Likay]

Some ideas if you're going to make your own ghosting test: Make an application which shows a white object on a dark surface like in "standard" ghostingtests (and vice versa of course). Then in the software you can adjust the red/green and blue (or have them linked for greyscale) portions in the evaluationfield until it matches the ghostimage perfectly. This way you'll get a figure directly defined by the software and also indirectly get an evaluation for simplier colormess too. By moving the test around the screen you can try ghosting on different parts of the screen. The corners and the middle (5 positions) are obvious while the lower/upper/left/right centre might cause too much work (9 positions...) to evaluate a screen. This is only a principle but maybe a good idea?
Problems are getting the app becoming standard, experimenting what eventual correctionvalues that should be used to show a somewhat linear figure/crosstalk comparison, interfacing with stereosolutions (direct support different 3d-hardware or d3d support for iz3d,ddd, nvidia etc...). Ps3 and xbox360 connected displays can't be evaluated unless you also compile the app for those. Maybe some other problems but anyway...

[Fredz]

I think the test should only consist in an image so anybody can try it without hassle on any platform. It's already quite difficult to make people take photos of their screens with only a test image, I don't think most of them would do it with an application. It would also need to be arguably better than the DDD test for people to use it instead of the DDD test.

I think a good amount of work would be needed to create it, in order to limit problems like burned black levels or uneven lighting. Unfortunately I'm not qualified enough to create such a test, it'll need to be done by professional testers like the ones at DigitalVersus or these kind of websites.

[Fredz]

Continuing on this idea of an application, I think the best thing would probably be to create an application that takes one photo (or two, L + R) of the DDD test as input and return a ghosting value in output. It would need to be completely automatic so the results can be objective, and it could also tell if the images are suitable for the test by analysing them (too much blur, black levels burned, uneven lighting, etc.).

The software should be able to automatically rectify the geometry of the images (radial distorsion, bad perspective, etc.) and choose the exact same zones in all the images for the measurements. The grey level values should also be measured using a meaningful method, probably by calculating the mean of the distribution of grey values in the concerned zone (equivalent to a normal distribution I guess).

[Likay]

That's also a great idea. There's need to physically take photos of the screen for the soft to analyse but that's no bigger obstacle. The human factor will be eliminated but cameras aren't perfect either. For evaluating ghosting it should work very well though.
Nothing actually makes it more difficult to analyse many parts of the screen either (just project the testimage/images at several defined spots) since the soft does it all in one blow.
As you say: Making a new testimage which put all different solutions on a fairgame test is very difficult. Just add the iz3d...

[cybereality]

I think these are all great ideas. What we should do is a combination of these. First and foremost, the test should be using one single 3D image (which can be in all standard formats JPS, PNS, MPO, and frame-compatible SBS). The user will be required to take a photo of this image from only one eye. This photo, as-is, could be uploaded and shown for comparative purposes (much like the DDD photo is used now). Additionally there will be a software which will run on a PC, that can process that image. I could write that type of software myself using Adobe AIR, which can run on Windows, OSX and Linux. Accounting for distortion can get pretty complicated, so we may have to require the images be taken from a straight-on view. They could, of course, be cropped. Then the software could automatically adjust the brightness, convert to grayscale, or whatever else we can do to try to even the playing field. Then it could take a number of samples from key areas in the image and produce some result. Ideally there should just be one number or final score, although there can be more advanced results as well. I don't think it would be especially hard to write this program once we had the specifics of the calculations.

[Fredz]

Yep, a website dedicated to that could be better than a standalone app, but I'm not a big fan of Adobe Air though (or Flash for that matter).

I think the only interaction needed with the user would be to allow him to provide one or two images and write some text (or select from a list) for the maker/model of the display and glasses. The calculations could be done server-side with some PHP for example.

A list of all displays (or the ones from specified categories) for which images were submitted could also be displayed, and other features like a ranking table or side-by-side comparisons between displays could also be added. The ghosting calculations should be quite easy to implement, I think the real problem for now is to find a way to have really comparable images with some sort of image correction.

[cybereality]

Yeah, it could be a web app as well, I just mentioned AIR because its an easy way to develop cross-platform. But it could be a Flash application on a website just as easily.