The Nine Shameful Sins of S-3D Hardware Review Journalism

Conclusion
Introduction

Meant to be Seen doesn’t do hardware reviews because we believe it will put us in a conflict of interest and give unfair advantages to different manufacturers. More than that, we are an advocacy group, and we want to see growth throughout the S-3D industry.

However, even without our input, there are countless websites and blogs that have published 3D display hardware reviews for some time – but who’s reviewing the reviewers?

With new and complex stereoscopic 3D technology being heavily marketed in the home, we are concerned that many media sources are printing stories and opinions that are based on false data, preconceived notions, and product positioning.

In the interests of protecting the consumer and industry alike, MTBS has written this guide so readers will have an instant sense of whether or not 3D display hardware reviews are being written fairly and with your interests at heart.

We appreciate that this guide may make one manufacturer appear more ethical than another. This is not our intention or the subject of this article. In fact, the display manufacturers should not have any influence or responsibility in how reviews are written or printed. What is up for discussion is whether or not display reviewers and media are following the correct ethical practices when reviewing stereoscopic 3D displays for their audience, and what telltale signs readers should look out for if this is not the case.

Understanding The Problem

To be fair, reviewing 3D hardware is probably one of the toughest tasks any technology reporter can face. It’s easier to review a car or a computer graphics card because people understand what the benchmarks are, but what is 3D? How do you explain it to people who have never even been to an IMAX 3D theater or experienced S-3D gaming firsthand?

This presents a double challenge. Apart from having to review the hardware itself, journalists are also burdened with having to explain what stereoscopic 3D actually is, and while they indeed may succeed in making people think it’s “cool” and “interesting”, the fact is it’s impossible to demonstrate it to users who are stuck with a traditional 2D monitor.

Therefore, explaining the intricacies of a 3D display to the casual viewer is a seemingly daunting, even impossible task. In fact, very often the review is the first time the writer has had firsthand experience with 3D devices, and this leads to a very public and very influential learning curve.

So how do you recognize the difference between a reviewer who has done his homework, and a reviewer who is playing with fire?

The Nine Shameful Sins of S-3D Hardware Review Journalism

Using modern reviews of available stereoscopic 3D equipment, we have finalized the Nine Shameful Sins of S-3D Hardware Review Journalism. If you spot any of these when researching products, take the reviews with a grain of salt.

Sin #1: The Reviewer Didn’t Do His Homework

When the reviewer doesn’t take the time to familiarize themselves with how stereoscopic 3D technologies work, the whole review gets ruined. To make matters worse, when writers don’t admit what they don’t know, they get poor results, and with poor results come poor reviews.

The upside is when tech writers see 3D for the first time, the awe they experience translates greatly in the review, and this instills a feeling of confidence in the technology. Unfortunately, not all journalists will take this enthusiasm and try additional products to see if other solutions match up.

When reading reviews, look for experienced writers who demonstrate knowledge by properly using words like:

Brand-awareness is equally important. Some common brands:

Look for reviewers that demonstrate awareness in their writing style. For example, “solution A has some ghosting” or “solution B causes eyestrain when the shutter glasses are not refreshing fast enough”, etc.

Sin #2: When Making Comparisons, the Reviewer Compares Apples to Oranges

It is very upsetting to read reviews announcing clear industry winners without fair comparisons. It is irresponsible to both the industry and the customer.

“…I set out to compare the company’s 22 inch 3D display (iZ3D) against what I had just been working with the week prior—and the solution is just not there yet. On Nvidia’s GeForce 3D Vision, you adjust depth on a per-game basis, at most. Most of the time, on the titles you actually want to play in 3D, you don’t have to touch anything. But on the iZ3Ds display, we had to tweak separation (depth, just like Nvidia’s 3D Vision) and convergence. And depending on what we were looking at, those settings needed to be constantly tweaked. The latest technology built into the display, called Auto Focus, attempts to correct for differences in separation and convergence. But both myself and Tom’s Guide managing editor Rachel Rosmarin saw double of everything with the feature turned on. It’s still not final yet, so hopefully the company sees more favorable luck as Auto Focus matures.” CES ’09: The Real Stereoscopic Story, Tom’s Hardware

This Tom’s Hardware CES snapshot review is a perfect example. The NVIDIA GeForce 3D Vision Solution has been justifiably well received in the market. However, a little known fact is their software does not let you control the convergence or “pop-out” experience by default. It is only when the user turns these features on through “advanced settings” that the full 3D experience is activated. It is also when this convergence or pop-out experience is adjusted that most problems and anomalies appear.

In contrast, the IZ3D solution does not default with this limitation. If their display is more complex, it is because they offer more options to adjust in their driver’s default setting. Not knowing this, the reviewer treated NVIDIA’s handicap as an advantage, and the review misrepresented the industry’s offerings to the consumer.

Sin #3: Reviewers That Write As Though 3D Was Invented TODAY

“How many of you are old enough to remember going to the movies on Saturday afternoon or the drive in on Saturday night to watch the latest 3D movie? The usher would give you the 3D glasses that consisted of a paper frame with one red and one blue lens that barely hung onto your face. Then you would get to watch the movie and be less than satisfied with the technology that was being employed to create the effect. I know I’m old enough to remember those times and have been patiently waiting for the technology to actually catch up to make 3D a thing of the present, not an unpleasant memory from the past.

Nvidia now has that technology and is bringing it to the masses with the Geforce 3D Vision system. The Geforce 3D Vision is a package that allows games as well as movies and 3D images to be viewed in full stereoscopic 3D. This should bring all of the action on your favorite PC games to life, so you are no longer just a player but feel as though you are immersed in the action. The system is not for use with just new games, as Nvidia has completed profiles for well over 350 different games so that you have no worries when it comes to the games you play. If that’s the case I can’t wait to see how this all plays out and whether the technology actually works.” – Nvidia Geforce 3D Vision Review, OVERCLOCKERS CLUB

Stereoscopic 3D gaming has been around for over ten years. In fact, NVIDIA was the pioneer of this industry! Unfortunately, several reviewers write as though there is only one product in the market when there have been equally well promoted and equally viable 3D display solutions for years. This also suggests that the writer has no modern reference to compare the experience to.

Sin #4: Reviewers That Use the Term “Dorky Glasses”

“Not that the whole dorky 3D glasses thing is any sort of a slam-dunk proposition, of course. But I’ve spent some time playing around with…” – The Tech Report

“…includes special drivers to access the 3D vision inside many PC and console games, as well as those dorky 3D glasses.” – Tom’s Guide

“…and this one requires you to wear a pair of dorky 3D glasses…” – Ubergizmo.com

“…it still requires you to wear glasses—dorky ones…” – Popular Mechanics.com

“In order to make the picture 3D, you have to wear the dorky glasses supplied…” – T3.com

“…3D glasses still have the dorky feel to them though the company says they are modeled after modern sunglasses…” – blog.wired.com

Based on the U-DECIDE Initiative findings, 3D glasses are not a deterrent for the majority of 3D gamers. Just 12% of inexperienced 2D gamers object to wearing 3D glasses, and just 3% of experienced S-3D gamers think of the glasses as a barrier to their gaming experience.

When reviewers criticize the glasses like this, it implies negativity before even trying the S-3D display products, and likely shapes the mentality of their review. It is the equivalent of denying millions of customers the iPod because you need to wear headphones to listen to music.

Watch out for reviewers that mention anaglyph or red/blue glasses too. True 3D gamers have been doing without this out of date technology for years, and it has nothing to do with modern stereoscopic 3D solutions.

Sin #5: Reviewing Premium 3D Display Solutions with Subpar Equipment

“I tested the monitor on my home Alienware PC with a NVIDIA 6800GT video card and was able to muster only 22 frames per second in World of Warcraft. Bill Detwiler tried the monitor on his Dell XPS system with an ATI Radeon card and got similar results. Both of us normally see frame rates in the neighborhood of 35 frames per second. A mere 22fps is not acceptable.”– Techrepublic.com, Optimism for the iZ3D Monitor Deteriorates into Misery Over the Final Product

Stereoscopic 3D displays are intended for hardcore gamers. Hardcore gamers are a community of early adopters who spend more money than most to get superior game immersion and are looking for the maximum thrill factor in their games.

By nature, stereoscopic 3D gaming requires rendering the same image twice: once for the left eye, and once for the right. On paper, a drop in performance can be as much as 50%. In this case, the reviewer was using equipment that was years out of date. The performance was barely good enough for 2D, let alone stereoscopic 3D gaming.

Unless the graphics card equipment is considered mid-range or better by modern standards, the reported performance will not accurately reflect the gaming enjoyment the 3D display solution has to offer.

Sin #6: Reviewing 3D Display Solutions with Old Games

Until all game developers support 3D displays natively, successful 3D gaming will be dependent on stereoscopic drivers (e.g. iZ3D and NVIDIA drivers). Most video games are rendered in volumetric 3D, and stereo drivers can play most games off the shelf. However, there will occasionally be visual anomalies that need to be corrected on a case by case basis.

The solution is to have individual game profiles which optimize the drivers to overcome these flaws. While driver developers do their best, most will place greater focus on modern video games as that is what is popular and on store shelves today.

“Far Cry: The stereo effect is present in this game, but there are very distracting artifacts that show up as light shadows around objects (I will explain their origin below). It is virtually impossible to aim with the accurate sight when the gun’s barrel is in the center of the screen. The eyes get tired in 30-40 minutes of play.” – XBIT LABS iZ3D Stereoscopic Monitor Review

The above review by XBIT Labs was printed in October, 2008. Similar to the problems of testing a premium display with subpar equipment, FarCry is a 2004 video game which is years out of date. In contrast, FarCry 2, the modern hit by Ubisoft, would have fared much better, and is more representative of this product’s target market.

Even so, MTBS is familiar with FarCry, and this review also ties in with Sin #1 as the game was not set up properly for S-3D at all.

Sin #7: Not Reviewing the Product!

“For a premium gaming product that costs $1,000, the lack of performance dooms the iZ3D to oblivion as a viable monitor option. With 24-inch HD monitors available for under $500 and 28-inch monitors shipping at $700, there is absolutely no reason to buy the iZ3D monitor. Even if the 3D mode worked flawlessly, I would not be interested. The premium price is too much when compared to the supposed benefit of a 3D display.

Now, before I get backlash from fan-boys, let me emphasize that I am not the only one to be saying these things. Bill Detwiler is in complete agreement with me — he used the iZ3D for a week and came to similar conclusions. Unless you are reading this blog post on an iZ3D monitor, you really have no idea how bad the equipment performs.” – Optimism for the iZ3D Monitor Deteriorates into Misery Over the Final Product, Techrepublic.com

After sitting in front of a stereoscopic 3D monitor, a solution designed to enhance the visual gaming experience, 100% of this reviewer’s judgment focused on frames per second performance with out of date graphics card equipment. Not only were his remarks irrelevant to the display equipment, not only was he testing with GPU equipment years out of date, but he reserved no space in his review to actually talk about the 3D experience the monitor was intended for.

Sin #8: Bashing the Product They Didn’t Try

We had to print the entire excerpt because there are so many things wrong with it:

“Now polarization is the most common way of translating stereo 3D to your brain and it has many of the same technological aspects that the older anaglyph technology. The main difference in this technology is that instead of using read and blue colored lenses transparent lenses are used that polarize, or filter out the images and allow each eye to see its own image clearly. Polarization technology is nothing new; it’s been in your sunglasses for years filtering out various color patterns form the sun that can harm your eyes. It works in the same way with stereo 3D only in a slightly modified way. This as well has its limitations as well as you must be facing the monitor at just the right angle to see the picture correctly translated in 3D. The reason for this factor is that there is not enough power in a normal monitor to correctly change the view accordingly with your heads position to the monitor and although IZ3D did solve most of this problem by placing one monitor screen on top of another in conjunction with polarized glasses, it still gave you that headache associated with your retinal translation of 3D and slow framerates.

NVIDIA has gone a different direction entirely with this process, but this idea is not new just redefined by them and reintroduced as 3D vision, as technology in monitors has advanced to a new level by way of 120Hz refresh rates, which play a major role in this application. The major problem with any of the older passive or active technology using 3D glasses in conjunction with a monitor has been poor framerates, as most monitors at their highest resolution only offered 60Hz refresh rates, which translates into barely thirty framreates per second, per each eye of a pair of stereo 3D glasses. When playing games 30 framerates a second was the minimum requirement to keep games playing without the games playing sluggish, so take that and apply it to the 3D environment and you can understand why older monitors and glasses never quite worked right. Ah, but alas now come along monitors that have a refresh rate of 120Hz and new possibilities can be realized as now each eye of the new powered glasses can now give you the same refresh rate per eye in stereoscopic 3D as your old monitor did in standard gaming.” – NVDIA Stereoscopic 3D Vision Gaming Review, Gamepyre.com

As mentioned earlier, there is so much wrong with these statements. First, his explanation of how polarized technology works in stereoscopic 3D is false. Polarized technology is the same technique used in revenue record breaking 3D movie theaters, and if this writer honestly believes millions of people need to hold their head perfectly still while watching 3D movies, perhaps he also screws his light bulbs in by trying to spin the room around.

His explanation of the importance of refresh rate is also incorrect. High refresh rates are needed by LCD shutter glasses, like the NVIDIA GeForce 3D Vision solution, because when they are too low, the viewer gets a headache causing strobe effect. This has little if anything to do with frame rates, and frame rates have nothing to do with 3D headaches.

In addition to misunderstanding the technology on several levels, there are two problems here. First, we are unconvinced that Gamepyre has actually tried the iZ3D monitor – at least in some credible manner. If they did, they would have had a very basic understanding of how the technology works, what the real gripes are if any, and would have steered clear of this pointless tripe. Finally, they would have prevented embarrassing themselves by writing these baseless potshots in their article, and instead would have referenced a credible iZ3D review that they had written out of personal experience.

The second issue is we have no idea why Gamepyre mentioned the iZ3D solution in this review at all. We are confident that the NVIDIA GeForce 3D Vision solution would have stood on its own merits, and we can’t imagine what motivation there would be to take negative best guesses on untried and misunderstood technology.

Sin #9: Letting the Product Write the Review!

“Unlike some other stereoscopic 3D solutions which rely upon a driver “wrapper” which can dramatically decrease performance and adversely affect compatibility, GeForce 3D Vision makes use of NVIDIA’s own graphics driver. In doing so, NVIDIA is able to leverage the same resources and relationships from The Way It’s Meant to be Played program to ensure the best possible stereoscopic 3D gaming experience.

The NVIDIA Stereoscopic 3D control panel includes a compatibility list that specifies the level of stereoscopic 3D compatibility for hundreds of games. In addition, this information will be available online at www.nvidia.com/ged3d” – Written by NVIDIA PR, NVIDIA GeForce 3D Vision Review, Legitreviews.com

This HAS to be the worst sin of all. While MTBS recognizes and acknowledges NVIDIA’s solution, and we are supportive of their product in every way, it was completely inappropriate to print these remarks in the product review as is. Especially since these comments were never tested or challenged!

On paper, there could be truth to what NVIDIA has to say. For example, their intimate GPU access and driver development makes stereoscopic 3D support of SLI possible. For the same reasons, NVIDIA could have some performance advantages as well.

In practice, anything within the range of 40% to 60% of 2D performance in stereoscopic 3D mode is normal for ALL solutions. Ironically, using Legitreview’s own NVIDIA GeForce 3D Vision benchmarking results, NVIDIA earned a performance rating of just 40% of the potential 2D performance. While reasonable, this is at the low end of the spectrum, and does not support NVIDIA’s remarks.

NVIDIA’s statements about superior compatibility should have also been scrutinized. Yes, they have a list of 350 games, but this list is more a statement of their level of compatibility than their actual compatibility. Legitreviews should have determined what it means to be on this list of games firsthand before printing NVIDIA’s statements as fact.

We are confident that both NVIDIA’s performance and compatibility will eventually reach the levels that they claim. However, until that time, it’s not appropriate for Legitreviews to print their statements word for word, and accept their remarks of superiority as fact without testing them firsthand.

Conclusion

And there you have it! The Nine Shameful Sins of S-3D Display Review Journalism!

If it is too difficult to find hardware reviews that meet these ethical standards, an alternative is to talk amongst fellow gamers who own or have owned stereoscopic 3D equipment. MTBS’ discussion forums are filled with experienced S-3D gamers, and there is no better resource to learn what to expect and how to get the best results out of your true 3D gaming experience.

If you have additional sins to add to the S-3D display review list, please share them in our Shameful Sins thread. We welcome feedback on reviews for all manufacturers.

Until then, eat what the chef is eating, as they say – and let the reviewers worry about eating crow.