For many, one of the big headlines that offered Pixel 2 / XL from Google was their prodigious 98 score in DxOMark. That milestone seemed to demonstrate right away that we were facing the best mobile camera on the market, but is that true?
The answer is complex, especially since that global score is only part of the equation, and there are many other sections that should be considered, some of which DxOMark offers in their analyzes. We talked to Nicolas Touchard, vice president of marketing for the company, to explain how the tests that have become the reference of mobile cameras work, and attentive, because there is more than one surprise.
The best camera is the one you always carry with you
The company DxOMark Image Labs had been developing software for post-processing of images for years, and its managers came up with a curious idea in 2009: why not create a website to analyze cameras?
This is how DxOMark was born, which initially dedicated its efforts to professionals and great fans of the world of photography. The idea was to establish a simple way that those users could know which camera was the best in general and also in certain specific sections.
That focus on the most specialized segment changed in 2012, when DxOMark published its first analysis of a mobile phone. The iPhone 4 took a score of 50, and little by little those responsible for this service realized that the reality in the segment of the cameras was that the mobiles had been made with absolute prominence for most of the users.
That made DxOMark analysis of the cameras of various mobile phones become something not only more and more frequent, but also more demanded by users. His test bed was adapted to this new market that had left far behind the original specialized sector, and although they continue to analyze DSLR cameras and also targets for those cameras, if there is something that is known lately DxOMark is the scores that they give to the cameras of the smartphones that are appearing in the market.
To obtain these scores in DxOMark they perform a series of tests that take place both in a highly controlled study environment and with specialized equipment, such as those carried out in all kinds of real environments, both indoors and outdoors, where it is possible to verify the behavior of those cameras with different types of light in everyday situations.
Unlike what happens with traditional cameras, mobile phones take into account the images obtained with the sensors but also the impact of the software that smartphones include to improve the final result. The binomial between the sensors and the post-processing software that acts every time we take a picture with a mobile phone has its maximum exponent in that computational photography of which Google presumes.
It is precisely this set that allows DxOMark to evaluate the final result and not just the “raw” image taken with the sensor , which is what they do when analyzing DSLR cameras and evaluate only the RAW file of each shot.
Let’s go in parts, what would Jack say
This joint analysis allows DxOMark to analyze parameters such as exposure, contrast, dynamic range , color (hue, saturation, white balance), texture and noise, the operation of autofocus, the possible artifacts that appear in the photo (aliasing, distortions, flares), the behavior of the flash, the zoom (optical, manual) and in recent times it is even evaluated that bokeh effect or depth of field that has revolutionized photography in portrait mode among users of mobiles.
This analysis actually has two parts, since in addition to the photographic mode in DxOMark they also evaluate the video recording mode, in which Flash, zoom, noise or bokeh do not come into play, but a key section does for those who use that function: the behavior of video stabilization.
All these parameters are analyzed independently with different specialized tests and each of them is scored independently, but the final score of DxOMark, that which is used as “absolute value” of the analysis has two particularities that make it somewhat Lends to confusion:
- Not scored above 100: This is one of the most deceptive sections of those global DxOMark scores, which do seem to have a maximum score of 100 that would be destined to the “perfect camera”. This is not the case since there are no established maximums, although obviously the more score a camera takes, the better it will be in its overall assessment. To prove it, there are already two cameras that have matched or exceeded that score: the Nikon D850 has obtained a 100, and the Hasselblad X1D-50x, a 102. Not only that: the analysis of the iPhone Xhas a global score of 97, but in the photo section this cell phone shows a 101. They explained it at the end of the analysis of Pixel 2, where they indicated that “our new test protocols and scoring systems have ample room for maneuver for new devices. There is nothing magical in a score of 100, and there is no reason why other cameras can not overcome it. We are sure that the time will come when they do it “
- The final score is not an average: If you stay in the final score you can lose a lot of things, because in fact in the DxOMark analyzes you evaluate as we said many parameters independently. If you look at these independent scores at the end of each of your analyzes, you can see how the final score is not the average of the scores of each section. Actually in DxOMark apply certain coefficients to each parameter to give them more or less importance for the final grade, and it is the combination of those independent notes and those coefficients (in DxOMark do not specify them) that gives the final grade. As explained to us by Nicolas Touchard, VP of Marketing at DxO Labs, the way in which these coefficients are applied is secret.
DxOMark, alliances and susceptibilities
It also happens that in DxOMark not only perform analysis of the most interesting mobile cameras that are coming to market: they also perform consulting tasks – that service is provided by another division called Image Quality Solutions – working with manufacturers who wish to sell equipment of laboratory with which to prove his cameras, for example, and advising in the processes of improvement of those sensors.
That, of course, can generate susceptibilities. Does it mean that DxOMark favors those companies with which it is allied? These suspicions could appear between manufacturers and users, but as Nicolas Touchard explained, it is very easy to dispel those doubts: “In DxOMark the tests are always the same for everyone”, he clarified, “regardless of whether we worked with the manufacturer or not. The test equipment works independently, “and the test bench is the same for all the devices that evaluate, with the evaluations also indistinct as to whether there has been collaboration or not.
Continue Reading: OLED screens on a smartphone: What we gain and lose?
Their test methodology is public and there is complete information about DxO Analyzer (they have a YouTube channel full of explanatory videos), so although manufacturers may try to do more to get more score in those tests that “magic recipe” with the that the final score is obtained makes it very difficult to “cheat” and improve on that score without the camera behaving in accordance with said assessment.
In DxOMark they explain how they work from within
We had the opportunity to speak with Nicolas Touchard, vice president of marketing at DxOMark, and he himself explained how the company, based in Paris, has recently changed its structure and name. DxOMark Image Labs (within which is DxOMark) is a company completely independent of DxO, which focuses on marketing products for end users such as the curious DxO ONE Camera.
We asked Mr. Touchard for more details about his testing process and methodology, and there he confirmed that difference between the tests they perform on DSLR cameras and targets -where they focus on RAW image analysis through objective metrics- and those made to smartphones, where what is evaluated “are the final JPEG images obtained from the camera”.
In that analysis there are both objective measures and a perceptual analysis. In the latter case, the methodology is rigorous: “images are taken of well-defined scenes that are always the same”. The engineers of DxOMark who take these images “always do it to the same places (approximately 50), and they take the images in the same way, some with tripods and some without them, even specifying the times of day to take them. and the necessary climate to be able to make tests”. If the weather fails, he explains, they wait until the conditions are right, which means that the time to complete an analysis can be extended.
In the end, he stressed, “they have about 1500 images both indoors (including those in the lab) and outdoors, plus two hours of video.” Many of these images are not evaluated “as a whole”, but before making this perceptual analysis, portions of the photo are cut out on which precisely one wants to analyze some aspect such as color, exposure, level of detail or noise.
Jan perceptual analysis, in fact, not only take the images with the camera analyzed, “but they are taken with several additional cameras that serve as a reference.” At least one camera is chosen very good, another normal and another worse (already evaluated by DxOMark) with which benchmarks are established and then evaluate those images (or those clippings of images) when looking for how they behave in different conditions.
These analyzes have been updated as well as time, said Mr. Touchard, who indicated how the tests for bokeh have been added in which smartphones “usually introduce artifacts”. The effect obtained with a DSLR camera is much more refined, and that makes precisely that analysis of the photos with that effect of depth of field, increasingly popular among smartphone users, has become part of that field of tests.
How does DxOMark decide which smartphone to analyze and which one not? Here they have a clear problem, and it is the lack of time: taking into account that a complete analysis can take the team “about five days”, they must be careful when selecting which devices to analyze. The demands of the users are numerous – some like LG V30 are very demanded, but simply they have not had occasion to be able to carry out those tests -, but as Mr. Touchard explained to us, they try to dedicate themselves to the most popular among the users. Of course, he clarifies, it is DxOMark who chooses those devices to analyze, “nobody else makes the decision”.
The work of DxOMark engineers means that it is precisely there that we have a clear idea of what will influence more in the future in the cameras of our smartphones, and for Mr. Touchard one of the key trends was the use of several modules photographic (sensor + lens + chip). In his opinion, dual camera systems will be increasingly numerous.
In fact, surprises here, because according to this directive the possibility of integrating even more modules in the smartphones of the future is a perfectly feasible option. “To improve the quality of the photos, the thickness of smartphones is a problem, the only way to improve that quality is to increase the sensor surface, silicon, and you can not do it with one, you have to do it with several”.
According to his opinion “we will see mobile phones with three and four modules in the future”, which will make mobile phones with triple and quadruple camera systems actually become a reality that according to Touchard is already being evaluated in laboratories of manufacturers in prototypes “multimodule “that among other things give option to greater options in topics such as zoom. The forecast, of course, is the most interesting.
The perspective, once again, is everything
For a while DxOMark has become the yardstick under which many measure the “theoretical” quality of a camera. Their analysis and especially their scores have become an argument for those who defend certain mobile cameras, both among those who write about these smartphones in the media and even in the industry itself.
This is demonstrated by the latest product presentations from Google or Huawei: they clearly mentioned that excellent DxOMark score of the cameras of their new phones, transmitting the message that these ratings made it clear that in fact we are facing the best mobile cameras in the world. current market. That’s right?
Not necessarily, of course. To begin with is that confusing final note that tries to describe the absolute quality of a mobile (or non-mobile) camera, which is, like any score, an assessment that has a subjective component on the part of those who have analyzed it.
The same thing happens with analyzes, which have a global assessment and a series of independent scores for certain sections in mobile phones and laptops, for example: that is our vision of these products, but each one of them can adjust better or worse to certain users and needs, which obviously do not coincide with our valuation or with another of another different analysis. As with these numerical assessments, the factors at play in the decision are very difficult to consider, so they simply try to establish homogeneous references for those who evaluate them.
This also happens in the case of DxOMark, whose scores must also be analyzed in more detail by users who want to know where a mobile camera behaves better or worse according to the evidence of those responsible. That is what the independent scores of each section we are talking about offer, and that is what in the end must be crucial for the users.
The comparison is simple. Imagine that you want to assess which camera you are most interested in, whether the iPhone 8 Plus (94 in the global DxOMark score) or the one in the Google Pixel 2 (98 in the global score). If you are interested in a mobile that manages color or autofocus, your decision is clear: shoot towards the Pixel 2. If, on the other hand, you want better management of the portrait and bokeh mode, you should bet on the iPhone 8 (55 points for 45). of Pixel 2, in addition to having that second lens “tele” 2x that also helps). That, of course, if you trust the DxOMark tests.
The work of DxOMark is therefore ambitious and of course all the information that they offer us about their methodology and results makes it clear that we are facing a rigorous tests, but as in everything, that final number that appears in each of those analyzes must be seen with a certain perspective.
Like any other assessment, it is not an absolute truth, since other sections – such as user experience, camera application, the phone’s own operation in terms of fluidity or resources- greatly influence those sensations and results that these cameras give mobile to its users.