27 Human visual metrics
Suggestions of all kinds for this book draft are welcome — whether it’s fixing small errors, raising bigger questions, or offering new perspectives. Please share comments through GitHub Issues. To make feedback easier to address, please point to the section you have in mind — by section number or a short snippet of text.
How can we evaluate the design of a camera? Lord Kelvin put it in that lovely style of the British upper crust
when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind (Popular Lectures and Addresses “Electrical Units of Measurement” (1889).)
Goodhart’s Law as a warning: “When a measure becomes a target, it ceases to be a good measure.”
The answer depends on your intended use. If the camera is intended to create images that people see, then the design must be guided by the capabilities and limits of the human eye. If the camera serves as input to a robot, or a self-driving car, or a medical image diagnostic system, we expect the camera’s design to be evaluated by how well it assists those systems.
28 Human visual metrics overview
Engineering progress relies on measurement. Improving a system is an iterative process that involves a loop of designing, measuring, and redesigning. Consequently, performance metrics are fundamental to nearly all engineering disciplines.
It is also a common principle that systems of even moderate complexity have more than one performance measure. We are often forced to make trade-offs, for example, between a system’s speed and its accuracy. We see this in modern AI systems that offer a choice between a ‘fast’ mode and a more thorough ‘thinking’ mode.
Imaging systems are no different. They present numerous trade-offs. For instance, increasing a sensor’s spatial resolution can reduce the number of photons captured per pixel, thus lowering the signal-to-noise ratio (SNR). Conversely, increasing SNR by using wider color filter bandwidths may reduce color accuracy.
When there is only one parameter to optimize, assessing system design is straightforward. However, most systems involve optimizing multiple parameters simultaneously. A design change might improve some metrics while degrading others, making it difficult to determine an overall improvement.
This complexity is reflected in the field of image quality metrics. Many different metrics exist. Some assess a specific aspect of system performance, guiding the design of a particular component. Others, however, aim to capture the general notion of image quality as a human would judge it. The most famous of these—the Structural Similarity Index Measure (SSIM), with over 100,000 citations—is a prime example.
Engineers, computer scientists, and product managers often desire a single number they can optimize and market. As a student or young researcher, you might be tempted to resist this simplification. A word of advice: understand the limitations of single-number metrics, but don’t let their imperfection stall your progress. Use them as the valuable, albeit incomplete, tools they are.
28.1 Human metrics
In the image processing section, I introduce a number of image systems metrics that are designed to evaluate the system and its components directly (Section 34.1). Those metrics might assess the signal-to-noise (SNR) of a pixel, or the MTF of a lens.
In this section, I review metrics that are designed with the human viewer in mind: How sharp will the image appear? Will the noise be visible? Do the colors in the rendering match those in the original? I have divided these human-centric metrics into three parts. The frist part covers metrics that are largely dependent on the spatial image properties, such as image sharpness and noise. The second part covers metrics that are largely dependent on color properties. The third section describes metrics that measure the temporal sensitivity, including the visibility of flicker or motion blur.
28.2 Spatial metrics
I have links to both original measurements, modern measurements (Standard Spatial Observer), and many commercial versions of spatial metrics that are implemented in ISETCam. This section will describe those.
28.2.1 Classical
That single channel guy with his metric and formula. Repeated everywhere. Barten. SQRI.
Earlier single channel. Value of multiple channel metrics? Existence? S-CIELAB accounts for distance
28.2.2 Modern
SSIM and MS-SSIM
LPIP goes here because it is human judgments of image quality?
Pyramid from Beau? Multi-parameter
I forked the Mantiuk github repository: The Mantiuk repository is here:. His metric goes here, I think. His code reads classic datasets. Maybe we include the data files in the book?
The surprising metric LPIPS
28.3 Color metrics
CIELAB
S-CIELAB
Mark Fairchild contributions. Others?
28.4 Temporal metrics
temporal MTF metrics