4 Light field properties
The book is still taking shape, and your feedback is an important part of the process. Suggestions of all kinds are welcome—whether it’s fixing small errors, raising bigger questions, or offering new perspectives. I’ll do my best to respond, but please keep in mind that the text will continue to change significantly over the next two years.
You can share comments through GitHub Issues.
Feel free to open a new issue or join an existing discussion. To make feedback easier to address, please point to the section you have in mind—by section number or a short snippet of text. Adding a label characterizing your issue would also be helpful.
Last updated: October 6, 2025
4.1 Light field properties overview
Many image systems are engineered to quantify scene properties, such as an object’s size, its distance to the sensor, or its material. These measurement-oriented systems have many applications, including industrial quality control, autonomous navigation, and medical diagnostics. Even in consumer photography, where the primary goal is often to capture and render a visually compelling representation of a scene, estimating underlying physical characteristics — like the ambient illumination — can significantly enhance performance. As highlighted in the Preface with the quote from Helmholtz Figure 1.5, inferring scene properties is arguably the fundamental task of visual perception.
A common challenge in many image system applications is that the measurement process is often underdetermined. This means that multiple distinct scene properties could potentially lead to the same observed measurement. In such situations, a powerful principle, often attributed to Thomas Bayes, becomes invaluable. Bayes’ Rule suggests that when faced with multiple plausible solutions, we should select the one that is most probable. While the general principle is conceptually elegant, the practical challenge lies in accurately determining the likelihood of each potential solution.
While we may not always possess explicit knowledge about the probabilities of different solutions, we can often leverage prior information or learn from experience. One effective strategy for making more informed inferences is to understand the inherent regularities present in the world. In this section, we will explore some fundamental regularities concerning the spectral (wavelength-dependent) and spatial characteristics of natural images.
Identifying and leveraging regularities within scene radiance is a central theme we will revisit throughout this book In specific, well-defined applications, such as medical imaging or controlled industrial inspection environments, these regularities can significantly constrain the set of plausible interpretations, providing a valuable and precise guide for analysis.
4.2 Representing regularities
When considering the scene regularities, we face two interconnected challenges. First, how do we come to understand these underlying patterns? In some instances, our knowledge stems from building physical models of the signals, where regularities emerge as a natural consequence of our understanding. The solar radiation analysis below provides a good example of this. In other cases, we learn about regularities through empirical observation of numerous examples, employing computational tools to quantify these patterns. Our discussions on surface reflectances and the characteristic spatial structure of natural images will illustrate this approach.
Two classical mathematical frameworks are fundamental for specifying regularities across various scientific and engineering disciplines, and both will feature prominently in this book. Bayes’ Rule offers a powerful yet straightforward way to reason about probabilities. More generally, probability distributions provide a versatile language for representing the likelihood of observations. While these distributions can sometimes be directly used in simulations, their incorporation can often present significant computational hurdles.
A second widely employed mathematical tool is dimensionality reduction, particularly techniques based on linear models. Dimensionality reduction offers a highly practical computational approach for simplifying complex datasets. However, it often provides only an approximate representation of the underlying statistical regularities.
More recently, neural network training has emerged as a potent method for inferring statistical regularities from large datasets, simultaneously yielding a computationally tractable model. This data-driven approach is proving increasingly valuable across numerous applications.
In this section, we will delve into the two classical techniques. They remain indispensable tools, often providing crucial insights into the physical principles governing the system. We will then apply them to key descriptions of scene spectral radiance. As we progress through subsequent sections of the book, we will explore how neural network training is increasingly being integrated into image systems engineering.
4.2.1 Bayes Rules
If we wish to know the probability of a specific condition, \(C\), being true given measured data, \(d\), we can use the laws of probability to write
\[ P(C | d) = \frac{P(d \cap C)}{P(d)} = \frac{P(d \cap C) P(C)}{P(d) P(C)}= \frac{P(d | C) P(C)}{P(d)} \]
- \(P(C | d)\) - posterior probability of C given d.
- \(P(d|C)\) - likelihood of d given C.
- \(P(C)\) - prior probability of C.
- \(P(d)\) - marginal likelihood of d.
Implementing Bayes’ Rule through this formula requires knowing the likelihood of the data given the condition, \(P(d|C)\). This knowledge can come from a model of the process that enables us to simulate the data. Being able to work out this probability is a fundamental task for the scientist or engineer. A more difficult challenge in using Bayes’ Rule is that we often have little idea about the prior, \(P(C)\). The cases in which we do have this world knowledge can be very valuable.
There are many cases, often in the applied sciences and engineering, when enough is known to apply Bayes’ Rule. A good example is a medical image diagnostic, say the chance of having a dark spot on of contrast \(d\) on the lung to detect a cancerous lesion. We may have measurements from many people, and in retrospect they either will have the disease or not. We can use these data to estimate the likelihood of \(P(d \mid C)\) and \(P(d \mid \neg C)\). We may further have historical data of the percentage of people in the population who have the disease, \(P(C)\). The marginal probability is
\[ P(d) = P(d \mid C) P(C) + P(d \mid \neg C) P(\neg C) \]
From these terms, we can compute the probability of having the posterior probability, $P(C d)$. There are many cases in which we do not have the data to estimate all of the needed likelihoods. In those cases, we can use other tools.
4.2.2 Linear models
Linear models have many uses in science and engineering. One application is representing certain regularities in the data. Specifically, suppose we have a collection of measurement vectors, each with \(D\) values. We can express each of the vectors as the weighted sum of a set of \(M\) vectors, which we call the basis vectors. If \(M\) is much smaller than \(D\), the basis set captures a regularity that is present in the original data set.
As an example, suppose the data are a set of radiance spectra, \(d(\lambda)\), we measured over the visible range from 380 nm to 780 nm in 1 nm steps; hence, each vector, \(d\), has 401 values. After making a few thousand (or million) measurements, we discover that we can closely approximate any of the measurements, \(d_j\), as the weighted sum of \(M\) basis functions, \(B_i\). We write the approximation this way:
\[ d_j(\lambda) \approx \sum_{i=1}^{M} B_i(\lambda) w_{ij} \tag{4.1}\]
There are a many tools to find basis vectors, \(\mathbf{B_i}\), and these tools rely on different objectives. Sometimes, the basis vectors are selected from physical or computational principles. This is the case for the Fourier basis functions, which are the harmonic functions, and are a natural part of magnetic resonance imaging, sound, and heat transfer. It is also common to select \(M\) basis functions that explain the most variance in the data, which is the choice we make when using Principal Components Analysis (PCA) that relies on the important matrix decomposition the singular value decomposition (SVD).
In both of these cases, Fourier and PCA, the basis vectors are orthogonal, which means that the inner product1 of any pair of basis vectors is zero. The mathematical expression for the inner product and orthogonality is
\[ 0 = \sum_{\lambda} B_i(\lambda) B_j(\lambda) \]
It is quite common to use matrix notation to express the linear model. With this notation, the linear model places the basis vectors into the columns of a matrix, \(\mathbf{B}\). Each data vector, \(\mathbf{d_j}\), is the product of a weight vector, \(\mathbf{w_j}\) times the matrix of basis vectors.
\[ \mathbf{d_j} = \mathbf{B} \mathbf{w_j} \tag{4.2}\]
This regularity is often called dimension-reduction. It is a helpful regularity, but quite different from knowing the full probability distribution. The regularity allows us to take the high dimensional data, say with 401-vectors, and represent them with many fewer numbers, say the 10 or 20 weights. In that sense, the reduction informs us about the likelihood of certain data sets. But there remains a much more that can be learned about the data because not all of the different weights are likely, or in some cases physically realizable. Thus, the dimensionality gets you part of the way there, but studying the properties of the weights can provide even more information (DiCarlo and Wandell (2003))
Linear models appear in many places in this book, with many different applications. If you would like to read more about them, have a look at the introduction to linear systems in Appendix Section 29.1. From there you can find your way to more advanced and mathematical literature. The Appendix also contains pointers to important nonlinear methods that - in some circumstances - can be very helpful (Maaten et al. (2008)).
In the following sections I describe regularities that have been discovered about various spectral measures. We begin with regularities concerning the spectral properties in the visible band. We then turn to regularities concerning the spatial properties of natural images, again in the visible band.
Widely used synonyms for the term inner product are dot product and scalar product.↩︎