2 Light and seeing
Suggestions of all kinds for this book draft are welcome — whether it’s fixing small errors, raising bigger questions, or offering new perspectives. Please share comments through GitHub Issues. To make feedback easier to address, please point to the section you have in mind — by section number or a short snippet of text.
2.1 Light and seeing overview
Electromagnetic radiation fills the world around us, providing a rich and reliable stream of information. Mobile organisms — whether a hawk, a rabbit, or a human — have implemented ways to sense and interpret this information. They use the information to make decisions and interact with their surroundings. We call the sensing and interpretation of the radiation visual perception or more simply, seeing.
Visual perception is fundamental for guiding movement, enabling animals to locate resources, avoid threats, and navigate their environments. The use of radiation for mobility and decision-making by mobile organisms is in contrast to immobile life forms, such as trees and plants. For these organisms the ambient radiation serves primarily as a source of energy, rather than information.
In robotics and most computer applications, visual sensing serves a similar purpose: to enable actions and movement within dynamic and complex environments. Robots with imaging systems identify and manipulate parts. Cars with image systems navigate through their surroundings. Medical imaging provides diagnostic information to guide interventions. Image systems provide information that enables goal-oriented actions for both artificial and biological systems.
The deep connection between mobility, visual sensing, and the brain is illustrated by the remarkable life path of the sea squirt. It begins life as tadpole-like larvae, equipped with a brain and a tail for swimming. Once the tadpole finds a suitable spot to settle, it attaches to a hard surface like a rock or coral and becomes immobile.
As soon as the sea squirt fixes itself to the rock, with no more navigation or swimming, its brain is no longer needed. In a truly bizarre twist, the sea squirt digests its own brain. This adaptation reallocates energy resources from the now unnecessary brain to more vital functions.
My Stanford colleague, Irv Weissmann, and his group wrote many papers about neurodegeneration using the sea squirt.
Consumer photography - a large and important image system application - is an exception. Consumer photographs are created to store memories and evoke emotions, not to guide navigation or immediate decisions. Because of photography’s deep connection to human perception, the technologies for evaluating consumer photography differ from the engineering metrics that are appropriate for robots, cars, and medical imaging. For this reason, we will devote considerable attention to the properties of the human visual system (Section 1).
In summary, some image systems measure electromagnetic radiation because the signal is useful for movement or interacting with objects. Other image systems measure radiation to record a moment, enabling us to reproduce the signals at a later time. And finally, some image systems measure electromagnetic radiation at wavelengths we do not see, enabling us to explore domains that are beyond our senses 1.
This section of the book is devoted to understanding the signal we use to to form images.
2.2 The electromagnetic spectrum
One of the great achievements of science is the discovery that seemingly different phenomena are deeply connected. By the early 1800s, scientists had uncovered links between electricity and magnetism: Øersted (1820) showed that electric currents generate magnetic fields, and Faraday (1832) demonstrated that changing magnetic fields induce electric currents.
James Clerk Maxwell formalized this discovery in a set of mathematical equations, simplified by Heaviside, and now known as Maxwell-Heaviside equations. He demonstrated that oscillating electric and magnetic fields behave as waves that propagate through space. Remarkably, he found that the predicted speed of these waves matched the measured speed of light. Maxwell concluded that light itself is an electromagnetic wave, thereby uniting electricity, magnetism, and optics into a single theoretical framework: electromagnetic radiation spanning a range of wavelengths.
We will encounter Maxwell’s work multiple times in this book. In addition to this remarkable unifying set of equations, we will see his insights about optics and his fundamental demonstrations of the properties of human color vision.
Einstein kept images of three scientists in his office —Isaac Newton, James Clerk Maxwell, and Michael Faraday. These captured the lineage of thought that led to his own work. Newton’s laws of motion laid the foundation for classical mechanics. Maxwell unified electricity, magnetism, and light with his equations, building upon the experimental work of Faraday, who originated the concept of electric and magnetic “fields.” Einstein’s theories of relativity were a direct response to Maxwell’s equations, and he saw himself as a successor to this great tradition. I have (several) friends who remind me that two of these three scientists were Scottish. I very much appreciate this homage to Newton by Neil de Grasse Tyson/
James Clerk Maxwell died in 1879 at the age of 48 from abdominal cancer. His mother, Frances Clerk Maxwell, also died from the same disease at the age of 48 when he was just eight years old.
2.3 Light
Not all electromagnetic wavelengths are equally useful for seeing. The human eye detects electromagnetic radiation within a specific range, approximately 380 nm to 770 nm. This range is referred to as light—electromagnetic radiation that is visible to the human eye. Radiation in this range is particularly useful for seeing because it is strongly absorbed and reflected by objects in the environment, making it ideal for detecting and interpreting the world around us.
In contrast, high-energy radiation, such as X-rays (0.01–10 nm) and gamma rays (< 0.01 nm), tends to pass through most objects without significant interaction. On the other end of the spectrum, low-frequency radiation (> 1500 nm) provides lower spatial resolution and is heavily influenced by thermal effects, limiting its utility for detailed imaging. The visible spectrum occupies a unique and advantageous position for vision and perception.
2.4 The environmental light field
Electromagnetic radiation fills the environment, traveling in all directions and interacting with the objects that surround us. We call the environmental radiation within the visible wavelength band the environmental light field.
Leonardo Da Vinci’s notebook (Da Vinci 1970) describes why he concluded that light fills the environment. He placed a small hole in a wall of a windowless room that was adjacent to a brightly illuminated piazza (Figure 2.3). This produced an (inverted) image of the piazza on the wall within the room. Leonardo observed that one can place the pinhole anywhere in the wall and an image of the same objects is produced. He concluded that the light field is “all everywhere and all in each part”2.
For the moment, it is convenient to consider the light field as comprising a large set of rays. We can describe the light field in the environment, \(L_E\), by the intensity of each ray, expressing the intensity as a function of its various parameters. Rays cross through each point in the volume of space, \((x,y,z)\), in many directions. We specify these directions by two direction angles \((\alpha, \beta)\), the azimuth and elevation. Each ray has a wavelength \(\lambda\) and a polarization \(\rho\). The ray intensities are described by a function of seven parameters:
\[ L_E(x,y,z,\alpha,\beta,\lambda,\rho) \tag{2.1}\]
People in many fields are familiar with the concept of the light field, but they use different terminology. Physicists often describe the environmental light field as the spectral radiance of the environment (see Feynman’s lovely description in the callout below). The light field terminology was introduced into optical engineering by Gershun (1939) as part of his work in understanding how to design lighting environments, such as the lighting in school rooms and public places3. Ted Adelson and Jim Bergen (1991) used the term plenoptic function to describe the same idea4. In computer graphics, the value of the light field concept was explained by Marc Levoy and Pat Hanrahan (1996). I use the light field terminology because I find it less intimidating than spectral radiance, electromagnetic radiation, or plenoptic.
Richard Feynman’s lovely description of light filling the environment, in the language physicists call spectral radiance.
2.5 The incident light field
An imaging system—whether an eye, a camera, or a simple pinhole—can only capture the portion of the environmental light field that reaches its aperture, or entrance pupil (Figure 2.4). We call this subset of rays the incident light field.
Because the incident light field is defined on the two-dimensional surface of the entrance pupil, we can describe it with fewer parameters than the full environmental light field. Instead of a 3D position in space \((x,y,z)\), we use a 2D position \((u,v)\) on the pupil plane. The complete description of the imaging system also requires its position in space, \(\mathbf{p}\), and its viewing direction, \(\mathbf{n}\). We treat these as fixed conditions for a given measurement.
The incident light field, \(L_I\), can then be expressed as a function of six variables, plus the system’s position and orientation:
\[ L_I(u,v,\alpha,\beta,\lambda,\rho; ~\mathbf{p},\mathbf{n}) \tag{2.2}\]
To capture the full environmental light field, we must follow Leonardo’s insight and measure the incident light field from multiple positions and directions. Many animals have two eyes, providing two simultaneous measurements. This redundancy not only offers a backup if one eye is damaged but also provides more information about the scene. Prey animals, like rabbits, often have eyes pointing in different directions to achieve a wide field of view. Predators, on the other hand, typically have forward-facing eyes with overlapping fields, which enables stereoscopic vision for depth perception. We also gather more information about the light field over time as we move our heads and our eyes rotate (motion parallax).
Engineers have built camera arrays that capture many incident light fields simultaneously. By measuring from multiple positions and directions, these systems can create immersive displays that reproduce the experience of moving and looking around within a scene. If the scene is static, a single moving camera can achieve the same result.
A large camera array built at the Stanford Graphics Lab illustrates this principle , Figure 2.5. The array shown at the top of the figure captures views from many positions, all pointing in the same direction. This setup allows a viewer to experience moving side-to-side or forward and backward. The arrays at the bottom, with cameras pointing outward from a central point, capture the light field from many directions. This allows a viewer to experience looking around from a fixed position. By combining these approaches, systems can capture enough data to simulate both movement and changes in gaze direction (Thatte et al. 2017). These camera arrays capture a coarse but powerful sample of the environmental light field, enabling the rendering of novel viewpoints through interpolation.
2.6 Optical light field
The optics of an imaging system—whether a pinhole, a simple lens, or a complex multi-element lens—transform the incident light field into a new light field inside the camera. This optical light field exists in the space between the exit pupil of the optics and the sensor. The journey of light begins in the environment, is narrowed to the incident light field at the entrance pupil, and is then transformed by the optics into the optical light field. The sensor, whether a digital chip or the retina, measures this optical light field5.
While the environmental and incident light fields are naturally described by a ray’s position and angle, the optical light field is often described differently. Because we are interested in the rays that travel from the optics to the sensor, it is convenient to parameterize them by their start and end points. We can define a ray by its position \((u,v)\) on the exit pupil plane and its destination \((r,c)\) on the sensor plane. This is often called the two-plane parameterization.
Using this convention, we can describe the optical light field as a function of these four spatial parameters, along with wavelength and polarization:
\[ L_O(u,v,r,c,\lambda,\rho) \tag{2.3}\]
This parameterization is particularly useful because it directly relates to what a light field sensor measures. Of course, it is possible to convert between this two-plane representation and the position-angle representation used for the other light fields.
The data recorded by the sensor is all we have to create an image or interpret the scene. The ability of any system to derive information from the light field is ultimately limited by what portion of the environmental light field passes through the optics to become the optical light field and is finally captured by the sensor.
There are instruments that use different wavebands of the electromagnetic spectrum. A brief introduction is here.. More examples of instruments will be presented in later chapters.↩︎
PROVE HOW ALL OBJECTS, PLACED IN ONE POSITION, ARE ALL EVERYWHERE AND ALL IN EACH PART
“I say that if the front of a building—or any open piazza or field—which is illuminated by the sun has a dwelling opposite to it, and if, in the front which does not face the sun, you make a small round hole, all the illuminated objects will project their images through that hole and be visible inside the dwelling on the opposite wall which may be made white; and there, in fact, they will be upside down, and if you make similar openings in several places in the same wall you will have the same result from each. Hence the images of the illuminated objects are all everywhere on this wall and all in each minutest part of it. The reason, as we clearly know, is that this hole must admit some light to the said dwelling, and the light admitted by it is derived from one or many luminous bodies. If these bodies are of various colours and shapes the rays forming the images are of various colours and shapes, and so will the representations be on the wall. (Leonardo’s 1509 Notebook, curated by John Paul Richter)”↩︎I enjoyed the first sentence in the 1939 translation of Gershun’s (1939) paper. The translators, Moon and Timoshenko, express frustration with the pace of advances in photometry. “Theoretical photometry constitutes a case of ‘arrested development’, and has remained basically unchanged since 1760 while the rest of physics has swept triumphantly ahead.” Gershun himself expresses the same sentiment: “The problems of theoretical photometry were pushed aside from the main path of the development of physics.”↩︎
Adelson and Bergen (1991) introduced the plenoptic function for a pinhole camera and a sensor, making it closely related to the incident light field. (Section 2.5). Using a pinhole camera model, the two parameters describing the aperture position are not needed. With these restrictions the plenoptic function becomes equivalent to the spectral irradiance at the image sensor (Wandell and Brainard 2021). The authors clearly had the idea of the environmental and incident light fields in mind.↩︎
Ren Ng used the phrase ‘in-camera light field’ in his Ph.D. thesis (Ng et al. 2005). I like that expression, too.↩︎