6 Spatial regularities
Suggestions of all kinds for this book draft are welcome — whether it’s fixing small errors, raising bigger questions, or offering new perspectives. Please share comments through GitHub Issues. To make feedback easier to address, please point to the section you have in mind — by section number or a short snippet of text.
6.1 Spatial regularities overview
Information about the spectral characteristics of natural images, including the illuminant and surfaces, is helpful for rendering images and for reasoning about their contents. The reason is simple: the more we know about these statistics, the better we can do at interpreting the images we measure.
Characterizing the spatial statistics of natural images is a second important quest within image systems engineering and vision science. Knowing about the likely spatial image will enable us to remove measurement noise more effectively, especially to the extent that natural images are not like noise. We can even hope that we do a better job of estimating the spatial properties of images that are just beyond our spatial resolution, being able to interpolate what we do see to a higher resolution based on knowing the likely structure of the object (Section 4.2).
Over the last four decades, our knowledge about the spatial statistics has grown considerably. The development of machine learning algorithms, particularly diffusion methods for generating natural images, has been a recent striking advance. In this chapter we will start with the foundations and work our way towards the most recent insights.
I am fond of a distinction that Ted Adelson (Adelson 2001) introduced to help think about the environment: “things and stuff” . Things are the countable, discrete objects with defined shapes and boundaries (e.g., cars, chairs, animals). Stuff is amorphous materials or textures without fixed boundaries (e.g., water, grass, sky, sand). Dividing the environment into these categories also helps us set computational goals.
The distinction helps separate out the types of algorithms we might apply to understand the environment. Segmentation and edges applies to things which are analyzed for object identity and function; in contrast, stuff is regionally based on texture and material properties. A great deal of computer vision focuses on tasks such as naming or counting cars on the road, people in the room, airplanes in the sky. We wish to be able to recognize such things whatever their material might be. A vision system must also be able to make judgments about a sandy beach, a cloudy sky, a grassy hillside. We wish to be able to recognize such stuff whatever its shape might be.
A related challenge in human vision is quantifying the appearance of things and stuff, and for this challenge surface reflectance, texture, and context (e.g., ambient illumination, or what things and stuff are nearby) all play a role Schmid et al. (2023). A corresponding concept - something we might call appearance for a computer vision system - seems desirable. One way to formulate the problem of appearance for a computer vision system is to use an estimate of material reflectance and texture to mean appearance for the computer.
The COCO-Stuff 164k dataset has many images categorized in terms of ‘things’ and stuff.
6.2 Spatial correlations
Image scientists were immediately drawn to analyzing the very different spatial statistics of images. If we examine the three panels of Figure 4.1, one very obvious feature jumps out. In the left panel, every pixel is completely independent of the other. That was how this image was created: every pixel intensity is drawn from an independent random simple. This pointwise independence is rarely found in natural images at the scale we see; the light level at one point is quite likely to be similar to the level at a nearby point.
In the middle panel, the similarity of adjacent pixels is extreme. Almost all the pixels are the same as their neighbor. That doesn’t happen in natural images either. The image is so unnatural, it might be surprising that we can recognize its contents at all. I often wonder whether other species can interpret such line drawings, that only represent the object boundaries.
What are the regularities that make the right panel - synthesized by a diffusion models - appear to be natural? Here are some ideas.
6.3 Natural scenes and the 1/f spatial frequency falloff
One idea is that natural images satisfy a simple statistical regularity: when an image is expressed as a weighted sum of harmonic functions (sines and cosines), the harmonic amplitudes decline inversely with spatial frequency. This property was demonstrated in an influential paper by Field (1987) and further measured and analyzed by (Ruderman and Bialek 1994; Simoncelli and Olshausen 2001; Geisler 2008).
We can express this property using a simple equation:
\[ A(f) = \frac{C}{f^\alpha} \]
where \(f\) is spatial frequency, \(\alpha\) is a decay parameter typically close to 1, and \(C\) is an overall amplitude scaling factor related to image contrast. In terms of the power spectrum (amplitude squared), this falloff is \(P(f) \propto 1/f^{2\alpha}\) (often approximated as \(1/f^2\)).
Notice what is missing from this equation: there is no parameter with units of spatial frequency (such as an \(f_0\)) that specifies a “typical” size or frequency. This is what is meant by the phrase has no characteristic spatial scale.
To see what this means, consider an image that does have a characteristic scale—for example, a brick wall or a checkerboard. In those images, there is a dominant, privileged spatial frequency determined by the physical size of the bricks or squares. If the spectrum were an exponential falloff, \(A(f) = C e^{-f/f_0}\), the constant \(f_0\) would define a characteristic scale where contrast drops off sharply.
In a power law, however, scaling the spatial frequency by a factor \(s\) (such as magnifying or demagnifying the image) simply multiplies the amplitude by an overall constant: \(A(s f) = s^{-\alpha} A(f)\). The relative distribution of frequencies remains identical: the contrast drop across any octave (e.g., from 1 to 2 cycles/deg, or from 10 to 20 cycles/deg) is always the same factor \(2^{-\alpha}\). Because no particular scale is privileged, people often say that natural scenes are scale invariant: zooming in or out on natural textures—like clouds, coastlines, or rocks—yields images with the same relative frequency statistics.
6.3.1 The paradox of the 1/f rule
Despite its widespread prominence in the literature, treating the \(1/f\) rule as a fundamental property of natural scenes raises important conceptual difficulties.
First, we must distinguish between properties of the incident light field and properties of acquired images. In acquired images, optical blur and sensor noise inevitably alter the measured spectral slope (Zoran and Weiss 2009). While that is an important practical consideration, a more fundamental conceptual issue concerns the basis functions themselves: harmonic functions (sines and cosines) span the entire field of view, whereas natural scene textures are localized.
To appreciate the paradox that arises from using global harmonics, consider a surface texture measured at viewing distance \(d\). Suppose the pattern has a spatial frequency of \(f\) cycles per degree (Figure 6.2).
Now consider observing the same physical pattern from half the distance, \(d/2\) (the yellow shaded region in Figure 6.2). Because the viewing distance is halved, the visual angle subtended by each cycle doubles, meaning the pattern now has a frequency of \(f/2\) cycles per degree. If the Fourier amplitude simply reflected the physical contrast of the surface pattern, the amplitude measured at \(f/2\) from distance \(d/2\) should equal the amplitude measured at \(f\) from distance \(d\). Yet the empirical \(1/f\) rule asserts that the lower frequency (\(f/2\)) should have substantially higher amplitude. Why should stepping closer to a surface boost the measured contrast?
6.3.2 Local carrier contrast and spatial coherence length
The resolution to this apparent paradox hinges on distinguishing between global Fourier amplitude density and local carrier contrast.
Global Fourier analysis presumes infinite sinusoidal coherence across the entire field of view. But natural image textures possess a finite spatial coherence length (\(L_c\)). Natural textures modulate in contrast, local frequency, and phase across an image; they remain consistent only across a limited spatial extent before encountering surface boundaries, occlusions, perspective changes, or physical surface irregularities.
Crucially, natural textures tend to remain coherent over a comparable number of cycles rather than a fixed angular distance. Consequently, higher spatial frequencies have proportionally shorter spatial coherence lengths: \[ L_c(f) \propto \frac{1}{f} \]
If we measure the pattern’s contrast over an aperture smaller than this coherence length—scaling the measurement window to span a fixed number of cycles (\(W(f) \propto 1/f\))—we find that the local carrier contrast is scale-invariant and roughly constant across frequencies. A high-frequency texture (like the weave of a shirt or fine bark) has essentially the same physical modulation contrast locally as a low-frequency texture (like large sand dunes or rolling hills).
The measured reduction in global Fourier amplitude across frequency arises from the loss of spatial coherence, not from a loss of local carrier contrast:
- Low spatial frequencies have long coherence lengths (\(L_c \propto 1/f\)), meaning they maintain phase coherence across a large fraction of the camera’s field of view and sum constructively into a tall, sharp Fourier peak.
- High spatial frequencies have shorter coherence lengths. A fixed field of view encompasses many independent, phase-incoherent patches. When integrated across the entire image, destructive phase interference and boundary discontinuities scatter the high-frequency energy across adjacent frequencies, depressing the measured global peak amplitude.
Thus, the \(1/f\) rule does not mean that the world fades away in contrast at fine scales. It means that high-frequency contrast is packaged into smaller, spatially localized packets. This is precisely why the visual system—and modern multi-scale representations—abandons global Fourier harmonics in favor of localized receptive fields (wavelets) whose spatial extent scales with wavelength.
The Matlab calculation in the link below illustrates how a finite spatial coherence length affects measured Fourier spectra across viewing distances. In the simulation, a surface texture is modeled with constant local carrier contrast and a finite coherence length (\(N_c \approx 3\text{--}5\) cycles). A camera with a fixed angular field of view records this surface from different viewing distances:
- Close distance (\(d = 0.25\)): The camera’s field of view intercepts only a narrow physical region spanning a few cycles (\(\sim N_c\)). The pattern maintains phase coherence across the aperture, concentrating its contrast into a tall, sharp Fourier peak.
- Far distance (\(d = 1.00\)): The camera’s field of view intercepts a much wider physical region spanning many cycles (\(\gg N_c\)). Random phase drift across independent patches causes destructive interference, dispersing spectral energy into adjacent frequencies and reducing the measured peak Fourier amplitude.
The coherence loss and spectral broadening produce the empirical \(1/f^\alpha\) amplitude falloff. You can view the full calculation, waveforms, and experiment with different parameters, in the ISETCam file linked here fise_spatialCoherence1d.html.
6.4 The dead leaves model
The spatial coherence analysis highlights the critical role of phase and localized structure. The dead leaves model (Lee et al. 2001) takes this insight even further, demonstrating that a \(1/f\) amplitude spectrum emerges naturally from object occlusion boundaries alone, without requiring any internal surface texture.
Figure 6.3 illustrates an example. The image is constructed by randomly placing disks of varying sizes and gray levels across the image plane, allowing later disks to occlude earlier ones—much like fallen leaves gathering on the ground1. Notice that the disks themselves are completely flat and uniform; all of the image contrast arises strictly from the step edges formed by the occluding boundaries.
Remarkably, when the distribution of object sizes follows a scale-invariant power law (\(p(r) \propto r^{-3}\)), the resulting image mathematically yields a \(1/f^\alpha\) power spectrum (with \(\alpha \approx 2\), corresponding to an amplitude spectrum of \(1/f\)). In this model, the interior of each surface is constant - with no texture - and all the contrast arises from the boundaries. The \(1/f\) falloff is thus a geometric consequence of object occlusions and scale-free size distributions in the physical world, rather than a property of wave-like oscillations.
To me, these two perspectives reflect Ted Adelson’s distinction between things and stuff. The traditional \(1/f\) view invites us to think about the spatial statistics of continuous textures (stuff). The dead leaves model shows that the very same \(1/f\) falloff can emerge entirely from the boundaries between discrete objects (things) of many sizes. Both contribute to natural images, of course, but you can choose which intuition resonates more with you. Personally, I find myself in the dead leaves camp.
An upcoming ISETCam script (fise_deadleaves) will demonstrate the generation and spectral calculation for synthetic dead leaves images.
For an exploration of the historical connection between \(1/f^\alpha\) power spectra, fractal dimension, and texture perception—along with opportunities for student code development in ISETCam—see the resource page on Fractals, Scale Invariance, and Natural Textures.
6.5 Diffusion models
Developing code for this section.
All notes below here
To this point the analysis has been classical; we have examined well understood statistics such as spatial correlations and spatial frequency falloff. These statistics have advanced our understanding, and perhaps the most important point we might take away is that images depend on more than very local correlations. The dead leaves construction based on spatial distribution of edges across the images - a decidedly non-local statistic - produces images that match critical statistical measures such as 1/f. Plainly, the dead leaves image in Figure 6.3 is not a natural image, and we must conclude that there is a great deal missing from the local correlations and 1/f summary.
The ability to train neural networks with many billions of parameters has enabled investigators to build system that learn very complex image statistics. The machine learning community created models that could produce patterns that appeared to be natural images using ideas from non-equilibrium thermodynamics (Sohl-Dickstein et al. (2015)) and score-based modeling (Song and Ermon (2019), Ho et al. (2020)). These insights led to a massive growth in the technology enabling, particularly when the process could be guided to produce images with specific types of content. This technology - generative diffusion imaging - has also created the massive number of deep fakes; images that do not capture a real event.
The computational imaging community arrived at the same destination but coming from opposite direction. In that field, the dominant technology was the removal of noise. The idea is that we the appearance of noise implies that the image contains statistical properties that do not confirm to the properties of real natural images. Noise removal algorithms (denoising) - using methods such as empirical Bayes that incorporates natural image statistics - could also drive pure noise images to become a (fake) natural image (Kadkhodaie & Simoncelli, 2020/2021). The two fields arrived at similar conclusions, but I think it is fair to say that the theorists learned more from the implementation than the implementation has learned from the theorists.2
The idea that the networks learn fairly complex image statistics and when we start them out with noise, they drive the image towards something that is within the realm of a natural image.
This doesn’t get us to objects. But it is great for generative, when coupled with a text prompt that guides it to a particular content of natural images.
Notes below that need development.
Eero Simoncelli and collaborators formalized the concept that an optimal denoiser acts as an implicit score estimator of the natural image distribution, demonstrating how coarse-to-fine Bayesian denoising can both sample from the image manifold and solve general inverse problems (Kadkhodaie and Simoncelli 2020). Subsequent theoretical work from his group further established that the generalization ability of deep denoisers in diffusion architectures stems from an inductive bias toward geometry-adaptive harmonic bases (Kadkhodaie et al. 2023).
Core Framework: Denoising as an Implicit Prior. The foundational paper by Kadkhodaie & Simoncelli (2020) demonstrated that a convolutional neural network (CNN) trained strictly for least-squares Gaussian denoising implicitly encodes the score function \(\nabla_x \log p_\sigma(x)\) via Miyasawa’s (and Tweedie’s) formulation. Rather than formulating explicit parametric priors (such as Gaussian Scale Mixtures or Markov random fields), the authors showed that:Sampling from the Manifold: Iterating coarse-to-fine stochastic gradient ascent steps using only the denoiser’s residual vector field generates novel, visually coherent images from pure white noise.
Universal Inverse Problem Solving: By projecting the denoising updates onto measurement constraints, the implicit prior solves deblurring, super-resolution, inpainting, and compressive sensing without task-specific retraining @Kadkhodaie and Simoncelli (2020). To address whether these networks truly learn a continuous manifold density or merely memorize discrete data points, Kadkhodaie et al. (2023) analyzed the mathematical mechanics of trained denoisers. They proved that deep neural denoisers exhibit strong generalization: independent networks trained on disjoint subsets of image datasets converge to virtually identical score functions. Mechanistically, the networks decompose images using geometry-adaptive harmonic bases (GAHBs), performing localized shrinkage along continuous contours and homogeneous regions.
Supporting Paradigms from Other Groups. Regularization by Denoising (RED): Preceding the full diffusion formulation, Romano et al. (2017) introduced RED, proposing that any arbitrary, state-of-the-art image denoiser could serve directly as an explicit regularizer within an optimization objective. This established that modular denoisers encode sufficient natural image statistics to resolve ill-posed inverse problems (Romano et al., 2017).
Multi-Scale Score Matching: Working in parallel from a machine learning perspective, Song and Ermon (2019) introduced Noise-Conditioned Score Networks (NCSN). They demonstrated that perturbing images with multiple scales of Gaussian noise resolves the problem of ill-defined score gradients in low-density regions, proving that annealed Langevin dynamics over learned denoisers accurately samples the true data manifold.
Debates, Critiques, and Alternative InterpretationsNon-Conservative Vector Fields and Asymmetric Jacobians: A key dispute regarding the “denoiser as prior” framework emerged around whether deep neural denoisers actually define a valid probability density. Reehorst & Schniter (2019) demonstrated that practical CNN denoisers generally do not possess symmetric Jacobians. Because a conservative vector field requires a symmetric Jacobian (\(\nabla^2 f(x) = (\nabla^2 f(x))^T\)), practical denoisers do not strictly correspond to the gradient of any scalar energy functional or true prior density. Reehorst & Schniter (2019) proposed “Score-Matching by Denoising” (SMD) to clarify that iterative denoising algorithms are better interpreted as score estimators rather than variational minimizers of an explicit prior.
Manifold Learning vs. Empirical Memorization: While Kadkhodaie et al. (2023) emphasize strong inductive biases toward smooth harmonic representations, empirical security and privacy studies present a contrasting perspective. Carlini et al. (2023) demonstrated that large-scale diffusion models can memorize specific training instances and reproduce them with high fidelity during inference. This disputes the ideal assumption that diffusion models strictly sample from a smooth, continuous manifold, showing that capacity, replication, and conditioning can distort the learned score field into localized memorization basins (Carlini et al., 2023).
Diffusion vs. LLMs on the Image Manifold: While discrete autoregressive models (analogous to Large Language Models) model images by quantizing patches into sequences of categorical tokens (e.g., via VQ-VAE or discrete codebooks), diffusion architectures operate over continuous probability spaces. Denoising diffusion models interact directly with the continuous geometry and differential topology of the image manifold via gradient fields, whereas autoregressive token models capture joint distributions through next-token cross-entropy without an explicit continuous score mechanism.
Note: Save image compression and JPEG, which is a big one, for later. It depends a great deal on human vision and so it is too early to describe here. I do wonder whether human vision properties would be of any value in thinking about diffusion imaging, but I suspect not.
I may be a sensitive soul, but I would have preferred the name ‘fallen leaves’. Too many years in California, I guess.↩︎
This is a famous old line from Henderson, who wrote ‘Science owes more to the steam engine than the steam engine owes to science’ (Henderson (1917)). The idea was made famous at MIT by Nathan Rosenberg (who wrote extensively on whether science drives technology or vice versa) who argued for the symbiotic relationship between shop-floor mechanical practice and fundamental research. He put it this way: ‘Engineering owes more to the diesel engine than the diesel engine owes to engineering.’↩︎