6  Spatial regularities

Published

September 8, 2026

Work in Progress

Suggestions of all kinds for this book draft are welcome — whether it’s fixing small errors, raising bigger questions, or offering new perspectives. Please share comments through GitHub Issues. To make feedback easier to address, please point to the section you have in mind — by section number or a short snippet of text.

6.1 Spatial regularities overview

Information about the spectral characteristics of natural images, including the illuminant and surfaces, is helpful for rendering images and for reasoning about their contents. The reason is simple: the more we know about these statistics, the better we can do at interpreting the images we measure.

Characterizing the spatial statistics of natural images is a second important quest within image systems engineering and vision science. Knowing about the likely spatial image will enable us to remove measurement noise more effectively, especially to the extent that natural images are not like noise. We can even hope that we do a better job of estimating the spatial properties of images that are just beyond our spatial resolution, being able to interpolate what we do see to a higher resolution based on knowing the likely structure of the object (Section 4.2).

Over the last four decades, our knowledge about the spatial statistics has grown considerably. The development of machine learning algorithms, particularly diffusion methods for generating natural images, has been a recent striking advance. In this chapter we will start with the foundations and work our way towards the most recent insights.

Things and Stuff

I am fond of a distinction that Ted Adelson introduced to help think about the environment: “things and stuff” Adelson (2001). Things are the countable, discrete objects with defined shapes and boundaries (e.g., cars, chairs, animals). Stuff is amorphous materials or textures without fixed boundaries (e.g., water, grass, sky, sand). Dividing the environment into these categories also helps us set computational goals.

Figure 6.1: The distinction between things and stuff plays a role in vision science and now computer vision. These images, from Schmidt et al. (2025), illustrate what is meant by stuff. The authors explored the appearance of materials and stuff, in contrast to much other work aimed at identifying things.

The distinction helps separate out the types of algorithms we might apply to understand the environment. Segmentation applies to things which are analyzed for object identity and function; in contrast, stuff is regionally based on texture and material properties. A great deal of computer vision focuses on tasks such as naming or counting cars on the road, people in the room, airplanes in the sky. We wish to be able to recognize such things whatever their material might be. A vision system must also be able to make judgments about a sandy beach, a cloudy sky, a grassy hillside. We wish to be able to recognize such stuff whatever its shape might be.

A related challenge in human vision is quantifying the appearance of things and stuff, and for this challenge surface reflectance, texture, and context (e.g., ambient illumination, or what things and stuff are nearby) all play a role Schmid et al. (2023). A corresponding concept - something we might call appearance for a computer vision system - seems desirable. One way to formulate the problem of appearance for a computer vision system is to use an estimate of material reflectance and texture to mean appearance for the computer.

The COCO-Stuff 164k dataset has many images categorized in terms of ‘things’ and stuff.

6.2 Spatial correlations

Image scientists were immediately drawn to analyzing the very different spatial statistics of images. If we examine the three panels of Figure 4.1, one very obvious feature jumps out. In the left panel, every pixel is completely independent of the other. That was how this image was created: every pixel intensity is drawn from an independent random simple. This independence is almost never found in natural images; the light level at one point is quite likely to be similar to the level at a nearby point.

In the middle panel, the similarity of adjacent pixels is extreme. Almost all the pixels are the same. That doesn’t happen in natural images either. The image is so unnatural, it might be surprising that we can recognize its contents at all. I often wonder whether other species can interpret such line drawings, that only represent the object boundaries.

What are the regularities that make the right panel - synthesized by a diffusion models - appear to be natural? Here are some ideas.

6.3 Natural scenes and the 1/f spatial frequency falloff

One idea is that natural images must satisfy this simple statistical measure: if we express the image as the weighted sum of harmonic functions (sines and cosines), the amplitudes of the harmonics decline with spatial frequency. This was the idea suggested in an early paper by Ruderman and Bialek (Ruderman and Bialek 1994) who set out to measure natural image spatial statistics. The analysis was confirmed and extended by others, including Simoncelli and Olshausen (2001), Geisler (2008), and Lee et al. (2001).

Field, Olshausen. I think they have work on this topic?

The power spectrum of natural images follows:

\[ P(\omega) \propto \frac{1}{|\omega|^{2-\eta}} \quad \text{where } \eta \approx 0.8\text{--}1.5 \]

Here, \(\omega\) represents spatial frequency magnitude, and \(\eta\) controls the decay rate

The scale invariance of natural images arises from their power-law spectral decay, which mathematically ensures statistical consistency across spatial scales. The connection between \(\fract{1}{f}^\alpha\) spectra and scale invariance can be expressed through:

1,2,3.

For scale invariance under spatial scaling \(x \to \lambda x\), the power spectrum must satisfy:

\[ P(\lambda\omega) = \lambda^{-\kappa}P(\omega) \]

Substituting the power-law form:

\[ P(\lambda\omega) = \frac{1}{|\lambda\omega|^{2-\eta}} = \lambda^{-(2-\eta)}P(\omega) \]

This matches the scale-invariance condition with \(\kappa = 2-\eta\), demonstrating that statistical properties remain consistent across scales4,5,6.

6.4 The dead leaves model

To me, it makes no sense to say that there is a single harmonic across the image. If there were, then getting closer to it would shift its frequency but not its amplitude.

This gets to the importance of phase over amplitude. The coherence length is frequency dependent, I would guess. This has an impact on the amplitude. The literature is not good on this point, although this model is helpful.

Lee et al. (2001)

Jon’s Matlab script as a basis for discussing this. Software from Jon showing 1/f issues.

Also, the deadleaves function in ISETCam.

6.4.1 Scale invariance

Fractal connection The power-law exponent relates to fractal dimension \(D\) through:

\[ D = 3 - \frac{\eta}{2} \]

where \(D\) quantifies space-filling characteristics. Natural images typically exhibit \(D \approx 2.2\text{--}2.6\), consistent with their 1/f^α spectra7,8.

This mathematical framework shows that 1/f^α spectra inherently encode fractal, scale-invariant structure - the same statistical regularities appear whether analyzing fine details or coarse features of natural scenes9,10,11.

6.5 Diffusion models

Diffusion models implemented in a neural network.

Eero’s analysis of the statistical properties of images using diffusion models.

This doesn’t get us to objects. But it is great for generative, when coupled with a text prompt that guides it to a particular content of natural images.

Explaining such models might be complicated, but let’s see if I can get us there from the Simoncelli and other papers.

Note: Save image compression and JPEG, which is a big one, for later. It depends a great deal on human vision and so it is too early here. But maybe some of the Simoncelli explanations of diffusion imaging networks could fit here. The idea that the networks learn fairly complex image statistics and when we start them out with noise, they drive the image towards something that is within the realm of a natural image.

  1. https://people.csail.mit.edu/danielzoran/zoranweiss09.pdf↩︎

  2. https://web.mit.edu/torralba/www/ne3302.pdf↩︎

  3. https://www.sciencedirect.com/science/article/pii/0042698996000028↩︎

  4. https://people.csail.mit.edu/danielzoran/zoranweiss09.pdf↩︎

  5. https://web.mit.edu/torralba/www/ne3302.pdf↩︎

  6. http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_ICCV_2009/contents/pdf/iccv2009_285.pdf↩︎

  7. https://people.csail.mit.edu/danielzoran/zoranweiss09.pdf↩︎

  8. https://www.nature.com/articles/srep46672↩︎

  9. https://people.csail.mit.edu/danielzoran/zoranweiss09.pdf↩︎

  10. https://web.mit.edu/torralba/www/ne3302.pdf↩︎

  11. http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_ICCV_2009/contents/pdf/iccv2009_285.pdf –>↩︎