33 Image processing: stereo
Suggestions of all kinds for this book draft are welcome — whether it’s fixing small errors, raising bigger questions, or offering new perspectives. Please share comments through GitHub Issues. To make feedback easier to address, please point to the section you have in mind — by section number or a short snippet of text.
33.1 Image processing: stereo overview
There is more to depth than stereo
33.2 Stereo depth information
In a conventional image system, the light imaged onto each pixel arises from a small region (point) in three-dimensional space. The sensor data only have an \((x,y)\) position, and additional inferencing is required to estimate the distance (three dimensional position,Z) of the object point \((X,Y,Z)\). There are many algorithms to infer the distance to the object; stereo cameras are one important technology.
Consider the ray tracing in Figure 33.1 (A). The image at a point \((x,y)\) could be anywhere along the three dimensional line. We could be looking at a star, many light-years away, or an LED two meters away. The distance to the object can be estimated by using a second camera imaging in the same direction (Figure 33.1 (B)). Knowing the corresponding location in the second camera disambiguates where the object is in three space.
In human vision the depth information collected by the two eyes is commonly specified relative to the position of the fovea, which is assumed to be directed at the object and thus called the fixation point. To the extent that a point is imaged at corresponding locations on the two retinae, we say there is zero disparity. To the extent that the two points are in different positions relative to the fovea, we say that there is disparity. The disparity can be crossed or uncrossed, which classifies the point as closer or further than the fixation point.
33.3 Epipolar geometry
The ideas of epipolar geometry are enormously helpful when we work with stereo cameras. The principles, derived using ray tracing geometry, help us search efficiently for potential stereo matches between two images. As we saw earlier, knowing which the correspondence between image points allows us to infer the position of the object in three-space.
The development of epipolar geometry was a gradual process that emerged from photogrammetry and computer vision research in the mid-20th century. The basic principles actually date back to early aerial surveying and mapping.
The fundamental concept builds on projective geometry principles established in the 19th century by mathematicians like Jean-Victor Poncelet. However, the specific application to stereo imaging and camera geometry took shape in the 1950s and 1960s with the rise of photogrammetry for mapping and surveying.
The key breakthrough was understanding that when two cameras view the same 3D point, the image points, the camera centers, and the 3D point all lie in the same plane - what we now call the epipolar plane. This geometric constraint significantly simplifies the stereo matching problem by reducing the search for corresponding points to a line rather than the entire image.
The formalization of epipolar geometry as we know it today, with the essential matrix and fundamental matrix, was developed in the 1980s. A major contribution came from Hans Longuet-Higgins in 1981 with his paper “A computer algorithm for reconstructing a scene from two projections,” which introduced the essential matrix that captures the geometric relationship between corresponding points in calibrated cameras.
The fundamental matrix, which handles uncalibrated cameras, was developed later in the 1990s, with significant contributions from researchers like Richard Hartley and Andrew Zisserman.
Harlyn Baker (not Harlan) did indeed work in computer vision and robotics, particularly at SRI International in the 1980s along with Robert Bolles and David Marimont. They made contributions to the practical application and implementation of stereo vision systems during an important period when computer vision was becoming more computationally feasible.
The SRI team, including Baker, Bolles, and Marimont, worked on various aspects of computer vision and robotics systems, including stereo vision. Their work was part of a broader effort at SRI International to develop more robust computer vision systems for robotics and automated inspection applications.
There was a significant paper by Robert Bolles, Harlyn Baker, and David Marimont published in 1987 in the International Journal of Computer Vision. The Epipolar-Plane Image (EPI) analysis they described was an innovative approach to the structure-from-motion problem.
Their key contribution was developing the concept of analyzing the epipolar-plane image - essentially looking at a slice of a sequence of images taken from a moving camera. When you stack these images and look at them from the side (taking a slice), you get patterns of lines whose slopes relate to the depths of points in the scene. This was a clever way to exploit the constraints of camera motion to simplify the 3D reconstruction problem.
What made their approach particularly notable was:
- It provided a way to analyze motion sequences without having to solve the correspondence problem for every frame
- The linear patterns in the EPIs made it easier to detect and track features
- It worked well for analyzing continuous motion sequences, not just discrete viewpoints
Their work at SRI International came during an important transition period in computer vision, when researchers were developing more practical approaches to 3D reconstruction that could actually be implemented with the computing power available at the time.
This paper influenced later work in structure-from-motion and multi-view stereo, as it provided both theoretical insights and practical methods for extracting 3D information from image sequences. It’s still referenced today, particularly when dealing with camera arrays or linear camera motions.