The Mathematics of Optical Triangulation
This course is primarily concerned with the estimation of 3D shape by illuminating the world with certain known patterns, and observing the illuminated objects with cameras. In this chapter we derive models describing this image formation process, leading to the development of reconstruction equations allowing the recovery of 3D shape by geometric triangulation.
We start by introducing the basic concepts in a coordinate-free fashion, using elementary algebra and the language of analytic geometry (e.g., points, vectors, lines, rays, and planes). Coordinates are introduced later, along with relative coordinate systems, to quantify the process of image formation in cameras and projectors.
Perspective Projection and the Pinhole Model
Figure 2.1 Perspective projection under the pinhole model.
A simple and popular geometric model for a camera or a projector is the pinhole model, composed of a plane and a point external to the plane (see Figure 2.1). We refer to the plane as the image plane, and to the point as the center of projection. In a camera, every 3D point (other than the center of projection) determines a unique line passing through the center of projection. If this line is not parallel to the image plane, then it must intersect the image plane in a single image point. In mathematics, this mapping from 3D points to 2D image points is referred to as a perspective projection. Except for the fact that light traverses this line in the opposite direction, the geometry of a projector can be described with the same model. That is, given a 2D image point in the projector's image plane, there must exist a unique line containing this point and the center of projection (since the center of projection cannot belong to the image plane). In summary, light travels away from a projector (or towards a camera) along the line connecting the 3D scene point with its 2D perspective projection onto the image plane.
Geometric Representations
Since light moves along straight lines (in a homogeneous medium such as air), we derive 3D reconstruction equations from geometric constructions involving the intersection of lines and planes, or the approximate intersection of pairs of lines (two lines in 3D may not intersect). Our derivations only draw upon elementary algebra and analytic geometry in 3D (e.g., we operate on points, vectors, lines, rays, and planes). We use lower case letters to denote points \(p\) and vectors \(v\). All the vectors will be taken as column vectors with real-valued coordinates \(v\in\mathbb{R}^3\), which we can also regard as matrices with three rows and one column \(v\in\mathbb{R}^{3\times 1}\). The length of a vector \(v\) is a scalar \(\|v\|\in\mathbb{R}\). We use matrix multiplication notation for the inner product \(v_1^tv_2\in\mathbb{R}\) of two vectors \(v_1\) and \(v_2\), which is also a scalar. Here \(v_1^t\in\mathbb{R}^{1\times 3}\) is a row vector, or a \(1\times 3\) matrix, resulting from transposing the column vector \(v_1\). The value of the inner product of the two vectors \(v_1\) and \(v_2\) is equal to \(\|v_1\|\|v_2\|\cos(\alpha)\), where \(\alpha\) is the angle formed by the two vectors ( \(0\leq\alpha\leq 180^{\circ}\)). The \(3\times N\) matrix resulting from concatenating \(N\) vectors \(v_1,\ldots,v_N\) as columns is denoted \([v_1|\cdots|v_N]\in\mathbb{R}^{3\times N}\). The vector product \(v_1\times v_2\in\mathbb{R}^3\) of the two vectors \(v_1\) and \(v_2\) is a vector perpendicular to both \(v_1\) and \(v_2\), of length \(\|v_1\times v_2\|=\|v_1\|\,\|v_2\|\,\sin(\alpha)\), and direction determined by the right hand rule (i.e., such that the determinant of the matrix \([v_1|v_2|v_1\times v_2]\) is non-negative). In particular, two vectors \(v_1\) and \(v_2\) are linearly dependent ( i.e., one is a scalar multiple of the other), if and only if the vector product \(v_1\times v_2\) is equal to zero.Points and Vectors
Since vectors form a vector space, they can be multiplied by scalars and added to each other. Points, on the other hand, do not form a vector space. But vectors and points are related: a point plus a vector \(p+v\) is another point, and the difference between two points \(q-p\) is a vector. If \(p\) is a point, \(\lambda\) is a scalar, and \(v\) is a vector, then \(q=p+\lambda v\) is another point. In this expression, \(\lambda v\) is a vector of length \(|\lambda|\,\|v\|\). Multiplying a point by a scalar \(\lambda p\) is not defined, but an affine combination of \(N\) points \(\lambda_1 p_1+\cdots+\lambda_N p_N\), with \(\lambda_1+\cdots+\lambda_N=1\), is well defined:
\begin{equation} \lambda_1 p_1+\cdots+\lambda_Np_N = p_1+\lambda_2(p_2-p_1)+\cdots+\lambda_N(p_N-p_1)\;. \end{equation}
Parametric Representation of Lines and Rays
Figure 2.2 Parametric representation of lines and rays.
A line \(L\) can be described by specifying one of its points \(q\) and a direction vector \(v\) (see Figure 2.2). Any other point \(p\) on the line \(L\) can be described as the result of adding a scalar multiple \(\lambda v\), of the direction vector \(v\), to the point \(q\) ( \(\lambda\) can be positive, negative, or zero): \begin{equation} L = \{\; p=q+\lambda v\;:\;\lambda\in\mathbb{R}\;\}\;. \label{eq:line-parametric} \end{equation} This is the parametric representation of a line, where the scalar \(\lambda\) is the parameter. Note that this representation is not unique, since \(q\) can be replaced by any other point on the line \(L\), and \(v\) can be replaced by any non-zero scalar multiple of \(v\). However, for each choice of \(q\) and \(v\), the correspondence between parameters \(\lambda\in\mathbb{R}\) and points \(p\) on the line \(L\) is one-to-one.
A ray is half of a line . While in a line the parameter \(\lambda\) can take any value, in a ray it is only allowed to take non-negative values. \[ R = \{\; p=q+\lambda v \;:\;\lambda\geq 0\;\} \] In this case, if the point \(q\) is changed, a different ray results. Since it is unique, the point \(q\) is called the origin of the ray. The direction vector \(v\) can be replaced by any positive scalar multiple, but not by a negative scalar multiple. Replacing the direction vector \(v\) by a negative scalar multiple results in the opposite ray. By convention in projectors, light traverses rays along the direction determined by the direction vector. Conversely in cameras, light traverses rays in the direction opposite to the direction vector (i.e., in the direction of decreasing \(\lambda\)).
Parametric Representation of Planes
Figure 2.3 Parametric and implicit representations of planes.
Similar to how lines are represented in parametric form, a plane \(P\) can be described in parametric form by specifying one of its points \(q\) and two linearly independent direction vectors \(v_1\) and \(v_2\) (see Figure 2.3. Any other point \(p\) on the plane \(P\) can be described as the result of adding scalar multiples \(\lambda_1 v_1\) and \(\lambda_2 v_2\) of the two vectors to the point \(q\), as follows. \[ P = \{\; p=q+\lambda_1 v_1 + \lambda_2 v_2 \;:\;\lambda_1,\lambda_2\in\mathbb{R}\;\} \] As in the case of lines, this representation is not unique. The point \(q\) can be replaced by any other point in the plane, and the vectors \(v_1\) and \(v_2\) can be replaced by any other two linearly independent linear combinations of \(v_1\) and \(v_2\).
Implicit Representation of Planes
A plane \(P\) can also be described in implicit form as the set of zeros of a linear equation in three variables. Geometrically, the plane can be described by one of its points \(q\) and a normal vector \(n\). A point \(p\) belongs to the plane \(P\) if and only if the vectors \(p-q\) and \(n\) are orthogonal:
\begin{equation} P = \{\; p\;:\: n^t(p-q)=0 \;\}\;. \label{eq:plane-implicit} \end{equation}
Again, this representation is not unique. The point \(q\) can be replaced by any other point in the plane, and the normal vector \(n\) by any non-zero scalar multiple \(\lambda n\).
To convert from the parametric to the implicit representation, we can take the normal vector \(n=v_1\times v_2\) as the vector product of the two basis vectors \(v_1\) and \(v_2\). To convert from implicit to parametric, we need to find two linearly independent vectors \(v_1\) and \(v_2\) orthogonal to the normal vector \(n\). In fact, it is sufficient to find one vector \(v_1\) orthogonal to \(n\). The second vector can be defined as \(v_2=n\times v_1\). In both cases, the same point \(q\) from one representation can be used in the other.
Implicit Representation of Lines
A line \(L\) can also be described in implicit form as the intersection of two planes, both represented in implicit form, such that \begin{equation} L = \{\; p\;:\; n_1^t(p-q)=n_2^t(p-q)=0 \;\}, \label{eq:line-implicit} \end{equation} where the two normal vectors \(n_1\) and \(n_2\) are linearly independent (if \(n_1\) an \(n_2\) are linearly dependent, rather than a line, the two equations describe the same plane). Note that when \(n_1\) and \(n_2\) are linearly independent, the two implicit representations for the planes can be defined with respect to a common point belonging to both planes, rather than to two different points. Since a line can be described as the intersection of many different pairs of planes, this representation is not unique. The point \(q\) can be replaced by any other point belonging to the intersection of the two planes, and the two normal vectors can be replaced by any other pair of linearly independent linear combinations of the two vectors.
To convert from the parametric representation of Equation \ref{eq:line-parametric} to the implicit representation of Equation \ref{eq:line-implicit}, one needs to find two linearly independent vectors \(n_1\) and \(n_2\) orthogonal to the direction vector \(v\). One way to do so is to first find one non-zero vector \(n_1\) orthogonal to \(v\), and then take \(n_2\) as the vector product \(n_2=v\times n_1\) of \(v\) and \(n_1\). To convert from implicit to parametric, one needs to find a non-zero vector \(v\) orthogonal to both normal vectors \(n_1\) and \(n_2\). The vector product \(v=n_1\times n_2\) is one such vector, and any other is a scalar multiple of it.
Reconstruction by Triangulation
It is common for projected illumination patterns used in laser scanning and structured lighting to contain identifiable lines or points. Under the pinhole projector model, a projected line creates a plane of light (the unique plane containing the line on the image plane and the center of projection), and a projected point creates a ray of light (the unique line containing the image point and the center of projection).
While the intersection of a ray of light with the object being scanned can be considered as a single illuminated point, the intersection of a plane of light with the object generally contains many illuminated curved segments. Each of these segments is composed of many illuminated points. A single illuminated point, visible to the camera, defines a camera ray. For now, we assume that the locations and orientations of projector and camera are known with respect to the global coordinate system (with procedures for estimating these quantities covered in the Chapter on Camera and Projector Calibration ). Under this assumption, the equations of projected planes and rays, as well as the equations of camera rays corresponding to illuminated points, are defined by parameters which can be measured. From these measurements, the location of illuminated points can be recovered by intersecting the planes or rays of light with the camera rays corresponding to the illuminated points. Through such procedures the depth ambiguity introduced by pinhole projection can be eliminated, allowing recovery of a 3D surface model.
Line-Plane Intersection
Figure 2.4 Triangulation by line-plane intersection.
Computing the intersection of a line and a plane is straightforward when the line is represented in parametric form \[ L = \{\; p=q_L+\lambda v \;:\;\lambda\in\mathbb{R}\;\}, \] and the plane is represented in implicit form \[ P = \{\; p\;:\: n^t(p-q_P)=0 \;\}\;. \] Note that the line and the plane may not intersect, in which case we say that the line and the plane are parallel. This is the case if the vectors \(v\) and \(n\) are orthogonal \(n^tv=0\). The vectors \(v\) and \(n\) are also orthogonal when the line \(L\) is contained in the plane \(P\). Whether or not the point \(q_L\) belongs to the plane \(P\) differentiates one case from the other. If the vectors \(v\) and \(n\) are not orthogonal \(n^tv\neq 0\), then the intersection of the line and the plane contains exactly one point \(p\). Since this point belongs to the line, it can be written as \(p=q_L+\lambda v\), for a value \(\lambda\) which we need to determine. Since the point also belongs to the plane, the value \(\lambda\) must satisfy the linear equation \[ n^t(p-q_p)= n^t(\lambda v + q_L-q_p)=0\;, \] or equivalently \begin{equation} \lambda = \frac{n^t(q_P-q_L)}{n^tv}\;. \label{eq:lambda-line-plane-intersection} \end{equation} Since we have assumed that the line and the plane are not parallel (i.e., by checking that \(n^tv\neq 0\) beforehand), this expression is well defined. A geometric interpretation of line-plane intersection is provided in Figure 2.4.
Line-Line Intersection
Figure 2.5 Triangulation by line-line intersection.
We consider here the intersection of two arbitrary lines \(L_1\) and \(L_2\), as shown in Figure 2.5. \[ L_1=\{\; p=q_1+\lambda_1 v_1\;:\;\lambda_1\in\mathbb{R}\} \quad \hbox{and} \quad L_2=\{ p=q_2+\lambda_2 v_2\;:\;\lambda_2\in\mathbb{R}\;\} \]
Let us first identify the special cases. The vectors \(v_1\) and \(v_2\) can be linearly dependent (i.e., if one is a scalar multiple of the other) or linearly independent.
The two lines are parallel if the vectors \(v_1\) and \(v_2\) are linearly dependent. If, in addition, the vector \(q_2-q_1\) can also be written as a scalar multiple of \(v_1\) or \(v_2\), then the lines are identical. Of course, if the lines are parallel but not identical, they do not intersect.
If \(v_1\) and \(v_2\) are linearly independent, the two lines may or may not intersect. If the two lines intersect, the intersection contains a single point. The necessary and sufficient condition for two lines to intersect, when \(v_1\) and \(v_2\) are linearly independent, is that scalar values \(\lambda_1\) and \(\lambda_2\) exist so that \[ q_1+\lambda_1 v_1 = q_2+\lambda_2 v_2, \] or equivalently so that the vector \(q_2-q_1\) is linearly dependent on \(v_1\) and \(v_2\).
Since two lines may not intersect, we define the approximate intersection as the point which is closest to the two lines. More precisely, whether two lines intersect or not, we define the approximate intersection as the point \(p\) which minimizes the sum of the square distances to both lines \( \phi(p,\lambda_1,\lambda_2)=\|q_1+\lambda_1 v_1-p\|^2+\|q_2+\lambda_2 v_2-p\|^2\;.\) As before, we assume \(v_1\) and \(v_2\) are linearly independent, such the approximate intersection is a unique point.
To prove that the previous statement is true, and to determine the value of \(p\), we follow an algebraic approach. The function \(\phi(p,\lambda_1,\lambda_2)\) is a quadratic non-negative definite function of five variables, the three coordinates of the point \(p\) and the two scalars \(\lambda_1\) and \(\lambda_2\).
Figure 2.6 Parametric and implicit representations of planes.
We first reduce the problem to the minimization of a different quadratic non-negative definite function of only two variables \(\lambda_1\) and \(\lambda_2\). Let \(p_1=q_1+\lambda_1 v_1\) be a point on the line \(L_1\), and let \(p_2=q_2+\lambda_2 v_2\) be a point on the line \(L_2\). Define the midpoint \(p_{12}\), of the line segment joining \(p1\) and \(p2\), as \[ p_{12}=p_1+\frac{1}{2}(p_2-p_1)=p_2+\frac{1}{2}(p_1-p_2)\;. \] A necessary condition for the minimizer \((p,\lambda_1,\lambda_2)\) of \(\phi\) is that the partial derivatives of \(\phi\), with respect to the five variables, all vanish at the minimizer. In particular, the three derivatives with respect to the coordinates of the point \(p\) must vanish \[ \frac{\partial\phi}{\partial p} = (p-p_1)+(p-p_2) = 0\;, \] or equivalently, it is necessary for the minimizer point \(p\) to be the midpoint \(p_{12}\) of the segment joining \(p_1\) and \(p_2\) (see Figure 2.6).
As a result, the problem reduces to the minimization of the square distance from a point \(p_1\) on line \(L_1\) to a point \(p_2\) on line \(L_2\). Practically, we must now minimize the quadratic non-negative definite function of two variables \[ \psi(\lambda_1,\lambda_2)= 2\phi(p_{12},\lambda_1,\lambda_2)=\|(q_2+\lambda_2v_2)-(q_1+\lambda_1v_1)\|^2\;. \] Note that it is still necessary for the two partial derivatives of \(\psi\), with respect to \(\lambda_1\) and \(\lambda_2\), to be equal to zero at the minimum, as follows. \[ \frac{\partial \psi}{\partial\lambda_1} = v_1^t(\lambda_1 v_1-\lambda_2 v_2+q_1-q_2) = \lambda_1 \|v_1\|^2-\lambda_2 v_1^tv_2+v_1^t(q_1-q_2) = 0 \] \[ \frac{\partial \psi}{\partial\lambda_2} = v_2^t(\lambda_2 v_2-\lambda_1 v_1+q_2-q_1) = \lambda_2 \|v_2\|^2-\lambda_2 v_2^tv_1+v_2^t(q_2-q_1) = 0 \] These provide two linear equations in \(\lambda_1\) and \(\lambda_2\), which can be concisely expressed in matrix form as \[ \left(\begin{matrix} \|v_1\|^2 & -v_1^tv_2 \cr -v_2^tv_1 & \|v_2\|^2 \end{matrix}\right) \left(\begin{matrix} \lambda_1 \cr \lambda_2 \end{matrix}\right) = \left(\begin{matrix} v_1^t(q_2-q_1) \cr v_2^t(q_1-q_2) \end{matrix}\right)\;. \] It follows from the linear independence of \(v_1\) and \(v_2\) that the \(2\times 2\) matrix on the left hand side is non-singular. As a result, the unique solution to the linear system is given by \[ \left(\begin{matrix} \lambda_1 \cr \lambda_2 \end{matrix}\right) = \left(\begin{matrix} \|v_1\|^2 & -v_1^tv_2 \cr -v_2^tv_1 & \|v_2\|^2 \end{matrix}\right)^{-1} \left(\begin{matrix} v_1^t(q_2-q_1) \cr v_2^t(q_1-q_2) \end{matrix}\right) \] or equivalently \begin{equation} \left(\begin{matrix} \lambda_1 \cr \lambda_2 \end{matrix}\right) = \frac{1}{ \|v_1\|^2\|v_2\|^2-(v_1^tv_2)^2 } \left(\begin{matrix} \|v_2\|^2 & v_1^tv_2 \cr v_2^tv_1 & \|v_1\|^2 \end{matrix}\right) \left(\begin{matrix} v_1^t(q_2-q_1) \cr v_2^t(q_1-q_2) \end{matrix}\right)\;. \label{eq:lambda-line-line-intersection} \end{equation} In conclusion, the approximate intersection \(p\) can be obtained from the value of either \(\lambda_1\) or \(\lambda_2\) provided by these expressions.
Coordinate Systems
So far we have presented a coordinate-free description of triangulation. In practice, however, image measurements are recorded in discrete pixel units. In this section we incorporate such coordinates into our prior equations, as well as document the various coordinate systems involved.
Image Coordinates and the Pinhole Camera
Consider a pinhole model with camera coordinate sytem comprising a center of projection \(q\), and orthonormal basis \(\{v_1,v_2,v_3\}\), and image plane spanned by the first two basis vectros at focal length \(f=1\). Any 3D point \(p\), not necessarily on the image plane, has coordinates \((p^1,p^2,p^3)^t\) relative to the camera coordinate system. The projection of the point \(p\) onto the image plane is a 2D image point defined by the parameters \(u^1\) and \(u^2\), which can be written as a 3D vector \(u=(u^1,u^2,1)\). Using this notation the point \(p\) is expressed as \[ \left( \begin{matrix} p^1\cr p^2\cr p^3 \end{matrix} \right) = \lambda \, \left( \begin{matrix} u^1\cr u^2\cr 1 \end{matrix} \right) \] where \(\lambda\) is a non-negative parameter.
The Ideal Pinhole Camera
Figure 2.7 The ideal pinhole camera.
In the ideal pinhole camera shown in Figure 2.7, the center of projection \(o\) is at the origin of the world coordinate system, with coordinates \((0,0,0)^t\), and the point \(q\) and the vectors \(v_1\) and \(v_2\) are defined as \[ [v_1|v_2|q]= \left( \begin{matrix} 1&0&0\cr 0&1&0\cr 0&0&1 \end{matrix} \right)\;. \] Note that not every 3D point has a projection on the image plane. Points without a projection are contained in a plane parallel to the image passing through the center of projection. An arbitrary 3D point \(p\) with coordinates \((p^1,p^2,p^3)^t\) belongs to this plane if \(p^3=0\), otherwise it projects onto an image point with the following coordinates. \[ \begin{matrix} u^1 & = & p^1/p^3\cr u^2 & = & p^2/p^3 \end{matrix} \] There are other descriptions for the relation between the coordinates of a point and the image coordinates of its projection; for example, the projection of a 3D point \(p\) with coordinates \((p^1,p^2,p^3)^t\) has image coordinates \(u=(u^1,u^2,1)\) if, for some scalar \(\lambda\neq 0\), we can write
\begin{equation} \lambda\; \left( \begin{matrix} u^1\cr u^2\cr 1 \end{matrix} \right) = \left( \begin{matrix} p^1\cr p^2\cr p^3 \end{matrix} \right)\;. \label{eq:ideal-pinhole-projection} \end{equation}
The General Pinhole Camera
Figure 2.8 The general pinhole model.
The center of a general pinhole camera is not necessarily placed at the origin of the world coordinate system and may be arbitrarily oriented. However, it does have a camera coordinate system attached to the camera, in addition to the world coordinate system (see Figure 2.8). A 3D point \(p\) has world coordinates described by the vector \(p_W=(p_W^1,p_W^2,p_W^3)^t\) and camera coordinates described by the vector \(p_C=(p_C^1,p_C^2,p_C^3)^t\). These two vectors are related by a rigid body transformation specified by a translation vector \(T\in\mathbb{R}^3\) and a rotation matrix \(R\in\mathbb{R}^{3\times 3}\), such that \[ p_C=R\,p_W+T\;. \] In camera coordinates, the relation between the 3D point coordinates and the 2D image coordinates of the projection is described by the ideal pinhole camera projection (i.e., Equation \ref{eq:ideal-pinhole-projection}), with \(\lambda u = p_C\). In world coordinates this relation becomes \begin{equation} \lambda\,u = R\,p_W\,+\,T\;. \label{eq:general-pinhole-projection} \end{equation} The parameters \((R,T)\), which are referred to as the extrinsic parameters of the camera, describe the location and orientation of the camera with respect to the world coordinate system.
Equation \ref{eq:general-pinhole-projection} assumes that the unit of measurement of lengths on the image plane is the same as for world coordinates, that the distance from the center of projection to the image plane is equal to one unit of length, and that the origin of the image coordinate system has image coordinates \(u^1=0\) and \(u^2=0\). None of these assumptions hold in practice. For example, lengths on the image plane are measured in pixel units, and in meters or inches for world coordinates, the distance from the center of projection to the image plane can be arbitrary, and the origin of the image coordinates is usually on the upper left corner of the image. In addition, the image plane may be tilted with respect to the ideal image plane. To compensate for these limitations of the current model, a matrix \(K\in\mathbb{R}^{3\times 3}\) is introduced in the projection equations to describe intrinsic parameters as follows. \begin{equation} \lambda\,u = K(R\,p_W\,+\,T) \label{eq:general-pinhole-projection-with-intrinsic} \end{equation} The matrix \(K\) has the following form \[ K= \left( \begin{matrix} f\,s_1 & f\,s_\theta & o^1 \cr 0 & f\,s_2 & o^2 \cr 0 & 0 & 1 \end{matrix} \right)\;, \] where \(f\) is the focal length (i.e., the distance between the center of projection and the image plane). The parameters \(s_1\) and \(s_2\) are the first and second coordinate scale parameters, respectively. Note that such scale parameters are required since some cameras have non-square pixels. The parameter \(s_\theta\) is used to compensate for a tilted image plane. Finally, \((o^1,o^2)^t\) are the image coordinates of the intersection of the vertical line in camera coordinates with the image plane. This point is called the image center or principal point. Note that all intrinsic parameters embodied in \(K\) are independent of the camera pose. They describe physical properties related to the mechanical and optical design of the camera. Since in general they do not change, the matrix \(K\) can be estimated once through a calibration procedure and stored (as will be described in the following chapter). Afterwards, image plane measurements in pixel units can immediately be ``normalized'', by multiplying the measured image coordinate vector by \(K^{-1}\), so that the relation between a 3D point in world coordinates and 2D image coordinates is described by Equation \ref{eq:general-pinhole-projection}.
Real cameras also display non-linear lens distortion, which is also considered intrinsic. Lens distortion compensation must be performed prior to the normalization described above. We will discuss appropriate lens distortion models in Chapter Calibration.
Lines from Image Points
As shown in Figure 2.4, an image point with coordinates \(u=(u^1,u^2,1)^t\) defines a unique line containing this point and the center of projection. The challenge is to find the parametric equation of this line, as \(L=\{p=q+\lambda\,v\,:\,\lambda\in\mathbb{R}\}\). Since this line must contain the center of projection, the projection of all the points it spans must have the same image coordinates. If \(p_W\) is the vector of world coordinates for a point contained in this line, then world coordinates and image coordinates are related by Equation \ref{eq:general-pinhole-projection} such that \(\lambda\,u = R\,p_W\,+\,T\). Since \(R\) is a rotation matrix, we have \(R^{-1}=R^t\) and we can rewrite the projection equation as \[ p_W = (-R^tT)+\lambda\,(R^tu)\;. \] In conclusion, the line we are looking for is described by the point \(q\) with world coordinates \(q_W=-R^tT\), which is the center of projection, and the vector \(v\) with world coordinates \(v_W=R^tu\).
Planes from Image Lines
Figure 2.9 The plane defined by an image line and the center of projection.
A straight line on the image plane can be described in either parametric or implicit form, both expressed in image coordinates. Let us first consider the implicit case. A line on the image plane is described by one implicit equation of the image coordinates \[ L=\{u:l^tu=l^1u^1+l^2u^2+l^3=0\}\;, \] where \(l=(l^1,l^2,l^3)^t\) is a vector with \(l^1\neq 0\) or \(l^2\neq 0\). Using active illumination, projector patterns containing vertical and horizontal lines are common. Thus, the implicit equation of an horizontal line is \[ L_H=\{u:l^tu=u^2-\nu=0\}\;, \] where \(\nu\) is the second coordinate of a point on the line. In this case we can take \(l=(0,1,-\nu)^t\). Similarly, the implicit equation of a vertical line is \[ L_V=\{u:l^tu=u^1-\nu=0\}\;, \] where \(\nu\) is now the first coordinate of a point on the line. In this case we can take \(l=(1,0,-\nu)^t\). There is a unique plane \(P\) containing this line \(L\) and the center of projection. For each image point with image coordinates \(u\) on the line \(L\), the line containing this point and the center of projection is contained in \(P\). Let \(p\) be a point on the plane \(P\) with world coordinates \(p_W\) projecting onto an image point with image coordinates \(u\). Since these two vectors of coordinates satisfy Equation \ref{eq:general-pinhole-projection}, for which \(\lambda\,u = R\,p_W\,+\,T\), and the vector \(u\) satisfies the implicit equation defining the line \(L\), we have \[ 0 = \lambda l^tu = l^t(R\,p_W\,+\,T)= (R^tl)^t\,(p_W - (-R^tT))\;. \] In conclusion, the implicit representation of plane \(P\), corresponding to Equation \ref{eq:plane-implicit} for which \(P=\{p:n^t(p-q)=0\}\), can be obtained with \(n\) being the vector with world coordinates \(n_W=R^tl\) and \(q\) the point with world coordinates \(q_W=-R^tT\), which is the center of projection.