Given $k \leq 6$ points $(x_i,y_i) \in \mathbb{P}^2 \times \mathbb{P}^2$, we characterize rank deficiency of the $k \times 9$ matrix $Z_k$ with rows $x_i^\top \otimes y_i^\top$ in terms of the geometry of the point configurations $\{x_i\}$ and $\{y_i\}$. While this question comes from computer vision the answer relies on tools from classical algebraic geometry: For $k \leq 5$, the geometry of the rank-drop locus is characterized by cross-ratios and basic (projective) geometry of point configurations. For the case $k=6$ the rank-drop locus is captured by the classical theory of cubic surfaces.