According to a classical result of Spencer, Szemerédi, and Trotter (1984), the maximum number of times the unit distance can occur among $n$ points in the plane is $O(n^{4/3})$. This is far from Erdős's lower bound, $n^{1+O(1/\log\log n)}$, which is conjectured to be optimal. We prove a structural result for point sets with nearly $n^{4/3}$ unit distances and use it to reduce the problem to a conjecture on rigid frameworks. This conjecture, if true, would yield the first improvement on the bound of Spencer et al. A weaker version of this conjecture has been established by the last two authors.