We develop a new approach to address some classical questions concerning the size and structure of integer distance sets. Our main result is that any integer distance set in the Euclidean plane is either very sparse or has all but an exceedingly small proportion of its points lying on a single line or circle. From this, we deduce a near-optimal lower bound on the diameter of any non-collinear integer distance set of size $n$ and a strong upper bound on the size of any integer distance set in $[-N,N]^2$ with no three points on a line and no four points on a circle.