We show that for any large $n$, there exists a set of $n$ points in the plane with $O(n^2/\sqrt{\log n})$ distinct distances, such that any four points in the set determine at least five distinct distances. This answers (in the negative) a question of Erdős. The proof combines an analysis by Dumitrescu of forbidden four-point patterns with an algebraic construction of Thiele and Dumitrescu (to eliminate parallelograms), as well as a randomized transformation of that construction (to eliminate most other forbidden patterns).