We show that if a finite point set $P\subseteq \mathbb{R}^2$ has the fewest congruence classes of triangles possible, up to a constant $M$, then at least one of the following holds. (1) There is a $σ>0$ and a line $l$ which contains $Ω(|P|^σ)$ points of $P$. Further, a positive proportion of $P$ is covered by lines parallel to $l$ each containing $Ω(|P|^σ)$ points of $P$. (2) There is a circle $γ$ which contains a positive proportion of $P$. This provides evidence for two conjectures of Erdős. We use the result of Petridis-Roche-Newton-Rudnev-Warren on the structure of the affine group combined with classical results from additive combinatorics.