In this note, we show that extremal Szemerédi-Trotter configurations are rigid in the following sense: If $P,L$ are sets of points and lines determining at least $C|P|^{2/3}|L|^{2/3}$ incidences, then there exists a collection $P'$ of points of size at most $k = k_0(C)$ such that, heuristically, fixing those points fixes a positive fraction of the arrangement. That is, the incidence structure and a small number of points determine a large part of the arrangement. The key tools we use are the Guth-Katz polynomial partitioning, and also a result of Dvir, Garg, Oliveira and Solymosi that was used to show the rigidity of near-Sylvester-Gallai configurations.