We prove that the initial degenerations of the flag variety admit closed immersions into finite inverse limits of flag matroid strata, where the diagrams are derived from matroidal subdivisions of a suitable flag matroid polytope. As an application, we prove that the initial degenerations of $\operatorname{F\ell}^{\circ}(n)$ -- the open subvariety of the complete flag variety $\operatorname{F\ell}(n)$ consisting of flags in general position -- are smooth and irreducible when $n\leq 4$. We also study the Chow quotient of $\operatorname{F\ell}(n)$ by the diagonal torus of $\operatorname{PGL}(n)$, and show that, for $n=4$, this is a log crepant resolution of its log canonical model.