In this paper, we study the affine Springer fiber $\mathcal{F} l_N$ in type $A$ for rectangular type semisimple nil-element $N$ and calculate the relative position between irreducible components. In particular, we use the affine matrix ball construction to show the relative position map is compatible with the Kazhdan-Lusztig cell structure, generalizing the work of Steinberg and van Leeuwen.