This paper determines the relationship between the geometry of retractions and the combinatorics of folded galleries for arbitrary affine buildings, and so provides a unified framework to study orbits in affine flag varieties. We introduce the notion of labeled folded galleries for any affine building X and use these to describe the preimages of chimney retractions. When X is the building for a group with an affine Tits system, such as the Bruhat-Tits building for a group over a local field, we can then relate labeled folded galleries and shadows to double coset intersections in affine flag varieties. This result generalizes the authors' previous joint work with Naqvi on groups over function fields.