Last passage percolation and directed polymer models on $\mathbb Z^2$ are invariant under translation and certain reflections. When these models have an integrable structure coming from either the RSK correspondence or the geometric RSK correspondence (e.g. geometric last passage percolation or the log-gamma polymer), we show that these basic invariances can be combined with a decoupling property to yield a rich new set of symmetries. Among other results, we prove shift and rearrangement invariance statements for last passage times, geodesic locations, disjointness probabilities, polymer partition functions, and quenched polymer measures. We also use our framework to find `scrambled' versions of the classical RSK correspondence, and to find an RSK correspondence for moon polyominoes. The results extend to limiting models, including the KPZ equation and the Airy sheet.