Cluster integrable systems are a broad class of integrable systems modelled on bipartite dimer models on the torus. Many discrete integrable dynamics arise by applying sequences of local transformations, which form the cluster modular group of the cluster integrable system. This cluster modular group was recently characterized by the first author and Inchiostro. There exist some discrete integrable dynamics that make use of non-local transformations associated with geometric $R$-matrices. In this article we characterize the generalized cluster modular group -- which includes both local and non-local transformations -- in terms of extended affine symmetric groups. We also describe the action of the generalized cluster modular group on the spectral data associated with cluster integrable systems.