We study various properties of $f$-divergences and Csiszár indices between two probability distributions in very general setups for the convex function $f$ and for the probability distributions. We establish general structural properties of $f$-divergences and show how they are inherited by the associated Csiszár indices, including monotonicity and invariance under suitable transformations. We also study the relationship between Csiszár indices and copula representations of random vectors. When the marginal distributions have atoms, the copula representation is not unique and the Csiszár index of the transformed vectors may increase. We build a large family of interpolating copulas which minimize the Csiszár index and thus preserve the dependence structure of the initial vector.