We provide a perturbative expansion for the empirical spectral distribution of a Hermitian matrix with large size perturbed by a random matrix with small operator norm whose entries in the eigenvector basis of the first one are independent with a variance profile. We prove that, depending on the order of magnitude of the perturbation, several regimes can appear, called perturbative and semi-perturbative regimes. Depending on the regime, the leading terms of the expansion are either related to the one-dimensional Gaussian free field or to free probability theory.