We prove that independent families of permutation invariant random matrices are asymptotically free over the diagonal, both in probability and in expectation, under a uniform boundedness assumption on the operator norm. We can relax the operator norm assumption to an estimate on sums associated to graphs of matrices, further extending the range of applications (for example, to Wigner matrices with exploding moments and so the sparse regime of the Erdős-Rényi model). The result still holds even if the matrices are multiplied entrywise by bounded random variables (for example, as in the case of matrices with a variance profile and percolation models).