We study matrices whose entries are free or exchangeable noncommutative elements in some tracial $W^*$-probability space. More precisely, we consider operator-valued Wigner and Wishart matrices and prove quantitative convergence to operator-valued semicircular elements over some subalgebra in terms of Cauchy transforms. As direct applications, we obtain explicit rates of convergence for a large class of random block matrices with independent or correlated blocks. Our approach relies on a noncommutative extension of the Lindeberg method and operator-valued Gaussian interpolation techniques.