We establish a framework for weak and strong convergence of matrix models to operator-valued semicircular systems parametrized by operator-valued covariance matrices $η= (η_{i,j})_{i,j \in I}$. Non-commutative polynomials are replaced by covariance polynomials that can involve iterated applications of $η_{i,j}$, leading to the notion of covariance laws. We give sufficient conditions for weak and strong convergence of general Gaussian random matrices and deterministic matrices to a $B$-valued semicircular family and generators of the base algebra $B$. In particular, we obtain operator-valued strong convergence for continuously weighted Gaussian Wigner matrices, such as Gaussian band matrices with a continuous cutoff, and we construct natural strongly convergent matrix models for interpolated free group factors.