In this work, we study a class of random matrices which interpolate between the Wigner matrix model and various types of patterned random matrices such as random Toeplitz, Hankel, and circulant matrices. The interpolation mechanism is through the correlations of the entries, and thus these interpolating models are highly inhomogeneous in their correlation structure. Historically, the study of random matrices has focused on homogeneous models, i.e., those with imposed structure (such as independence of the entries), as such restrictions significantly simplify computations related to the models. However, in this paper we demonstrate that for these interpolating inhomogenous models, the fluctuations of the linear eigenvalue statistics are approximately Gaussian. To handle the difficulties that come with inhomogeneity in the entries, we incorporate combinatorial arguments and recent tools from non-asymptotic random matrix theory.