We introduce the notion of a $Y$-pattern with coefficients and its geometric counterpart: a cluster $\mathcal{X}$-variety with coefficients. We use these constructions to build a flat degeneration of every skew-symmetrizable specially completed cluster $\mathcal{X}$-variety $\widehat{\mathcal{X}}$ to the toric variety associated to its $\mathbf{g}$-fan. Moreover, we show that the fibers of this family are stratified in a natural way, with strata the specially completed $\mathcal{X}$-varieties encoded by $\mathrm{Star}(τ)$ for each cone $τ$ of the $\mathbf{g}$-fan. These strata degenerate to the associated toric strata of the central fiber. We further show that the family is cluster dual to $\mathcal{A}_{\mathrm{prin}}$ of Gross-Hacking-Keel-Kontsevich, and the fibers cluster dual to $\mathcal{A}_t$. Finally, we give two applications. First, we use our construction to identify the Rietsch-Williams toric degeneration of Grassmannians with the Gross-Hacking-Keel-Kontsevich degeneration in the case of $\mathrm{Gr}_2(\mathbb{C}^5)$. Next, we use it to link cluster duality to Batyrev-Borisov duality of Gorenstein toric Fanos in the context of mirror symmetry.