We study Swendsen--Wang dynamics for the critical $q$-state Potts model on the square lattice. For $q=2,3,4$, where the phase transition is continuous, the mixing time $t_{\textrm{mix}}$ is expected to obey a universal power-law independent of the boundary conditions. On the other hand, for large $q$, where the phase transition is discontinuous, the authors recently showed that $t_{\textrm{mix}}$ is highly sensitive to boundary conditions: $t_{\textrm{mix}} \geq \exp(cn)$ on an $n\times n$ box with periodic boundary, yet under free or monochromatic boundary conditions, $t_{\textrm{mix}} \leq\exp(n^{o(1)})$. In this work we classify this effect under boundary conditions that interpolate between these two (torus vs. free/monochromatic). Specifically, if one of the $q$ colors is red, mixed boundary conditions such as red-free-red-free on the 4 sides of the box induce $t_{\textrm{mix}} \geq \exp(cn)$, yet Dobrushin boundary conditions such as red-red-free-free, as well as red-periodic-red-periodic, induce sub-exponential mixing.