We establish quantitative stability bounds for the quadratic optimal transport map $T_μ$ between a fixed probability density $ρ$ and a probability measure $μ$ on $\mathbb{R}^d$. Under general assumptions on $ρ$, we prove that the map $μ\mapsto T_μ$ is bi-Hölder continuous, with dimension-free Hölder exponents. The linearized optimal transport metric $W_{2,ρ}(μ,ν)=\|T_μ-T_ν\|_{L^2(ρ)}$ is therefore bi-Hölder equivalent to the $2$-Wasserstein distance, which justifies its use in applications. We show this property in the following cases: (i) for any log-concave density $ρ$ with full support in $\mathbb{R}^d$, and any log-bounded perturbation thereof; (ii) for $ρ$ bounded away from $0$ and $+\infty$ on a John domain (e.g., on a bounded Lipschitz domain), while the only previously known result of this type assumed convexity of the domain; (iii) for some important families of probability densities on bounded domains which decay or blow-up polynomially near the boundary. Concerning the sharpness of point (ii), we also provide examples of non-John domains for which the Brenier potentials do not satisfy any Hölder stability estimate. Our proofs rely on local variance inequalities for the Brenier potentials in small convex subsets of the support of $ρ$, which are glued together to deduce a global variance inequality. This gluing argument is based on two different strategies of independent interest: one of them leverages the properties of the Whitney decomposition in bounded domains, the other one relies on spectral graph theory.