We study when optimal Bayesian estimators under Gaussian noise are approximately linear, and what this implies about the underlying prior distribution. Consider the classical model \(Y = X + Z\), where \(Z\) is Gaussian and independent of \(X\). It is well known that under squared-error loss, the conditional mean \(\mathbb{E}[X|Y]\) is a linear function of \(Y\) if and only if the prior is Gaussian. Much less is understood under absolute-error loss, where the optimal estimator is the conditional median and standard orthogonality-based tools no longer apply. Recent work has established that, in the Gaussian noise model, the Gaussian prior is also the unique distribution that induces an exactly linear conditional median. In this paper, we move beyond exact characterizations and develop a quantitative stability theory: if the optimal estimator is approximately linear, must the prior be close to Gaussian? For the \(L_2\) setting, we derive explicit rates showing that near-linearity of the conditional mean forces the prior to be close to Gaussian in the Levy metric. For the \(L_1\) setting, we develop a functional-analytic framework based on Hermite expansions and adjoint operators, establishing that approximate linearity of the conditional median implies proximity to the Gaussian family.