This paper investigates the stability of phase retrieval by analyzing the condition number of the nonlinear map $Ψ_{\boldsymbol{A}}(\boldsymbol{x}) = \bigl(\lvert \langle {\boldsymbol{a}}_j, \boldsymbol{x} \rangle \rvert^2 \bigr)_{1 \le j \le m}$, where $\boldsymbol{a}_j \in \mathbb{H}^n$ are known sensing vectors with $\mathbb{H} \in \{\mathbb{R}, \mathbb{C}\}$. For each $p \ge 1$, we define the condition number $β_{Ψ_{\boldsymbol{A}}}^{\ell_p}$ as the ratio of optimal upper and lower Lipschitz constants of $Ψ_{\boldsymbol{A}}$ measured in the $\ell_p$ norm, with respect to the metric $\mathrm {dist}_\mathbb{H}\left(\boldsymbol{x}, \boldsymbol{y}\right) = \|\boldsymbol{x} \boldsymbol{x}^\ast - \boldsymbol{y} \boldsymbol{y}^\ast\|_*$. We establish universal lower bounds on $β_{Ψ_{\boldsymbol{A}}}^{\ell_p}$ for any sensing matrix $\boldsymbol{A} \in \mathbb{H}^{m \times d}$, proving that $β_{Ψ_{\boldsymbol{A}}}^{\ell_1} \ge π/2$ and $β_{Ψ_{\boldsymbol{A}}}^{\ell_2} \ge \sqrt{3}$ in the real case $(\mathbb{H} = \mathbb{R})$, and $β_{Ψ_{\boldsymbol{A}}}^{\ell_p} \ge 2$ for $p=1,2$ in the complex case $(\mathbb{H} = \mathbb{C})$. These bounds are shown to be asymptotically tight: both a deterministic harmonic frame $\boldsymbol{E}_m \in \mathbb{R}^{m \times 2}$ and Gaussian random matrices $\boldsymbol{A} \in \mathbb{H}^{m \times d}$ asymptotically attain them. Notably, the harmonic frame $\boldsymbol{E}_m \in \mathbb{R}^{m \times 2}$ achieves the optimal lower bound $\sqrt{3}$ for all $m \ge 3$ when $p=2$, thus serving as an optimal sensing matrix within $\boldsymbol{A} \in \mathbb{R}^{m \times 2}$. Our results provide the first explicit uniform lower bounds on $β_{Ψ_{\boldsymbol{A}}}^{\ell_p}$ and offer insights into the fundamental stability limits of phase retrieval.