We prove that the spectrum of the stochastic Airy operator is rigid in the sense of Ghosh and Peres (Duke Math. J., 166(10):1789--1858, 2017) for Dirichlet and Robin boundary conditions. This proves the rigidity of the Airy-$β$ point process and the soft-edge limit of rank-$1$ perturbations of Gaussian $β$-Ensembles for any $β>0$, and solves an open problem mentioned in a previous work of Bufetov, Nikitin, and Qiu (Mosc. Math. J., 19(2):217--274, 2019). Our proof uses a combination of the semigroup theory of the stochastic Airy operator and the techniques for studying insertion and deletion tolerance of point processes developed by Holroyd and Soo (Electron. J. Probab., 18:no. 74, 24, 2013).