We develop further the theory of $q$-deformations of real numbers introduced by Morier-Genoud and Ovsienko, and focus in particular on the class of real quadratic irrationals. Our key tool is a $q$-deformation of the modular group $PSL_q(2,\mathbb{Z})$. The action of the modular group by Möbius transformations commutes with the $q$-deformations. We prove that the traces of the elements of $PSL_q(2,\mathbb{Z})$ are palindromic polynomials with positive coefficients. These traces appear in the explicit expressions of the $q$-deformed quadratic irrationals.