We prove a stability version of a general result that bounds the permanent of a matrix in terms of its operator norm. More specifically, suppose $A$ is an $n \times n$ matrix over $\mathbb{C}$ (resp. $\mathbb{R}$), and let $\mathcal{P}$ denote the set of $n \times n$ matrices over $\mathbb{C}$ (resp. $\mathbb{R}$) that can be written as a permutation matrix times a unitary diagonal matrix. Then it is known that the permanent of $A$ satisfies $|\text{perm}(A)| \leq \Vert A \Vert_{2} ^n$ with equality iff $A/ \Vert A \Vert_{2} \in \mathcal{P}$ (where $\Vert A \Vert_2$ is the operator $2$-norm of $A$). We show a stability version of this result asserting that unless $A$ is very close (in a particular sense) to one of these extremal matrices, its permanent is exponentially smaller (as a function of $n$) than $\Vert A \Vert_2 ^n$. In particular, for any fixed $α, β> 0$, we show that $|\text{perm}(A)|$ is exponentially smaller than $\Vert A \Vert_2 ^n$ unless all but at most $αn$ rows contain entries of modulus at least $\Vert A \Vert_2 (1 - β)$.