We consider quadratic forms of deterministic matrices $A$ evaluated at the random eigenvectors of a large $N \times N$ GOE or GUE matrix, or equivalently evaluated at the columns of a Haar-orthogonal or Haar-unitary random matrix. We prove that, as long as the deterministic matrix has rank much smaller than $\sqrt{N}$, the distributions of the extrema of these quadratic forms are asymptotically the same as if the eigenvectors were independent Gaussians. This reduces the problem to Gaussian computations, which we carry out in several cases to illustrate our result, finding Gumbel or Weibull limiting distributions depending on the signature of $A$. Our result also naturally applies to the eigenvectors of any invariant ensemble.