Our goal is to provide simple and practical algorithms in higher-order Fourier analysis which are based on spectral decompositions of operators. We propose a general framework for such algorithms and provide a detailed analysis of the quadratic case. Our results reveal new spectral aspects of the theory underlying higher-order Fourier analysis. Along these lines, we prove new inverse and regularity theorems for the Gowers norms based on higher-order character decompositions. Using these results, we prove a spectral inverse theorem and a spectral regularity theorem in quadratic Fourier analysis.