We focus on an alignment-free method to estimate the underlying signal from a large number of noisy randomly shifted observations. Specifically, we estimate the mean, power spectrum, and bispectrum of the signal from the observations. Since bispectrum contains the phase information of the signal, reliable algorithms for bispectrum inversion is useful in many applications. We propose a new algorithm using spectral decomposition of the normalized bispectrum matrix for this task. For clean signals, we show that the eigenvectors of the normalized bispectrum matrix correspond to the true phases of the signal and its shifted copies. In addition, the spectral method is robust to noise. It can be used as a stable and efficient initialization technique for local non-convex optimization for bispectrum inversion.