We consider the problem of inferring the interaction kernel of stochastic interacting particle systems from observations of a single particle. We adopt a semi-parametric approach and represent the interaction kernel in terms of a generalized Fourier series. The basis functions in this expansion are tailored to the problem at hand and are chosen to be orthogonal polynomials with respect to the invariant measure of the mean-field dynamics. The generalized Fourier coefficients are obtained as the solution of an appropriate linear system whose coefficients depend on the moments of the invariant measure, and which are approximated from the particle trajectory that we observe. We quantify the approximation error in the Lebesgue space weighted by the invariant measure and study the asymptotic properties of the estimator in the joint limit as the observation interval and the number of particles tend to infinity, i.e. the joint large time-mean field limit. We also explore the regime where an increasing number of generalized Fourier coefficients is needed to represent the interaction kernel. Our theoretical results are supported by extensive numerical simulations.