In this paper, we consider an inference problem for the first order autoregressive process with non-zero mean driven by a long memory stationary Gaussian process. Suppose that the covariance function of the noise can be expressed as $|k|^{2H-2}$ times a positive constant when $k$ tends to infinity, and the fractional Gaussian noise and the fractional ARIMA model are special examples that satisfy this assumption. We propose moment estimators and prove the strong consistency, the asymptotic normality and joint asymptotic normality.