In this paper, we consider an inference problem for the first order autoregressive process driven by a long memory stationary Gaussian process. Suppose that the covariance function of the noise can be expressed as $\abs{k}^{2H-2}$ times a function slowly varying at infinity. The fractional Gaussian noise and the fractional ARIMA model and some others Gaussian noise are special examples that satisfy this assumption. We propose a second moment estimator and prove the strong consistency and give the asymptotic distribution. Moreover, when the limit distribution is Gaussian, we give the upper Berry-Esséen bound by means of Fourth moment theorem.