In this paper, we consider an inference problem for an Ornstein-Uhlenbeck process driven by a general one-dimensional centered Gaussian process $(G_t)_{t\ge 0}$. The second order mixed partial derivative of the covariance function $ R(t,\, s)=\mathbb{E}[G_t G_s]$ can be decomposed into two parts, one of which coincides with that of fractional Brownian motion and the other is bounded by $(ts)^{β-1}$ up to a constant factor. This condition is valid for a class of continuous Gaussian processes that fails to be self-similar or have stationary increments. Some examples include the subfractional Brownian motion and the bi-fractional Brownian motion. Under this assumption, we study the parameter estimation for drift parameter in the Ornstein-Uhlenbeck process driven by the Gaussian noise $(G_t)_{t\ge 0}$. For the least squares estimator and the second moment estimator constructed from the continuous observations, we prove the strong consistency and the asympotic normality, and obtain the Berry-Esséen bounds. The proof is based on the inner product's representation of the Hilbert space $\mathfrak{H}$ associated with the Gaussian noise $(G_t)_{t\ge 0}$, and the estimation of the inner product based on the results of the Hilbert space associated with the fractional Brownian motion.