In this paper, we consider the statistical inference of the drift parameter $θ$ of non-ergodic Ornstein-Uhlenbeck~(O-U) process driven by a general Gaussian process $(G_t)_{t\ge 0}$. When $H \in (0, \frac 12) \cup (\frac 12,1) $ the second order mixed partial derivative of $R (t, s) = E [G_t G_s] $ can be decomposed into two parts, one of which coincides with that of fractional Brownian motion (fBm), and the other of which is bounded by $|ts|^{H-1}$. This condition covers a large number of common Gaussian processes such as fBm, sub-fractional Brownian motion and bi-fractional Brownian motion. Under this condition, we verify that $(G_t)_{t\ge 0}$ satisfies the four assumptions in references \cite{El2016}, that is, noise has Hölder continuous path; the variance of noise is bounded by the power function; the asymptotic variance of the solution $X_T$ in the case of ergodic O-U process $X$ exists and strictly positive as $T \to \infty$; for fixed $s \in [0,T)$, the noise $G_s$ is asymptotically independent of the ergodic solution $X_T$ as $T \to \infty$, thus ensure the strong consistency and the asymptotic distribution of the estimator $\tildeθ_T$ based on continuous observations of $X$. Verify that $(G_t)_{t\ge 0}$ satisfies the assumption in references \cite{Es-Sebaiy2019}, that is, the variance of the increment process $\{ ζ_{t_i}-ζ_{t_{i -1}}, i =1,..., n \}$ is bounded by the product of a power function and a negative exponential function, which ensure that $\hatθ_n$ and $\checkθ_n $ are strong consistent and the sequences $\sqrt{T_n} (\hat θ_n - θ)$ and $\sqrt {T_n} (\check θ_n - θ)$ are tight based on discrete observations of $X$