We consider non-local Ornstein-Uhlenbeck (OU) operators that correspond to Ornstein-Uhlenbeck processes driven by Lévy processes. These are ergodic Markov processes and the OU operator is in general non-normal in the $L^2$ space weighted with the invariant distribution. Under some mild assumptions on the Lévy process, we carry out in-depth analysis of the spectrum, spectral multilicities, eigenfunctions and co-eigenfunctions (eigenfunctions of the adjoint), and the existence of spectral expansion of the semigroups. When the drift matrix $B$ is diagonalizable, we derive explicit formulas for eigenfunctions and co-eigenfunctions which are also biorthogonal, and such results continue to hold when the Lévy process is a pure jump process. A key ingredient in our approach is \emph{intertwining relationship}: we prove that every Lévy-OU semigroup is intertwined with a diffusion OU semigroup. Additionally, we study the compactness properties of these semigroups and provide some necessary and sufficient conditions for compactness.