This paper derives novel results on the characterization of the the causal information rate-distortion function (IRDF) $R_{c}^{it}(D)$ for arbitrarily-distributed one-sided stationary $κ$-th order Markov source x(1),x(2),.... It is first shown that Gorbunov and Pinsker's results on the stationarity of the realizations to the causal IRDF (stated for two-sided stationary sources) do not apply to the commonly used family of asymptotic average single-letter (AASL) distortion criteria. Moreover, we show that, in general, a reconstruction sequence cannot be both jointly stationary with a one-sided stationary source sequence and causally related to it. This implies that, in general, the causal IRDF for one-sided stationary sources cannot be realized by a stationary distribution. However, we prove that for an arbitrarily distributed one-sided stationary source and a large class of distortion criteria (including AASL), the search for $R_{c}^{it}(D)$ can be restricted to distributions which yield the output sequence y(1), y(2),... jointly stationary with the source after $κ$ samples. Finally, we improve the definition of the stationary causal IRDF $\overline{R}_{c}^{it}(D)$ previously introduced by Derpich and Østergaard for two-sided Markovian stationary sources and show that $\overline{R}_{c}^{it}(D)$ for a two-sided source ...,x(-1),x(0),x(1),... equals $R_{c}^{it}(D)$ for the associated one-sided source x(1), x(2),.... This implies that, for the Gaussian quadratic case, the practical zero-delay encoder-decoder pairs proposed by Derpich and Østergaard for approaching $R_{c}^{it}(D)$ achieve an operational data rate which exceeds $R_{c}^{it}(D)$ by less than $1+0.5 \log_2(2 πe /12) \simeq 1.254$ bits per sample.