In this study we show how to represent a continuous time autoregressive moving average (CARMA) as a higher order stochastic delay differential equation, which may be thought of as a continuous-time equivalent of the AR($\infty$) representation. Furthermore, we show how this representation gives rise to a prediction formula for CARMA processes. To be used in the above mentioned results we develop a general theory for multivariate stochastic delay differential equations, which will be of independent interest, and which will have particular focus on existence, uniqueness and representations.