A continuous-time Markov process $X$ can be conditioned to be in a given state at a fixed time $T > 0$ using Doob's $h$-transform. This transform requires the typically intractable transition density of $X$. The effect of the $h$-transform can be described as introducing a guiding force on the process. Replacing this force with an approximation defines the wider class of guided processes. For certain approximations the law of a guided process approximates - and is equivalent to - the actual conditional distribution, with tractable likelihood-ratio. The main contribution of this paper is to prove that the principle of a guided process, introduced in Schauer et al. (2017) for stochastic differential equations, can be extended to a more general class of Markov processes. In particular we apply the guiding technique to jump processes in discrete state spaces. The Markov process perspective enables us to improve upon existing results for hypo-elliptic diffusions.