In this thesis, we develop analytical methods to study out-of-equilibrium stochastic processes driven by colored noise, i.e., noise with temporal correlations. These non-Markovian processes pose significant analytical challenges compared to processes driven by white noise, such as Brownian motion. A primary focus is on active particle systems, specifically the run-and-tumble particle subjected to an arbitrary force. We derive exact expressions for its mean first-passage time (MFPT) and exit probability from an interval using the backward Fokker-Planck equation. Remarkably, we find that the MFPT can be optimized as a function of the tumbling rate. Additionally, we investigate stochastic resetting and switching diffusion models. For switching diffusion models which are examples of "Brownian yet non-Gaussian diffusions", we use a renewal approach and large deviation theory to derive exact results for various observables. These include the distribution of the position of the particle and its moments, but also its cumulants which are key observables to characterize non-Gaussian fluctuations. Notably, we uncover an unexpected connection between this model and free cumulants. We also examine these models in the presence of a harmonic potential by using Kesten variables. This approach enables us to write an integral equation for the steady-state distribution, which we solve in specific cases. Furthermore, we extend Siegmund duality - a concept that is not widely known in the physics literature - to active particles, random diffusion models, stochastic resetting, and continuous-time random walks. This duality establishes a direct relation between first passage observables and the spatial properties of a dual process, which we explicitly construct.