We consider Perron-Frobenius and Koopman operators associated to time-inhomogeneous ordinary stochastic differential equations, and establish their Fréchet differentiability with respect to the drift. This result relies on a similar differentiability result for pathwise expectations of path functionals of the solution of the stochastic differential equation, which we establish using Girsanov's formula. We demonstrate the significance of our result in the context of dynamical systems and operator theory, by proving continuously differentiable drift dependence of the simple eigen- and singular values and the corresponding eigen- and singular functions of the stochastic Perron-Frobenius and Koopman operators.