In the study of open quantum systems modeled by a unitary evolution of a bipartite Hilbert space, we address the question of which parts of the environment can be said to have a "classical action" on the system, in the sense of acting as a classical stochastic process. Our method relies on the definition of the Environment Algebra, a relevant von Neumann algebra of the environment. With this algebra we define the classical parts of the environment and prove a decomposition between a maximal classical part and a quantum part. Then we investigate what other information can be obtained via this algebra, which leads us to define a more pertinent algebra: the Environment Action Algebra. This second algebra is linked to the minimal Stinespring representations induced by the unitary evolution on the system. Finally in finite dimension we give a characterization of both algebras in terms of the spectrum of a certain completely positive map acting on the states of the environment.