Traditionally stationarity refers to shift invariance of the distribution of a stochastic process. In this paper, we rediscover stationarity as a path property instead of a distributional property. More precisely, we characterize a set of paths denoted as $A$, which corresponds to the notion of stationarity. On one hand, the set $A$ is shown to be large enough, so that for any stationary process, almost all of its paths are in $A$. On the other hand, we prove that any path in $A$ will behave in the optimal way under any stationarity test satisfying some mild conditions. The results provide a unified framework to understand and assess the existing time series tests for stationarity, and can potentially lead to new families of stationarity tests.