This paper considers a sensor attack and fault detection problem for linear cyber-physical systems, which are subject to system noise that can obey an unknown light-tailed distribution. We propose a new threshold-based detection mechanism that employs the Wasserstein metric, and which guarantees system performance with high confidence employing a finite number of measurements. The proposed detector may generate false alarms with a rate $Δ$ in normal operation, where $Δ$ can be tuned to be arbitrarily small by means of a benchmark distribution which is part of our mechanism. Thus, the proposed detector is sensitive to sensor attacks and faults which have a statistical behavior that is different from that of the system's noise. We quantify the impact of stealthy attacks---which aim to perturb the system operation while producing false alarms that are consistent with the natural system's noise---via a probabilistic reachable set. To enable tractable implementation of our methods, we propose a linear optimization problem that computes the proposed detection measure and a semidefinite program that produces the proposed reachable set.