In an Industrial Control System (ICS), its complex network of sensors, actuators and controllers have raised security concerns for critical infrastructures and industrial production units. This opinion paper strives to initiate discussion on the design algorithms which can segregate attacks from faults. Most of the proposed anomaly detection mechanisms are not able to differentiate between an attack and an anomaly due to a fault. We argue on the need of solving this important problem form our experiences in CPS security research. First, we motivate using analysis of studies and interviews though economical and psychological aspects. Then main challenges are highlighted. Further, we propose multiple directions of approach with suitable reasoning and examples from ICS systems.