Industrial control systems (ICS), which in many cases are components of critical national infrastructure, are increasingly being connected to other networks and the wider internet motivated by factors such as enhanced operational functionality and improved efficiency. However, set in this context, it is easy to see that the cyber attack surface of these systems is expanding, making it more important than ever that innovative solutions for securing ICS be developed and that the limitations of these solutions are well understood. The development of anomaly based intrusion detection techniques has provided capability for protecting ICS from the serious physical damage that cyber breaches are capable of delivering to them by monitoring sensor and control signals for abnormal activity. Recently, the use of so-called stealthy attacks has been demonstrated where the injection of false sensor measurements can be used to mimic normal control system signals, thereby defeating anomaly detectors whilst still delivering attack objectives. In this paper we define a deep learning-based framework which allows an attacker to conduct stealthy attacks with minimal a-priori knowledge of the target ICS. Specifically, we show that by intercepting the sensor and/or control signals in an ICS for a period of time, a malicious program is able to automatically learn to generate high-quality stealthy attacks which can achieve specific attack goals whilst bypassing a black box anomaly detector. Furthermore, we demonstrate the effectiveness of our framework for conducting stealthy attacks using two real-world ICS case studies. We contend that our results motivate greater attention on this area by the security community as we demonstrate that currently assumed barriers for the successful execution of such attacks are relaxed.