The objective of this paper is to evaluate the effectiveness of a two-timescale transmission design in cell-free massive multi-input multiple-output (MIMO) systems incorporating reconfigurable intelligent surfaces (RISs) under the assumption of imperfect channel state information (CSI). We examine the Rician channel model and formulate the passive beamforming for the RISs based on statistical channel state information (S-CSI). To that end, we put forth a linear minimum mean square error (LMMSE) estimator with the aim of estimating the aggregation of channels from the users to the APs within each channel coherence interval. Meanwhile, the active beamforming for the radio units (APs) is executed using the maximum ratio combining (MRC) approach, which utilizes the instantaneous aggregated channels, that result from the combination of the direct and reflected channels from the RISs. Subsequently, we derive the closed-form expressions of the achievable uplink spectral efficiency (SE), which is a function of S-CSI elements such as distance-dependent path loss, Rician factors as well as the number of RIS elements and AP antennas. We then optimize the phase shifts of the RISs to maximize the sum SE of the users, utilizing the soft actor-critic (SAC) which is a deep reinforcement learning (RL) method, and relying on the derived closed-form expressions. Numerical evaluations affirm that, despite the presence of imperfect CSI, the deployment of RIS in cell-free systems can lead to significant performance improvement.