In time-varying fading channels, channel coefficients are estimated using pilot symbols that are transmitted every coherence interval. For channels with high Doppler spread, the rapid channel variations over time will require considerable bandwidth for pilot transmission, leading to poor throughput. In this paper, we propose a novel receiver architecture using deep recurrent neural networks (RNNs) that learns the channel variations and thereby reduces the number of pilot symbols required for channel estimation. Specifically, we design and train an RNN to learn the correlation in the time-varying channel and predict the channel coefficients into the future with good accuracy over a wide range of Dopplers and signal-to-noise ratios (SNR). The proposed training methodology enables accurate channel prediction through the use of techniques such as teacher-force training, early-stop, and reduction of learning rate on plateau. Also, the robustness of prediction for different Dopplers and SNRs is achieved by adapting the number of predictions into the future based on the Doppler and SNR. Numerical results show that good bit error performance is achieved by the proposed receiver in time-varying fading channels. We also propose a data decision driven receiver architecture using RNNs that further reduces the pilot overhead while maintaining good bit error performance.