Enabling highly-mobile millimeter wave (mmWave) systems is challenging because of the huge training overhead associated with acquiring the channel knowledge or designing the narrow beams. Current mmWave beam training and channel estimation techniques do not normally make use of the prior beam training or channel estimation observations. Intuitively, though, the channel matrices are functions of the various elements of the environment. Learning these functions can dramatically reduce the training overhead needed to obtain the channel knowledge. In this paper, a novel solution that exploits machine learning tools, namely conditional generative adversarial networks (GAN), is developed to learn these functions between the environment and the channel covariance matrices. More specifically, the proposed machine learning model treats the covariance matrices as 2D images and learns the mapping function relating the uplink received pilots, which act as RF signatures of the environment, and these images. Simulation results show that the developed strategy efficiently predicts the covariance matrices of the large-dimensional mmWave channels with negligible training overhead.