Early and accurately detecting faults in rotating machinery is crucial for operation safety of the modern manufacturing system. In this paper, we proposed a novel Deep fault diagnosis (DFD) method for rotating machinery with scarce labeled samples. DFD tackles the challenging problem by transferring knowledge from shallow models, which is based on the idea that shallow models trained with different hand-crafted features can reveal the latent prior knowledge and diagnostic expertise and have good generalization ability even with scarce labeled samples. DFD can be divided into three phases. First, a spectrogram of the raw vibration signal is calculated by applying a Short-time Fourier transform (STFT). From those spectrograms, discriminative time-frequency domain features can be extracted and used to form a feature pool. Then, several candidate Support vector machine (SVM) models are trained with different combinations of features in the feature pool with scarce labeled samples. By evaluating the pretrained SVM models on the validation set, the most discriminative features and best-performed SVM models can be selected, which are used to make predictions on the unlabeled samples. The predicted labels reserve the expert knowledge originally carried by the SVM model. They are combined together with the scarce fine labeled samples to form an Augmented training set (ATS). Finally, a novel 2D deep Convolutional neural network (CNN) model is trained on the ATS to learn more discriminative features and a better classifier. Experimental results on two fault diagnosis datasets demonstrate the effectiveness of the proposed DFD, which achieves better performance than SVM models and the vanilla deep CNN model trained on scarce labeled samples. Moreover, it is computationally efficient and is promising for real-time rotating machinery fault diagnosis.