It is known that, interference classification plays an important role in protecting the authorized communication system and avoiding its performance degradation in the hostile environment. In this paper, the interference classification problem for the frequency hopping communication system is discussed. Considering the possibility of presence multiple interferences in the frequency hopping system, in order to fully extract effective features of the interferences from the received signals, the linear and bilinear transform based composite time-frequency analysis method is adopted. Then the time-frequency spectrograms obtained from the time-frequency analysis are constructed as matching pairs and input to the deep neural network for classification. In particular, the Siamese neural network is used as the classifier, where the paired spectrograms are input into the two sub-networks of the deep networks, and these two sub-networks extract the features of the paired spectrograms for interference type classification. The simulation results confirm that the proposed algorithm can obtain higher classification accuracy than both traditional single time-frequency representation based approach and the AlexNet transfer learning or convolutional neural network based methods.