Modulation recognition using deep neural networks has shown promising advantages over conventional algorithms. However, most existing research focuses on single receive antenna. In this paper, two end-to-end feature learning deep architectures are introduced for modulation recognition with multiple receive antennas. The first is based on multi-view convolutional neural network by treating signals from different receive antennas as different views of a 3D object and designing the location and operation of view-pooling layer that are suitable for feature fusion of multi-antenna signals. Considering that the instantaneous SNRs could be different among receive antennas in wireless communications, we further propose weight-learning convolutional neural network which uses a weight-learning module to automatically learn the weights for feature combing of different receive antennas to perform end-to-end feature learning of multi-antenna signals. Results show that both end-to-end feature learning deep architectures outperform the existing algorithm, and the proposed weight-learning convolutional neural network achieves the best performance.