For 3D Synthetic Aperture Radar (SAR) imaging, one typical approach is to achieve the cross-track 1D focusing for each range-azimuth pixel after obtaining a stack of 2D complex-valued images. The cross-track focusing is the main difficulty as its aperture length is limited and the antenna positions are usually non-uniformly distributed. Sparsity regularization methods are widely used to tackle these problems. However, these methods are of obvious limitations. The most well-known ones are their heavy computational burdens and unsatisfied stabilities. In this letter, an efficient deep network-based cross-track imaging method is proposed. When trained, the imaging process, i.e. the forward propagation of the network, is made up of simple matrix-vector calculations and element-wise nonlinearity operations, which significantly speed up the imaging. Also, we find that the deep network is of good robustness against noise and model errors. Comprehensive simulations and experiments have been carried out, and the superiority of the proposed method can be clearly seen.