In this paper, we propose a new downlink beamforming strategy for mmWave communications using uplink sub-6GHz channel information and a very few mmWave pilots. Specifically, we design a novel dual-input neural network, called FusionNet, to extract and exploit the features from sub-6GHz channel and a few mmWave pilots to accurately predict mmWave beam. To further improve the beamforming performance and avoid over-fitting, we develop two data pre-processing approaches utilizing channel sparsity and data augmentation. The simulation results demonstrate superior performance and robustness of the proposed strategy compared to the existing one that purely relies on the sub-6GHz information, especially in the low signal-to-noise ratio (SNR) regions.