In this paper, we present a novel framework of 3D scene based beam selection for mmWave communications that relies only on the environmental data and deep learning techniques. Different from other out-of-band side-information aided communication strategies, the proposed one fully utilizes the environmental information, e.g., the shape, the position, and even the materials of the surrounding buildings/cars/trees that are obtained from 3D scene reconstruction. Specifically, we build the neural networks with the input as point cloud of the 3D scene and the output as the beam indices. Compared with the LIDAR aided technique, the reconstructed 3D scene here is achieved from multiple images taken offline from cameras and thus significantly lowers down the cost and makes itself applicable for small mobile terminals. Simulation results show that the proposed 3D scene based beam selection can outperform the LIDAR method in terms of accuracy.