The generalization for different scenarios and dif-ferent users is an urgent problem for millimeter wave gesture recognition for indoor fiber-to-the-room (FTTR) scenario. In order to solve this problem and verify the feasibility of FTTR Q-band millimeter wave in gesture recognition, we build a real-time millimeter wave gesture recognition system. The moving hand-gestures are represented as a variety of time-variant spec-trum features, such as micro-Doppler feature, and then the feature learning and classification is realized by using a convo-lution neural network (CNN). The experimental results show that the millimeter wave gesture recognition system can achieve the generalized gesture recognition for 2 scenarios and 4 users.