In this letter, we study a cellular-connected unmanned aerial vehicle (UAV) which aims to complete a mission of flying between two pre-determined locations while maintaining satisfactory communication quality with the ground base stations (GBSs). Due to the potentially long distance of the UAV's flight, frequent handovers may be incurred among different GBSs, which leads to various practical issues such as large delay and synchronization overhead. To address this problem, we investigate the trajectory optimization of the UAV to minimize the number of GBS handovers during the flight, subject to a communication quality constraint and a maximum mission completion time constraint. Although this problem is non-convex and difficult to solve, we derive useful structures of the optimal solution, based on which we propose an efficient algorithm based on graph theory and Lagrangian relaxation for finding a high-quality suboptimal solution in polynomial time. Numerical results validate the effectiveness of our proposed trajectory design.