In this paper, we study the trajectory design for a cellular-connected unmanned aerial vehicle (UAV) with given initial and final locations, while communicating with the ground base stations (GBSs) along its flight. We consider delay-limited communications between the UAV and its associated GBSs, where a given signal-to-noise ratio (SNR) target needs to be satisfied at the receiver. However, in practice, due to various factors such as quality-of-service (QoS) requirement, GBSs' availability and UAV mobility constraints, the SNR target may not be met at certain time periods during the flight, each termed as an outage duration. In this paper, we aim to optimize the UAV trajectory to minimize its mission completion time, subject to a constraint on the maximum tolerable outage duration in its flight. To tackle this non-convex problem, we first transform it into a more tractable form and thereby reveal some useful properties of the optimal trajectory solution. Based on these properties, we then further simplify the problem and propose efficient algorithms to check the feasibility of the problem as well as to obtain its optimal and high-quality suboptimal solutions, by leveraging graph theory and convex optimization techniques. Numerical results show that our proposed trajectory designs outperform the conventional method based on dynamic programming, in terms of both performance and complexity.