Unmanned aerial vehicle (UAV) communication is anticipated to be widely applied in the forthcoming fifth-generation (5G) wireless networks, due to its many advantages such as low cost, high mobility, and on-demand deployment. However, the broadcast and line-of-sight (LoS) nature of air-to-ground wireless channels gives rise to a new challenge on how to realize secure UAV communications with the destined nodes on the ground. This paper aims to tackle this challenge by applying the physical layer security technique. We consider both the downlink and uplink UAV communications with a ground node, namely UAV-to-ground (U2G) and ground-to-UAV (G2U) communications, respectively, subject to a potential eavesdropper on the ground. In contrast to the existing literature on wireless physical layer security only with ground nodes at fixed or quasi-static locations, we exploit the high mobility of the UAV to proactively establish favorable and degraded channels for the legitimate and eavesdropping links, respectively, via its trajectory design. We formulate new problems to maximize the average secrecy rates of the U2G and G2U transmissions, respectively, by jointly optimizing the UAV's trajectory and the transmit power of the legitimate transmitter over a given flight period of the UAV. Although the formulated problems are non-convex, we propose iterative algorithms to solve them efficiently by applying the block coordinate descent and successive convex optimization methods. Specifically, the transmit power and UAV trajectory are each optimized with the other being fixed in an alternating manner, until the algorithms converge. Simulation results show that the proposed algorithms can improve the secrecy rates for both U2G and G2U communications, as compared to other benchmark schemes without power control and/or trajectory optimization.