Wireless communication with unmanned aerial vehicles (UAVs) is a promising technology for future communication systems. In this paper, we study energy-efficient UAV communication with a ground terminal via optimizing the UAV's trajectory, a new design paradigm that jointly considers both the communication throughput and the UAV's energy consumption. To this end, we first derive a theoretical model on the propulsion energy consumption of fixed-wing UAVs as a function of the UAV's flying speed, direction and acceleration, based on which the energy efficiency of UAV communication is defined. Then, for the case of unconstrained trajectory optimization, we show that both the rate-maximization and energy-minimization designs lead to vanishing energy efficiency and thus are energy-inefficient in general. Next, we introduce a practical circular UAV trajectory, under which the UAV's flight radius and speed are optimized to maximize the energy efficiency for communication. Furthermore, an efficient design is proposed for maximizing the UAV's energy efficiency with general constraints on its trajectory, including its initial/final locations and velocities, as well as maximum speed and acceleration. Numerical results show that the proposed designs achieve significantly higher energy efficiency for UAV communication as compared with other benchmark schemes.