Unmanned aerial vehicle (UAV)-based communications is a promising new technology that can add a wide range of new capabilities to the current network infrastructure. Given the flexibility, cost-efficiency, and convenient use of UAVs, they can be deployed as temporary base stations (BSs) for on-demand situations like BS overloading or natural disasters. In this work, a UAV-based communication system with radio frequency (RF) access links to the mobile users (MUs) and a free-space optical (FSO) backhaul link to the ground station (GS) is considered. In particular, the RF and FSO channels in this network depend on the UAV's positioning and (in)stability. The relative position of the UAV with respect to the MUs impacts the likelihood of a line-of-sight (LOS) connection in the RF link and the instability of the hovering UAV affects the quality of the FSO channel. Thus, taking these effects into account, we analyze the end-to-end system performance of networks employing UAVs as buffer-aided (BA) and non-buffer-aided (non-BA) relays in terms of the ergodic sum rate. Simulation results validate the accuracy of the proposed analytical derivations and reveal the benefits of buffering for compensation of the random fluctuations caused by the UAV's instability. Our simulations also show that the ergodic sum rate of both BA and non-BA UAV-based relays can be enhanced considerably by optimizing the positioning of the UAV. We further study the impact of the MU density and the weather conditions on the end-to-end system performance.