This paper investigates a novel unmanned aerial vehicles (UAVs) secure communication system with the assistance of reconfigurable intelligent surfaces (RISs), where an UAV and a ground user communicate with each other, while an eavesdropper tends to wiretap their information. Due to the limited capacity of UAVs, a RIS is applied to further improve the quality of the secure communications. The time division multiple access (TDMA) protocol is applied for the communications between the UAV and the ground user, namely, the downlink (DL) and the uplink (UL) communications. In particular, the channel state information (CSI) of the eavesdropping channels is assumed to be imperfect. We aim to maximize the average worst-case secrecy rate by the robust joint design of the UAV's trajectory, RIS's passive beamforming, and transmit power of the legitimate transmitters. It is challenging to solve the Joint UL/DL optimization problem due to its non-convexity. To this end, we develop an efficient algorithm based on the alternating optimization (AO) technique. Specifically, the formulated problem is divided into three sub-problems, and the successive convex approximation (SCA), S-Procedure, and semidefinite relaxation (SDR) are applied to tackle these non-convex sub-problems. Numerical results demonstrate that the proposed algorithm can considerably improve the average secrecy rate compared with the benchmark algorithms, and also confirm the robustness of the proposed algorithm.