Millimeter-wave (mmWave) hybrid analog-digital beamforming is a promising approach to satisfy the low-latency constraint in multiple unmanned aerial vehicles (UAVs) systems, which serve as network infrastructure for flexible deployment. However, in highly dynamic multi-UAV environments, analog beam tracking becomes a critical challenge. The overhead of additional pilot transmission at the price of spectral efficiency is shown necessary to achieve high resilience in operation. An efficient method to deal with high dynamics of UAVs applies machine learning, particularly Q-learning, to analog beam tracking. The proposed Q-learning-based beam tracking scheme uses current/past observations to design rewards from environments to facilitate prediction, which significantly increases the efficiency of data transmission and beam switching. Given the selected analog beams, the goal of digital beamforming is to maximize the SINR. The received pilot signals are utilized to approximate the desired signal and interference power, which yield the SINR measurements as well as the optimal digital weights. Since the selected analog beams based on the received power do not guarantee the hybrid beamforming achieving the maximization SINR, we therefore reserve additional analog beams as candidates during the beam tracking. The combination of analog beams with their digital weights achieving the maximum SINR consequently provides the optimal solution to the hybrid beamforming.