Federated edge learning (FEL) is a promising paradigm of distributed machine learning that can preserve data privacy while training the global model collaboratively. However, FEL is still facing model confidentiality issues due to eavesdropping risks of exchanging cryptographic keys through traditional encryption schemes. Therefore, in this paper, we propose a hierarchical architecture for quantum-secured FEL systems with ideal security based on the quantum key distribution (QKD) to facilitate public key and model encryption against eavesdropping attacks. Specifically, we propose a stochastic resource allocation model for efficient QKD to encrypt FEL keys and models. In FEL systems, remote FEL workers are connected to cluster heads via quantum-secured channels to train an aggregated global model collaboratively. However, due to the unpredictable number of workers at each location, the demand for secret-key rates to support secure model transmission to the server is unpredictable. The proposed systems need to efficiently allocate limited QKD resources (i.e., wavelengths) such that the total cost is minimized in the presence of stochastic demand by formulating the optimization problem for the proposed architecture as a stochastic programming model. To this end, we propose a federated reinforcement learning-based resource allocation scheme to solve the proposed model without complete state information. The proposed scheme enables QKD managers and controllers to train a global QKD resource allocation policy while keeping their private experiences local. Numerical results demonstrate that the proposed schemes can successfully achieve the cost-minimizing objective under uncertain demand while improving the training efficiency by about 50\% compared to state-of-the-art schemes.