We study online federated learning over a wireless network, where the central server updates an online global model sequence to minimize the time-varying loss of multiple local devices over time. The server updates the global model through over-the-air model-difference aggregation from the local devices over a noisy multiple-access fading channel. We consider the practical scenario where information on both the local loss functions and the channel states is delayed, and each local device is under a time-varying power constraint. We propose Constrained Over-the-air Model Updating with Delayed infOrmation (COMUDO), where a new lower-and-upper-bounded virtual queue is introduced to counter the delayed information and control the hard constraint violation. We show that its local model updates can be efficiently computed in closed-form expressions. Furthermore, through a new Lyapunov drift analysis, we show that COMUDO provides bounds on the dynamic regret, static regret, and hard constraint violation. Simulation results on image classification tasks under practical wireless network settings show substantial accuracy gain of COMUDO over state-of-the-art approaches, especially in the low-power region.