Radio resource allocation often calls for the optimization of black-box objective functions whose evaluation is expensive in real-world deployments. Conventional optimization methods apply separately to each new system configuration, causing the number of evaluations to be impractical under constraints on computational resources or timeliness. Toward a remedy for this issue, this paper introduces a multi-fidelity continual optimization framework that hinges on a novel information-theoretic acquisition function. The new strategy probes candidate solutions so as to balance the need to retrieve information about the current optimization task with the goal of acquiring information transferable to future resource allocation tasks, while satisfying a query budget constraint. Experiments on uplink power control in a multi-cell multi-antenna system demonstrate that the proposed method substantially improves the optimization efficiency after processing a sufficiently large number of tasks.