The unprecedented requirements of the Internet of Things (IoT) have made fine-grained optimization of spectrum resources an urgent necessity. Thus, designing techniques able to extract knowledge from the spectrum in real time and select the optimal spectrum access strategy accordingly has become more important than ever. Moreover, 5G and beyond (5GB) networks will require complex management schemes to deal with problems such as adaptive beam management and rate selection. Although deep learning (DL) has been successful in modeling complex phenomena, commercially-available wireless devices are still very far from actually adopting learning-based techniques to optimize their spectrum usage. In this paper, we first discuss the need for real-time DL at the physical layer, and then summarize the current state of the art and existing limitations. We conclude the paper by discussing an agenda of research challenges and how DL can be applied to address crucial problems in 5GB networks.