Providing reliable cellular connectivity to Unmanned Aerial Vehicles (UAV) is a key challenge, as existing terrestrial networks are deployed mainly for ground-level coverage. The cellular network coverage may be available for a limited range from the antenna side lobes, with poor connectivity further exacerbated by UAV flight dynamics. In this work, we propose TransfoREM, a 3D Radio Environment Map (REM) generation method that combines deterministic channel models and real-world data to map terrestrial network coverage at higher altitudes. At the core of our solution is a transformer model that translates radio propagation mapping into a sequence prediction task to construct REMs. Our results demonstrate that TransfoREM offers improved interpolation capability on real-world data compared against conventional Kriging and other machine learning (ML) techniques. Furthermore, TransfoREM is designed for holistic integration into cellular networks at the base station (BS) level, where it can build REMs, which can then be leveraged for enhanced resource allocation, interference management, and spatial spectrum utilization.