Wireless channel modeling in complex environments is crucial for wireless communication system design and deployment. Traditional channel modeling approaches face challenges in balancing accuracy, efficiency, and scalability, while recent neural approaches such as neural radiance field (NeRF) suffer from long training and slow inference. To tackle these challenges, we propose voxelized radiance field (VoxelRF), a novel neural representation for wireless channel modeling that enables fast and accurate synthesis of spatial spectra. VoxelRF replaces the costly multilayer perception (MLP) used in NeRF-based methods with trilinear interpolation of voxel grid-based representation, and two shallow MLPs to model both propagation and transmitter-dependent effects. To further accelerate training and improve generalization, we introduce progressive learning, empty space skipping, and an additional background entropy loss function. Experimental results demonstrate that VoxelRF achieves competitive accuracy with significantly reduced computation and limited training data, making it more practical for real-time and resource-constrained wireless applications.