The prediction of electromagnetic spectra for MXene-based solar absorbers is a computationally intensive task, traditionally addressed using full-wave solvers. This study introduces an efficient deep learning framework incorporating transfer learning, multi-channel spectral refinement (MCSR), and Savitzky-Golay smoothing to accelerate and enhance spectral prediction accuracy. The proposed architecture leverages a pretrained MobileNetV2 model, fine-tuned to predict 102-point absorption spectra from $64\times64$ metasurface designs. Additionally, the MCSR module processes the feature map through multi-channel convolutions, enhancing feature extraction, while Savitzky-Golay smoothing mitigates high-frequency noise. Experimental evaluations demonstrate that the proposed model significantly outperforms baseline Convolutional Neural Network (CNN) and deformable CNN models, achieving an average root mean squared error (RMSE) of 0.0245, coefficient of determination \( R^2 \) of 0.9578, and peak signal-to-noise ratio (PSNR) of 32.98 dB. The proposed framework presents a scalable and computationally efficient alternative to conventional solvers, positioning it as a viable candidate for rapid spectral prediction in nanophotonic design workflows.