The rapid evolution of the internet of vehicles demands ultra-reliable low-latency communication in high-mobility environments, where conventional beam prediction methods suffer from high-dimensional inputs, prolonged training times, and limited interpretability. To address these challenges, the propagation environment semantics-aware wireless environment knowledge beam prediction (PES-WEKBP) framework is proposed. PES-WEKBP pioneers a novel electromagnetic (EM)-grounded knowledge distillation method, transforming raw visual data into an ultra-lean, interpretable material and location-related wireless environment knowledge matrix. This matrix explicitly encodes critical propagation environment semantics, which is material EM properties and spatial relationships through a physics-informed parameterization process, distilling the environment and channel interplay into a minimal yet information-dense representation. A lightweight decision network then leverages this highly compressed knowledge for low-complexity beam prediction. To holistically evaluate the performance of PES-WEKBP, we first design the prediction consistency-efficiency index (PCEI), which combines prediction accuracy with a stability-penalized logarithmic training time to ensure a balanced optimization of reliability and computational efficiency. Experiments validate that PES-WEKBP achieves a 99.75% to 99.96% dimension reduction and improves accuracy by 5.52% to 8.19%, which outperforms state-of-the-art methods in PCEI scores across diverse vehicular scenarios.