Distributed optical fiber vibration sensing (DVS) systems offer a promising solution for large-scale monitoring and intrusion event recognition. However, their practical deployment remains hindered by two major challenges: degradation of recognition accuracy in dynamic conditions, and the computational bottleneck of real-time processing for mass sensing data. This paper presents a new solution to these challenges, through a FPGA-accelerated extreme lightweight model along with a newly proposed knowledge distillation framework. The proposed three-layer depthwise separable convolution network contains only 4141 parameters, which is the most compact architecture in this field to date, and achieves a maximum processing speed of 0.019 ms for each sample covering a 12.5 m fiber length over 0.256 s. This performance corresponds to real-time processing capabilities for sensing fibers extending up to 168.68 km. To improve generalizability under changing environments, the proposed cross-domain distillation framework guided by physical priors is used here to embed frequency-domain insights into the time-domain model. This allows for time-frequency representation learning without increasing complexity and boosts recognition accuracy from 51.93% to 95.72% under unseen environmental conditions. The proposed methodology provides key advancements including a framework combining interpretable signal processing technique with deep learning and a reference architecture for real-time processing and edge-computing in DVS systems, and more general distributed optical fiber sensing (DOFS) area. It mitigates the trade-off between sensing range and real-time capability, bridging the gap between theoretical capabilities and practical deployment requirements. Furthermore, this work reveals a new direction for building more efficient, robust and explainable artificial intelligence systems for DOFS technologies.