Steel wire ropes (SWRs) are critical load-bearing components in industrial applications, yet their structural integrity is often compromised by local flaws (LFs). Magnetic Flux Leakage (MFL) is a widely used non-destructive testing method that detects defects by measuring perturbations in magnetic fields. Traditional MFL detection methods suffer from critical limitations: one-dimensional approaches fail to capture spatial relationships across sensor channels, while multi-dimensional image-based techniques introduce interpolation artifacts and computational inefficiencies. This paper proposes a novel detection framework based on signal matrices, directly processing raw multi-channel MFL signals using a specialized Convolutional Neural Network for signal matrix as input (SM-CNN). The architecture incorporates stripe pooling to preserve channel-wise features and symmetric padding to improve boundary defect detection. Our model achieves state-of-the-art performance with 98.74% accuracy and 97.85% recall. Additionally, it demonstrates exceptional computational efficiency, processing at 87.72 frames per second (FPS) with a low inference latency of 2.6ms and preprocessing time of 8.8ms. With only 1.48 million parameters, this lightweight design supports real-time processing, establishing a new benchmark for SWR inspection in industrial settings.