Rapid, reliable, and accurate interpretation of medical time-series signals is crucial for high-stakes clinical decision-making. Deep learning methods offered unprecedented performance in medical signal processing but at a cost: they were compute-intensive and lacked interpretability. We propose Sparse Mixture of Learned Kernels (SMoLK), an interpretable architecture for medical time series processing. SMoLK learns a set of lightweight flexible kernels that form a single-layer sparse neural network, providing not only interpretability, but also efficiency, robustness, and generalization to unseen data distributions. We introduce a parameter reduction techniques to reduce the size of SMoLK's networks while maintaining performance. We test SMoLK on two important tasks common to many consumer wearables: photoplethysmography (PPG) artifact detection and atrial fibrillation detection from single-lead electrocardiograms (ECGs). We find that SMoLK matches the performance of models orders of magnitude larger. It is particularly suited for real-time applications using low-power devices, and its interpretability benefits high-stakes situations.