Complex dynamical systems, from macromolecules to ecosystems, are often modeled by stochastic differential equations. To learn such models from data, a common approach involves sparse selection among a large function library. However, we show that overfitting arises not just from individual model complexity, but also from the combinatorial growth of possible models. To address this, we introduce Parsimonious Stochastic Inference (PASTIS), a principled method combining likelihood-estimation statistics with extreme value theory to suppress superfluous parameters. PASTIS outperforms existing methods and reliably identifies minimal models, even with low sampling rates or measurement error. It extends to stochastic partial differential equations, and applies to ecological networks and reaction-diffusion dynamics.