The estimation of high-dimensional physical parameters constrained by partial differential equations (PDEs) from limited and indirect measurements is a highly ill-posed problem. Traditional methods face significant accuracy and efficiency bottlenecks, particularly when observations are sparse, irregularly sampled, and constrained by real-world sensor placement. We propose the Physical Inversion Solver (PIS), a unified framework that couples Set-Conditioned Flow Matching with a Cosine-Annealed Sparsity Curriculum (CASC) to enable stable inversion from arbitrary, off-grid sensors even under minimal guidance. By leveraging straight-path transport, PIS achieves instantaneous inference (50 NFEs), offering orders-of-magnitude speedup over iterative baselines. Extensive experiments demonstrate that PIS reduces error by up to 88.7% under extreme sparsity (<1%) across subsurface characterization, wave-based characterization, and structural health monitoring, while providing robust uncertainty quantification for optimal sensor placement.