Accurate estimation in time-varying inverse problems under limited and sparse observations remains a fundamental challenge across scientific domains. For example, snow depth estimation requires inferring hidden parameters governing sea ice physics, which can be incorporated through physics-informed encoding. To address this challenge, we introduce Physics-Encoded Inversion (PhysE-Inv), a novel framework that combines deep sequential learning with physics-informed inference for solving inverse problems under real-world sparse observational settings. PhysE-Inv integrates an LSTM encoder-decoder to capture temporal dependencies, together with contrastive learning regularization that enforces noise-invariant latent representations. The framework learns latent parameters that, when combined with observational inputs, reconstruct snow depth while incorporating physics-informed guidance. PhysE-Inv consistently outperforms all evaluated baselines, achieving an average MSE reduction of 24.7\% across all baseline models and a 17.3\% improvement over the strongest baseline under parameter estimation settings. Overall, our work demonstrates a generalizable inverse modeling paradigm for data-scarce domains where physics-informed guidance can be incorporated into sparse observations.