Differential phase contrast (DPC) imaging plays an important role in the family of quantitative phase measurement. However, the reconstruction algorithm for quantitative DPC (qDPC) imaging is not yet optimized, as it does not incorporate the inborn properties of qDPC imaging. In this research, we propose a simple but effective image prior, the dark-field sparse prior (DSP), to facilitate the phase reconstruction quality for all DPC-based phase reconstruction algorithms. The DSP is based on the key observation that most pixel values for an idea differential phase contrast image are zeros since the subtraction of two images under anti-symmetric illumination cancels all background components. With this DSP prior, we formed a new cost function in which L0-norm was used to represent the DSP. Further, we developed two different algorithms based on (1) the Half Quadratic Splitting, and (2) the Richardson-Lucy deconvolution to solve this NP-hard L0-norm problem. We tested our new model on both simulated and experimental data and compare against state-of-the-art methods including L2-norm and total variation regularizations. Results show that our proposed model is superior in terms of phase reconstruction quality and implementation efficiency, in which it significantly increases the experimental robustness, while maintaining the data fidelity.