This paper proposes a regularized pairwise difference approach for estimating the linear component coefficient in a partially linear model, with consistency and exact rates of convergence obtained in high dimensions under mild scaling requirements. Our analysis reveals interesting features such as (i) the bandwidth parameter automatically adapts to the model and is actually tuning-insensitive; and (ii) the procedure could even maintain fast rate of convergence for $α$-Hölder class of $α\leq1/2$. Simulation studies show the advantage of the proposed method, and application of our approach to a brain imaging data reveals some biological patterns which fail to be recovered using competing methods.