This paper considers distributed M-estimation under heterogeneous distributions among distributed data blocks. A weighted distributed estimator is proposed to improve the efficiency of the standard "Split-And-Conquer" (SaC) estimator for the common parameter shared by all the data blocks. The weighted distributed estimator is shown to be at least as efficient as the would-be full sample and the generalized method of moment estimators with the latter two estimators requiring full data access. A bias reduction is formulated to the WD estimator to accommodate much larger numbers of data blocks than the existing methods without sacrificing the estimation efficiency, and a similar debiased operation is made to the SaC estimator. The mean squared error (MSE) bounds and the asymptotic distributions of the WD and the two debiased estimators are derived, which shows advantageous performance of the debiased estimators when the number of data blocks is large.