In this paper we propose and study local linear and polynomial based estimators for implementing Approximate Bayesian Computation (ABC) style indirect inference and GMM estimators. This method makes use of nonparametric regression in the computation of GMM and Indirect Inference models. We provide formal conditions under which frequentist inference is asymptotically valid and demonstrate the validity of the estimated posterior quantiles for confidence interval construction. We also show that in this setting, local linear kernel regression methods have theoretical advantages over local constant kernel methods that are also reflected in finite sample simulation results. Our results also apply to both exactly and over identified models. These estimators do not need to rely on numerical optimization or Markov Chain Monte Carlo (MCMC) simulations. They provide an effective complement to the classical M-estimators and to MCMC methods, and can be applied to both likelihood based models and method of moment based models.