This paper deals with the Bayesian estimation of high dimensional Gaussian graphical models. We develop a quasi-Bayesian implementation of the neighborhood selection method of Meinshausen and Buhlmann (2006) for the estimation of Gaussian graphical models. The method produces a product-form quasi-posterior distribution that can be efficiently explored by parallel computing. We derive a non-asymptotic bound on the contraction rate of the quasi-posterior distribution. The result shows that the proposed quasi-posterior distribution contracts towards the true precision matrix at a rate given by the worst contraction rate of the linear regressions that are involved in the neighborhood selection. We develop a Markov Chain Monte Carlo algorithm for approximate computations, following an approach from Atchade (2015). We illustrate the methodology with a simulation study. The results show that the proposed method can fit Gaussian graphical models at a scale unmatched by other Bayesian methods for graphical models.