Low-cost message passing (MP) algorithm has been recognized as a promising technique for sparse vector recovery. However, the existing MP algorithms either focus on mean square error (MSE) of the value recovery while ignoring the sparsity requirement, or support error rate (SER) of the sparse support (non-zero position) recovery while ignoring its value. A novel low-complexity Bernoulli-Gaussian MP (BGMP) is proposed to perform the value recovery as well as the support recovery. Particularly, in the proposed BGMP, support-related Bernoulli messages and value-related Gaussian messages are jointly processed and assist each other. In addition, a strict lower bound is developed for the MSE of BGMP via the genie-aided minimum mean-square-error (GA-MMSE) method. The GA-MMSE lower bound is shown to be tight in high signal-to-noise ratio. Numerical results are provided to verify the advantage of BGMP in terms of final MSE, SER and convergence speed.