In multi-robot systems, ensuring safe and reliable decision making under uncertain conditions demands robust multi-robot belief space planning (MR-BSP) algorithms. While planning with multiple robots, each robot maintains a belief over the state of the environment and reasons how the belief would evolve in the future for different possible actions. However, existing MR-BSP works have a common assumption that the beliefs of different robots are same at planning time. Such an assumption is often unrealistic as it requires prohibitively extensive and frequent data sharing capabilities. In practice, robots may have limited communication capabilities, and consequently beliefs of the robots can be different. Crucially, when the robots have inconsistent beliefs, the existing approaches could result in lack of coordination between the robots and may lead to unsafe decisions. In this paper, we present decentralized MR-BSP algorithms, with performance guarantees, for tackling this crucial gap. Our algorithms leverage the notion of action preferences. The base algorithm VerifyAC guarantees a consistent joint action selection by the cooperative robots via a three-step verification. When the verification succeeds, VerifyAC finds a consistent joint action without triggering a communication; otherwise it triggers a communication. We design an extended algorithm R-VerifyAC for further reducing the number of communications, by relaxing the criteria of action consistency. Another extension R-VerifyAC-simp builds on verifying a partial set of observations and improves the computation time significantly. The theoretical performance guarantees are corroborated with simulation results in discrete setting. Furthermore, we formulate our approaches for continuous and high-dimensional state and observation spaces, and provide experimental results for active multi-robot visual SLAM with real robots.