In decentralized multi-robot navigation, ensuring safe and efficient movement with limited environmental awareness remains a challenge. While robots traditionally navigate based on local observations, this approach falters in complex environments. A possible solution is to enhance understanding of the world through inter-agent communication, but mere information broadcasting falls short in efficiency. In this work, we address this problem by simultaneously learning decentralized multi-robot collision avoidance and selective inter-agent communication. We use a multi-head self-attention mechanism that encodes observable information from neighboring robots into a concise and fixed-length observation vector, thereby handling varying numbers of neighbors. Our method focuses on improving navigation performance through selective communication. We cast the communication selection as a link prediction problem, where the network determines the necessity of establishing a communication link with a specific neighbor based on the observable state information. The communicated information enhances the neighbor's observation and aids in selecting an appropriate navigation plan. By training the network end-to-end, we concurrently learn the optimal weights for the observation encoder, communication selection, and navigation components. We showcase the benefits of our approach by achieving safe and efficient navigation among multiple robots, even in dense and challenging environments. Comparative evaluations against various learning-based and model-based baselines demonstrate our superior navigation performance, resulting in an impressive improvement of up to 24% in success rate within complex evaluation scenarios.