In this letter, we investigate the anti-jamming defense problem in multi-user scenarios, where the coordination among users is taken into consideration. The Markov game framework is employed to model and analyze the anti-jamming defense problem, and a collaborative multi-agent anti-jamming algorithm (CMAA) is proposed to obtain the optimal anti-jamming strategy. In sweep jamming scenarios, on the one hand, the proposed CMAA can tackle the external malicious jamming. On the other hand, it can effectively cope with the mutual interference among users. Simulation results show that the proposed CMAA is superior to both sensing based method and independent Q-learning method, and has the highest normalized rate.