Intelligent reflecting surface (IRS) is deemed as a promising solution to improve the spectral and energy efficiency of wireless communications cost-effectively. In this paper, we consider a wireless network where multiple base stations (BSs) serve their respective users with the aid of distributed IRSs in the downlink communication. Specifically, each IRS assists in the transmission from its associated BS to user via passive beamforming, while in the meantime, it also randomly scatters the signals from other co-channel BSs, thus resulting in additional signal as well as interference paths in the network. As such, a new IRS-user/BS association problem arises pertaining to optimally balance the passive beamforming gains from all IRSs among different BS-user communication links. To address this new problem, we first derive a tractable lower bound of the average signal-to-interference-plus-noise ratio (SINR) at the receiver of each user, termed average-signal-to-average-interference-plus-noise ratio (ASAINR), based on which two ASAINR balancing problems are formulated to maximize the minimum ASAINR among all users by optimizing the IRS-user associations without and with BS transmit power control, respectively. We also characterize the scaling behavior of user ASAINRs with the increasing number of IRS reflecting elements to investigate the different effects of IRS-reflected signal versus interference power. Moreover, to solve the two ASAINR balancing problems that are both non-convex optimization problems, we propose an optimal solution to the problem without BS power control and low-complexity suboptimal solutions to both problems by applying the branch-and-bound method and exploiting new properties of the IRS-user associations, respectively. Numerical results verify our performance analysis and also demonstrate significant performance gains of the proposed solutions over benchmark schemes.