Reconfigurable intelligent surface (RIS) is a revolutionary technology for sixth-generation (6G) networks owing to its ability to manipulate wireless environments. As a frequency-selective device, RIS can only effectively shape the propagation of signals within a certain frequency band. Due to this frequency-selective property, the deployment of RIS in cellular networks will introduce a complicated base station (BS)-RIS-user association issue since adjacent BSs operate at different frequency bands. In this paper, with the consideration of the frequency-selective characteristics of RIS, we aim to jointly optimize BS-RIS-user association, active beamforming at BSs, and passive beamforming of RIS to maximize the sum-rate of a RIS-assisted cellular network. We first leverage $l_0$-norm to efficiently integrate BS-RIS-user association with active and passive beamforming. Then, we adopt fractional programming (FP) and block coordinate descent (BCD) methods to deal with logarithmic and fractional parts and decouple the joint association and beamforming design problem into several sub-problems. Efficient algorithms which combine $l_0$-norm approximation, majorization-minimization (MM), and alternating direction method of multipliers (ADMM) are developed to alternately solve the sub-problems. Extensive simulation results illustrate the importance of BS-RIS-user association optimization in RIS-assisted cellular networks and verify the effectiveness of the proposed joint association and beamforming design algorithm.