In this paper, we focus on the sum-rate optimization in a multi-user millimeter-wave (mmWave) system with distributed intelligent reflecting surfaces (D-IRSs), where a base station (BS) communicates with users via multiple IRSs. The BS transmit beamforming, IRS switch vector, and phase shifts of the IRS are jointly optimized to maximize the sum-rate under minimum user rate, unit-modulus, and transmit power constraints. To solve the resulting non-convex optimization problem, we develop an efficient alternating optimization (AO) algorithm. Specifically, the non-convex problem is converted into three subproblems, which are solved alternatively. The solution to transmit beamforming at the BS and the phase shifts at the IRS are derived by using the successive convex approximation (SCA)-based algorithm, and a greedy algorithm is proposed to design the IRS switch vector. The complexity of the proposed AO algorithm is analyzed theoretically. Numerical results show that the D-IRSs-aided scheme can significantly improve the sum-rate and energy efficiency performance.