We study the secrecy rate maximization problem in a millimeter wave (mmWave) network, consisting of a base station (BS), multiple intelligent reflecting surfaces (IRSs) (or reconfigurable intelligent surfaces (RISs)), multiple users, and a single eavesdropper. To ensure a fair secrecy rate among all the users, we adopt a max-min fairness criterion which results in a mixed integer problem. We first relax discrete IRSs phase shifts to the continuous ones. To cope with the non-convexity of the relaxed optimization problem, we leverage the penalty method and block coordinate descent approach to divide it into two sub-problems, which are solved by successive convex approximation (SCA). Then, we propose a low-complexity mapping algorithm where feasible IRSs phase shifts are obtained. Mathematical evaluation shows the convergence of sub-problems to a Karush-Kuhn-Tucker (KKT) point of the original ones. Furthermore, the convergence guarantee of the overall proposed algorithm and computational complexity are investigated. Finally, simulation results show our proposed algorithm outweighs the conventional solutions based on the semi-definite programming (SDP) in terms of convergence and secrecy rate, especially in a larger number of IRSs and phase shifts where SDP suffers from rank-one approximation. Maximum ratio transmission (MRT) and IRS-free systems are also considered as other benchmarks.