In this letter, the use of intelligent reflecting surface (IRS) to enhance the physical layer security of downlink wireless communication is investigated. Assuming a single-antenna legitimate user and a multi-antenna eavesdropper, we propose an effective algorithm to jointly optimize the active and passive beamforming. In the proposed algorithm, the optimal transmit beamforming vector at the BS under fixed IRS phase shifts is derived, and a low-complexity algorithm based on fractional programming (FP) and manifold optimization (MO) is proposed to obtain near optimal IRS phase shifts. Simulation results demonstrate that the proposed algorithm can almost achieve the performance upper bound with a fast convergence rate.