Millimeter wave offers a sensible solution to the capacity crunch faced by 5G wireless communications. This paper comprehensively studies physical layer security in a multi-input single-output (MISO) millimeter wave system where multiple single-antenna eavesdroppers are randomly located. Concerning the specific propagation characteristics of millimeter wave, we investigate two secure transmission schemes, namely maximum ratio transmitting (MRT) beamforming and artificial noise (AN) beamforming. Specifically, we first derive closed-form expressions of the connection probability for both schemes. We then analyze the secrecy outage probability (SOP) in both non-colluding eavesdroppers and colluding eavesdroppers scenarios. Also, we maximize the secrecy throughput under a SOP constraint, and obtain optimal transmission parameters, especially the power allocation between AN and the information signal for AN beamforming. Numerical results are provided to verify our theoretical analysis. We observe that the density of eavesdroppers, the spatially resolvable paths of the destination and eavesdroppers all contribute to the secrecy performance and the parameter design of millimeter wave systems.