We consider the problem of uplink channel estimation for millimeter wave (mmWave) systems, where the base station (BS) and mobile stations (MSs) are equipped with large antenna arrays to provide sufficient beamforming gain for outdoor wireless communications. Hybrid analog and digital beamforming structures are employed by both the BS and the MS due to hardware constraints. We propose a layered pilot transmission scheme and a CANDECOMP/PARAFAC (CP) decomposition-based method for joint estimation of the channels from multiple users (i.e. MSs) to the BS. The proposed method exploits the sparse scattering nature of the mmWave channel and the intrinsic multi-dimensional structure of the multiway data collected from multiple modes. The uniqueness of the CP decomposition is studied and sufficient conditions for essential uniqueness are obtained. The conditions shed light on the design of the beamforming matrix, the combining matrix and the pilot sequences, and meanwhile provide general guidelines for choosing system parameters. Our analysis reveals that our proposed method can achieve a substantial training overhead reduction by employing the layered pilot transmission scheme. Simulation results show that the proposed method presents a clear advantage over a compressed sensing-based method in terms of both estimation accuracy and computational complexity.