Millimeter wave (mmWave) communication by utilizing lens antenna arrays is a promising technique for realizing cost-effective 5G wireless systems with large MIMO (multiple-input multiple-output) but only limited radio frequency (RF) chains. This paper studies an uplink multi-user mmWave single-sided lens MIMO system, where only the base station (BS) is equipped with a full-dimensional (FD) lens antenna array with both elevation and azimuth angle resolution capabilities, and each mobile station (MS) employs the conventional uniform planar array (UPA) without the lens. By exploiting the angle-dependent energy focusing property of the lens antenna array at the BS as well as the multi-path sparsity of mmWave channels, we propose a low-complexity path division multiple access (PDMA) scheme, which enables virtually interference-free multi-user communications when the angle of arrivals (AoAs) of all MS multi-path signals are sufficiently separable at the BS. To this end, a new technique called path delay compensation is proposed at the BS to effectively transform the multi-user frequency-selective MIMO channels to parallel frequency-flat small-size MIMO channels for different MSs, for each of which the low-complexity single-carrier(SC) transmission is applied. For general scenarios with insufficient AoA separations, analog beamforming at the MSs and digital combining at the BS are jointly designed to maximize the achievable sum-rate of the MSs based on their effective MIMO channels resulting from path delay compensation. In addition, we propose a new and efficient channel estimation scheme tailored for PDMA, which requires negligible training overhead in practical mmWave systems and yet leads to comparable performance as that based on perfect channel state information (CSI).