Location-aware communication systems are expected to play a pivotal part in the next generation of mobile communication networks. Therefore, there is a need to understand the localization limits in these networks, particularly, using millimeter-wave technology (mmWave). Towards that, we address the uplink and downlink localization limits in terms of 3D position and orientation error bounds for mmWave multipath channels. We also carry out a detailed analysis of the dependence of the bounds of different systems parameters. Our key findings indicate that the uplink and downlink behave differently in two distinct ways. First of all, the error bounds have different scaling factors with respect to the number of antennas in the uplink and downlink. Secondly, uplink localization is sensitive to the orientation angle of the user equipment (UE), whereas downlink is not. Moreover, in the considered outdoor scenarios, the non-line-of-sight paths generally improve localization when a line-of-sight path exists. Finally, our numerical results show that mmWave systems are capable of localizing a UE with sub-meter position error, and sub-degree orientation error.