In this paper, we propose a robust design framework for IRS-aided communication systems in the presence of user location uncertainty. By jointly designing the transmit beamforming vector at the BS and phase shifts at the IRS, we aim to minimize the transmit power subject to the worse-case quality of service (QoS) constraint, i.e., ensuring the user rate is above a threshold for all possible user location error realizations. With unit-modulus, this problem is not convex. The location uncertainty in the QoS constraint further increases the difficulty of solving this problem. By utilizing techniques of Taylor expansion, S-Procedure and semidefinite relaxation (SDP), we transform this problem into a sequence of semidefinite programming (SDP) sub-problems. Simulation results show that the proposed robust algorithm substantially outperforms the non-robust algorithm proposed in the literature, in terms of probability of reaching the required QoS target.