Contact tracing has been considered as an effective measure to limit the transmission of infectious disease such as COVID-19. Trajectory-based contact tracing compares the trajectories of users with the patients, and allows the tracing of both direct contacts and indirect contacts. Although trajectory data is widely considered as sensitive and personal data, there is limited research on how to securely compare trajectories of users and patients to conduct contact tracing with excellent accuracy, high efficiency, and strong privacy guarantee. Traditional Secure Multiparty Computation (MPC) techniques suffer from prohibitive running time, which prevents their adoption in large cities with millions of users. In this work, we propose a technical framework called ContactGuard to achieve accurate, efficient, and privacy-preserving trajectory-based contact tracing. It improves the efficiency of the MPC-based baseline by selecting only a small subset of locations of users to compare against the locations of the patients, with the assist of Geo-Indistinguishability, a differential privacy notion for Location-based services (LBS) systems. Extensive experiments demonstrate that ContactGuard runs up to 2.6$\times$ faster than the MPC baseline, with no sacrifice in terms of the accuracy of contact tracing.