Localization in high-level Autonomous Driving (AD) systems is highly security critical. While the popular Multi-Sensor Fusion (MSF) based design can be more robust against single-source sensor spoofing attacks, it is found recently that state-of-the-art MSF algorithms is vulnerable to GPS spoofing alone due to practical factors, which can cause various road hazards such as driving off road or onto the wrong way. In this work, we perform the first systematic exploration of the novel usage of lane detection (LD) to defend against such attacks. We first systematically analyze the potentials of such a domain-specific defense opportunity, and then design a novel LD-based defense approach, $LD^3$, that aims at not only detecting such attacks effectively in the real time, but also safely stopping the victim in the ego lane upon detection considering the absence of onboard human drivers. We evaluate $LD^3$ on real-world sensor traces and find that it can achieve effective and timely detection against existing attack with 100% true positive rates and 0% false positive rates. Results also show that $LD^3$ is robust to diverse environmental conditions and is effective at steering the AD vehicle to safely stop within the current traffic lane. We implement $LD^3$ on two open-source high-level AD systems, Baidu Apollo and Autoware, and validate its defense capability in both simulation and the physical world in end-to-end driving. We further conduct adaptive attack evaluations and find that $LD^3$ is effective at bounding the deviations from reaching the attack goals in stealthy attacks and is robust to latest LD-side attack.