Autonomous Vehicle has been transforming intelligent transportation systems. As telecommunication technology improves, autonomous vehicles are getting connected to each other and to infrastructures, forming Connected and Autonomous Vehicles (CAVs). CAVs will help humans achieve safe, efficient, and autonomous transportation systems. However, CAVs will face significant security challenges because many of their components are vulnerable to attacks, and a successful attack on a CAV may have significant impacts on other CAVs and infrastructures due to their communications. In this paper, we conduct a survey on 184 papers from 2000 to 2020 to understand state-of-the-art CAV attacks and defense techniques. This survey first presents a comprehensive overview of security attacks and their corresponding countermeasures on CAVs. We then discuss the details of attack models based on the targeted CAV components of attacks, access requirements, and attack motives. Finally, we identify some current research challenges and trends from the perspectives of both academic research and industrial development. Based on our studies of academic literature and industrial publications, we have not found any strong connection between academic research and industry's implementation on CAV-related security issues. While efforts from CAV manufacturers to secure CAVs have been reported, there is no evidence to show that CAVs on the market have the ability to defend against some novel attack models that the research community has recently found. This survey may give researchers and engineers a better understanding of the current status and trend of CAV security for CAV future improvement.