With the rapid growth of new technological paradigms such as the Internet of Things (IoT), it opens new doors for many applications in the modern era for the betterment of human life. One of the recent applications of the IoT is the Internet of Vehicles (IoV) which helps to see unprecedented growth of connected vehicles on the roads. The IoV is gaining attention due to enhancing traffic safety and providing low route information. One of the most important and major requirements of the IoV is preserving security and privacy under strict latency. Moreover, vehicles are required to be authenticated frequently and fast considering limited bandwidth, high mobility, and density of the vehicles. To address the security vulnerabilities and data integrity, an ultralight authentication scheme has been proposed in this article. Physical Unclonable Function (PUF) and XOR function are used to authenticate both server and vehicle in two message flow which makes the proposed scheme ultralight, and less computation is required. The proposed Easy-Sec can authenticate vehicles maintaining low latency and resisting known security threats. Furthermore, the proposed Easy-Sec needs low overhead so that it does not increase the burden of the IoV network. Computational ( around 4 ms) and Communication (32 bytes) overhead shows the feasibility, efficiency, and also security features are depicted using formal analysis, Burrows, Abadi, and Needham (BAN) logic, and informal analysis to show the robustness of the proposed mechanisms against security threats.