Implantable medical devices (IMDs), such as pacemakers, implanted cardiac defibrillators, and neurostimulators are medical devices implanted into patients' bodies for monitoring physiological signals and performing medical treatments. Many IMDs have built-in wireless communication modules to facilitate data collecting and device reprogramming by external programmers. The wireless communication brings significant conveniences for advanced applications such as real-time and remote monitoring but also introduces the risk of unauthorized wireless access. The absence of effective access control mechanisms exposes patients' life to cyber attacks. In this paper, we present a lightweight and universally applicable access control system for IMDs. By leveraging Physically Obfuscated Keys (POKs) as the hardware root of trust, provable security is achieved based on standard cryptographic primitives while attaining high energy efficiency. In addition, barrier-free IMD access under emergent situations is realized by utilizing the patient's biometrical information. We evaluate our proposed scheme through extensive security analysis and a prototype implementation, which demonstrates our work's superiority on security and energy efficiency.