With the rapid development of health equipments, increasingly more patients have installed the implantable medical devices (IMD) in their bodies for diagnostic, monitoring, and therapeutic purposes. IMDs are extremely limited in computation power and battery capacity. Meanwhile, IMDs have to communicate with an external programmer device (i.e., IMD programmer) through the wireless channel, which put them under the risk of unauthorized access and malicious wireless attacks. In this paper, we propose a proxy-based fine-grained access control scheme for IMDs, which can prolong the IMD's lifetime by delegating the access control computations to the proxy device (e.g., smartphone). In our scheme, the proxy communicates with the IMD programmer through an audio cable, which is resistant to a number of wireless attacks. Additionally, we use the ciphertext-policy attribute-based encryption (CP-ABE) to enforce fine-grained access control. The proposed scheme is implemented on real emulator devices and evaluated through experimental tests. The experiments show that the proposed scheme is lightweight and effective.