In this paper, we consider the dynamic power control for delay-aware D2D communications. The stochastic optimization problem is formulated as an infinite horizon average cost Markov decision process. To deal with the curse of dimensionality, we utilize the interference filtering property of the CSMA-like MAC protocol and derive a closed-form approximate priority function and the associated error bound using perturbation analysis. Based on the closed-form approximate priority function, we propose a low-complexity power control algorithm solving the per-stage optimization problem. The proposed solution is further shown to be asymptotically optimal for a sufficiently large carrier sensing distance. Finally, the proposed power control scheme is compared with various baselines through simulations, and it is shown that significant performance gain can be achieved.