In the coded caching, the server uses the cached information at the users to serve multiple users in parallel with a single coded multi-casting message or packet, that is, a merged packet, and thus mitigates the peak network congestion. In order to deliver the timely messages to the users in the deadline-driven applications like the video streaming, we must determine online the messages to be merged for the delivery, as there is a time limit for each request. It is important to note that while the merging aids the current coded multi-casting packet, it could harm the future deliveries. Our solution employs the deep reinforcement learning to view the coded multi-casting delivery as a masked action-discrete state control problem, and our policy network, trained via the proximal policy optimization, performs better than SACM++. On the uniform-demand benchmark, our policy network reduces the broadcast-packet expiration ratio $ρ$ by $40.9\%$ ($0.208$ vs.\ $0.352$) with respect to the best coded multi-casting baseline (SACM++), while also attaining the best broadcast-efficiency score $σ$ across the Track~A battery among the coded multi-casting methods. One noteworthy phenomenon here is that, for the applications with stricter deadlines, the merging becomes selective instead of aggressive, since the policy network selectively merges at approximately $31.8\%$ of the chances, even though the same observation holds across the variations within the same simulator family. The focus of our design is on the efficient pairwise XOR merging, where the higher-order ($K{\ge}3$) coding can be considered as a natural generalization left for future work.