With the advent of wearable Internet of Things (IoT) devices, remote patient monitoring (RPM) emerged as a promising solution for managing heart failure. However, the heart rate can fluctuate significantly due to various factors, and without correlating it to the patient's actual physical activity, it becomes difficult to assess whether changes are significant. Although Artificial Intelligence (AI) models may enhance the accuracy and contextual understanding of remote heart rate monitoring, the integration of activity data is still rarely addressed. In this paper, we propose a Transformer model combined with a Laplace diffusion technique to model heart rate fluctuations driven by physical activity of the patient. Unlike prior models that treat activity as secondary, our approach conditions the entire modeling process on activity context using specialized embeddings and attention mechanisms to prioritize activity specific historical patents. The model captures both long-term patterns and activity-specific heart rate dynamics by incorporating contextualized embeddings and dedicated encoder. The Transformer model was validated on a real-world dataset collected from 29 patients over a 4-month period. Experimental results show that our model outperforms current state-of-the-art methods, achieving a 43% reduction in mean absolute error compared to the considered baseline models. Moreover, the coefficient of determination R2 is 0.97 indicating the model predicted heart rate is in strong agreement with actual heart rate values. These findings suggest that the proposed model is a practical and effective tool for supporting both healthcare providers and remote patient monitoring systems.