The Tactile Internet requires ultra-low latency and high-fidelity haptic feedback to enable immersive teleoperation. A key challenge is to ensure ultra-reliable and low-latency transmission of haptic packets under channel variations and potential network outages. To address these issues, one approach relies on local estimation of haptic feedback at the operator side. However, designing an accurate estimator that can faithfully reproduce the true haptic forces remains a significant challenge. In this paper, we propose a novel deep learning architecture, xHAP, based on cross-modal attention to estimate haptic feedback. xHAP fuses information from two distinct data streams: the teleoperator's historical force feedback and the operator's control action sequence. We employ modality-specific encoders to learn temporal representations, followed by a cross-attention layer where the teleoperator haptic data attend to the operator input. This fusion allows the model to selectively focus on the most relevant operator sensory data when predicting the teleoperator's haptic feedback. The proposed architecture reduces the mean-squared error by more than two orders of magnitude compared to existing methods and lowers the SNR requirement for reliable transmission by $10~\mathrm{dB}$ at an error threshold of $0.1$ in a 3GPP UMa scenario. Additionally, it increases coverage by $138\%$ and supports $59.6\%$ more haptic users even under 10 dB lower SNR compared to the baseline.