The ongoing need for effective epidemic modeling has driven advancements in capturing the complex dynamics of infectious diseases. Traditional models, such as Susceptible-Infected-Recovered, and graph-based approaches often fail to account for higher-order interactions and the nuanced structure pattern inherent in human contact networks. This study introduces a novel Human Contact-Tracing Hypergraph Neural Network framework tailored for epidemic modeling called EpiDHGNN, leveraging the capabilities of hypergraphs to model intricate, higher-order relationships from both location and individual level. Both real-world and synthetic epidemic data are used to train and evaluate the model. Results demonstrate that EpiDHGNN consistently outperforms baseline models across various epidemic modeling tasks, such as source detection and forecast, by effectively capturing the higher-order interactions and preserving the complex structure of human interactions. This work underscores the potential of representing human contact data as hypergraphs and employing hypergraph-based methods to improve epidemic modeling, providing reliable insights for public health decision-making.