Machine Learning (ML) has been applied to enable many life-assisting appli-cations, such as abnormality detection and emdergency request for the soli-tary elderly. However, in most cases machine learning algorithms depend on the layout of the target Internet of Things (IoT) sensor network. Hence, to deploy an application across Heterogeneous Sensor Networks (HSNs), i.e. sensor networks with different sensors type or layouts, it is required to repeat the process of data collection and ML algorithm training. In this paper, we introduce a novel framework leveraging deep learning for graphs to enable using the same activity recognition system across HSNs deployed in differ-ent smart homes. Using our framework, we were able to transfer activity classifiers trained with activity labels on a source HSN to a target HSN, reaching about 75% of the baseline accuracy on the target HSN without us-ing target activity labels. Moreover, our model can quickly adapt to unseen sensor layouts, which makes it highly suitable for the gradual deployment of real-world ML-based applications. In addition, we show that our framework is resilient to suboptimal graph representations of HSNs.