With the widespread adoption of information systems, recommender systems are widely used for better user experience. Collaborative filtering is a popular approach in implementing recommender systems. Yet, collaborative filtering methods are highly dependent on user feedback, which is often highly sparse and hard to obtain. However, such issues could be alleviated if knowledge from a much denser and a related secondary domain could be used to enhance the recommendation accuracy in the sparse target domain. In this publication, we propose a deep learning method for cross-domain recommender systems through the linking of cross-domain user latent representations as a form of knowledge transfer across domains. We assume that cross-domain similarities of user tastes and behaviors are clearly observable in the low dimensional user latent representations. These user similarities are used to link the domains. As a result, we propose a Variational Autoencoder based network model for cross-domain linking with added contextualization to handle sparse data and for better transfer of cross-domain knowledge. We further extend the model to be more suitable in cold start scenarios and to utilize auxiliary user information for additional gains in recommendation accuracy. The effectiveness of the proposed model was empirically evaluated using multiple datasets. The experiments proved that the proposed model outperforms the state of the art techniques.