In this article, we propose a covariance based method for combining partial data sets in the genotype to phenotype spectrum. In particular, an expectation-maximization algorithm that can be used to combine partially overlapping relationship/covariance matrices is introduced. Combining data this way, based on relationship matrices, can be contrasted with a feature imputation based approach. We used several public genomic data sets to explore the accuracy of combining genomic relationship matrices. We have also used the heterogeneous genotype/phenotype data sets in the https://triticeaetoolbox.org/ to illustrate how this new method can be used in genomic prediction, phenomics, and graphical modeling.