In indirect measurements, the measurand is determined by solving an inverse problem which requires a model of the measurement process. Such models are often approximations and introduce systematic errors leading to a bias of the posterior distribution in Bayesian inversion. We propose a unified framework that combines transport maps from a reference distribution to the posterior distribution with the model error approach. This leads to an adaptive algorithm that jointly estimates the posterior distribution of the measurand and the model error. The efficiency and accuracy of the method are demonstrated on two model problems, showing that the approach effectively corrects biases while enabling fast sampling.