Inverse problems can be described as limited-data problems in which the signal of interest cannot be observed directly. A physics-based forward model that relates the signal with the observations is typically needed. Unfortunately, unknown model parameters and imperfect forward models can undermine the signal recovery. Even though supervised machine learning offers promising avenues to improve the robustness of the solutions, we have to rely on model-based learning when there is no access to ground truth for the training. Here, we studied a linear inverse problem that included an unknown non-linear model parameter and utilized a Bayesian model-based learning approach that allowed signal recovery and subsequently estimation of the model parameter. This approach, called Bayesian Approximation Error approach, employed a simplified model of the physics of the problem augmented with an approximation error term that compensated for the simplification. An error subspace was spanned with the help of the eigenvectors of the approximation error covariance matrix which allowed, alongside the primary signal, simultaneous estimation of the induced error. The estimated error and signal were then used to determine the unknown model parameter. For the model parameter estimation, we tested different approaches: a conditional Gaussian regression, an iterative (model-based) optimization, and a Gaussian process that was modeled with the help of physics-informed learning. In addition, alternating optimization was used as a reference method. As an example application, we focused on the problem of reconstructing brain activity from EEG recordings under the condition that the electrical conductivity of the patient's skull was unknown in the model. Our results demonstrated clear improvements in EEG source localization accuracy and provided feasible estimates for the unknown model parameter, skull conductivity.