Maintaining consistent uncertainty estimates in localization systems is crucial as the perceived uncertainty commonly affects high-level system components, such as control or decision processes. A method for constructing an observability-constrained magnetic field-aided inertial navigation system is proposed to address the issue of erroneous yaw observability, which leads to inconsistent estimates of yaw uncertainty. The proposed method builds upon the previously proposed observability-constrained extended Kalman filter and extends it to work with a magnetic field-based odometry-aided inertial navigation system. The proposed method is evaluated using simulation and real-world data, showing that (i) the system observability properties are preserved, (ii) the estimation accuracy increases, and (iii) the perceived uncertainty calculated by the EKF is more consistent with the true uncertainty of the filter estimates.