This article proposes a method to diminish the pose (position plus attitude) drift experienced by an SVO (Semi-Direct Visual Odometry) based visual navigation system installed onboard a UAV (Unmanned Air Vehicle) by supplementing its pose estimation non linear optimizations with priors based on the outputs of a GNSS (Global Navigation Satellite System) Denied inertial navigation system. The method is inspired in a PI (Proportional Integral) control system, in which the attitude, altitude, and rate of climb inertial outputs act as targets to ensure that the visual estimations do not deviate far from their inertial counterparts. The resulting IA-VNS (Inertially Assisted Visual Navigation System) achieves major reductions in the horizontal position drift inherent to the GNSS-Denied navigation of autonomous fixed wing low SWaP (Size, Weight, and Power) UAVs. Additionally, the IA-VNS can be considered as a virtual incremental position (ground velocity) sensor capable of providing observations to the inertial filter. Stochastic high fidelity Monte Carlo simulations of two representative scenarios involving the loss of GNSS signals are employed to evaluate the results and to analyze their sensitivity to the terrain type overflown by the aircraft as well as to the quality of the onboard sensors on which the priors are based. The author releases the C ++ implementation of both the navigation algorithms and the high fidelity simulation as open-source software.