Underwater visual-inertial navigation is challenging due to the poor visibility and presence of outliers in underwater environments. The navigation performance is closely related to outlier detection and elimination. Existing methods assume the inertial odometry is accurate enough for outlier detection, which is not valid for low-cost inertial applications. We propose a novel iterative smoothing and outlier detection method aiming for underwater navigation. Using the dataset collected from an underwater robot and fiducial markers, experimental results confirm that the method can successfully eliminate the outliers and enhance navigation accuracy.