Haptic feedback is essential to acquire immersive experience when interacting in virtual or augmented reality. Although the existing promising magnetic levitation (maglev) haptic system has advantages of none mechanical friction, its performance is limited by its navigation method, which mainly results from the challenge that it is difficult to obtain high precision, high frame rate and good stability with lightweight design at the same. In this study, we propose to perform the visual-inertial fusion navigation based on sequence-to-sequence learning for the maglev haptic interaction. Cascade LSTM based-increment learning method is first presented to progressively learn the increments of the target variables. Then, two cascade LSTM networks are separately trained for accomplishing the visual-inertial fusion navigation in a loosely-coupled mode. Additionally, we set up a maglev haptic platform as the system testbed. Experimental results show that the proposed cascade LSTM based-increment learning method can achieve high-precision prediction, and our cascade LSTM based visual-inertial fusion navigation method can reach 200Hz while maintaining high-precision (the mean absolute error of the position and orientation is respectively less than 1mm and 0.02°)navigation for the maglev haptic interaction application.