The orbital angular momentum (OAM) wireless communication technique is emerging as one of potential techniques for the Sixth generation (6G) wireless communication system. The most advantage of OAM wireless communication technique is the natural orthogonality among different OAM states. However, one of the most disadvantages is the crosstalk among different OAM states which is widely caused by the atmospheric turbulence and misalignment between transmitting and receiving antennas. Considering the OAM-based multiple-input multiple-output (OAM-MIMO) transmission system with unaligned antennas, a new channel model is proposed for performance analysis. Moreover, a purity model of the OAM-MIMO transmission system with unaligned antennas is derived for the non-Kolmogorov turbulence. Furthermore, error probability and capacity models are derived for OAM-MIMO transmission systems with unaligned antennas. To overcome the disadvantage caused by unaligned antennas and non-Kolmogorov turbulence, a new optimization algorithm of OAM state interval is proposed to improve the capacity of OAM-MIMO transmission system. Numerical results indicate that the capacity of OAM-MIMO transmission system is improved by the optimization algorithm. Specifically, the capacity increment of OAM-MIMO transmission system adopting the optimization algorithm is up to 28.7% and 320.3% when the angle of deflection between transmitting and receiving antennas is -24 dB and -5 dB, respectively.