In [8] (Nakagawa, et.al., IEEE Trans. IT, 2021), we investigated the convergence speed of the Arimoto-Blahut algorithm. In [8], the convergence of the order $O(1/N)$ was analyzed by focusing on the second-order nonlinear recurrence formula consisting of the first- and second-order terms of the Taylor expansion of the defining function of the Arimoto-Blahut algorithm. However, in [8], an infinite number of inequalities were assumed as a "conjecture," and proofs were given based on the conjecture. In this paper, we report a proof of the convergence of the order $O(1/N)$ for a class of channel matrices without assuming the conjecture. The correctness of the proof will be confirmed by several numerical examples.