We study the high-temperature regime of a mean-field spin glass model whose couplings matrix is orthogonally invariant in law. The magnetization of this model is conjectured to satisfy a system of TAP equations, originally derived by Parisi and Potters using a diagrammatic expansion of the Gibbs free energy. We prove that this TAP description is correct in an $L^2$ sense, in a regime of sufficiently high temperature. Our approach develops a novel geometric argument for proving the convergence of an Approximate Message Passing (AMP) algorithm to the magnetization vector, which is applicable in models without i.i.d. couplings. This convergence is shown via a conditional second moment analysis of the free energy restricted to a thin band around the output of the AMP algorithm, in a system of many "orthogonal" replicas.