We prove quantitative bounds for the inverse theorem for Gowers uniformity norms $\mathsf{U}^5$ and $\mathsf{U}^6$ in $\mathbb{F}_2^n$. The proof starts from an earlier partial result of Gowers and the author which reduces the inverse problem to a study of algebraic properties of certain multilinear forms. The bulk of the work in this paper is a study of the relationship between the natural actions of $\operatorname{Sym}_4$ and $\operatorname{Sym}_5$ on the space of multilinear forms and the partition rank, using an algebraic version of regularity method. Along the way, we give a positive answer to a conjecture of Tidor about approximately symmetric multilinear forms in 5 variables, which is known to be false in the case of 4 variables. Finally, we discuss the possible generalization of the argument for $\mathsf{U}^k$ norms.