A set $P\subset \mathbb{F}_p^n\times\mathbb{F}_p^n$ is called $\textit{bilinear}$ when it is the zero set of a family of linear and bilinear forms, and $\textit{transverse}$ when it is stable under vertical and horizontal sums. A theorem of the first author provides a generalization of Bogolyubov's theorem to the bilinear setting. Roughly speaking, it implies that any dense transverse set $P\subset \mathbb{F}_p^n\times\mathbb{F}_p^n$ contains a large bilinear set. In this paper, we elucidate the extent to which a transverse set is forced to be (and not only contain) a bilinear set.