The pseudo-Gaussian portmanteau tests of Chitturi, Hosking, and Li and McLeod for VARMA models are revisited from a Le Cam perspective, providing a precise and more rigorous description of the asymptotic behavior of the multivariate portmanteau test statistic, which depends on the dimension $d$ of the observations, the number $m$ of lags involved, and the length $n$ of the observation period. Then, based on the concepts of center-outward ranks and signs recently developed (Hallin, del Barrio, Cuesta-Albertos, and Matr\' an, {\it Annals of Statistics} 49, 1139--1165, 2021), a class of multivariate rank- and sign-based portmanteau test statistics is proposed which, under the null hypothesis and under a broad family of innovation densities, can be approximated by an asymptotically chi-square variable. The asymptotic properties of these tests are derived; simulations demonstrate their advantages over their classical pseudo-Gaussian counterpart.