Multi-matrix invariants, and in particular the scalar multi-trace operators of $\mathcal{N}=4$ SYM with $U(N)$ gauge symmetry, can be described using permutation centraliser algebras (PCA), which are generalisations of the symmetric group algebras and independent of $N$. Free-field two-point functions define an $N$-dependent inner product on the PCA, and bases of operators have been constructed which are orthogonal at finite $N$. Two such bases are well-known, the restricted Schur and covariant bases, and both definitions involve representation-theoretic quantities such as Young diagram labels, multiplicity labels, branching and Clebsch-Gordan coefficients for symmetric groups. The explicit computation of these coefficients grows rapidly in complexity as the operator length increases. We develop a new method for explicitly constructing all the operators with specified Young diagram labels, based on an $N$-independent integer eigensystem formulated in the PCA. The eigensystem construction naturally leads to orthogonal basis elements which are integer linear combinations of the multi-trace operators, and the $N$-dependence of their norms are simple known dimension factors. We provide examples and give computer codes in SageMath which efficiently implement the construction for operators of classical dimension up to 14. While the restricted Schur basis relies on the Artin-Wedderburn decomposition of symmetric group algebras, the covariant basis relies on a variant which we refer to as the Kronecker decomposition. Analogous decompositions exist for any finite group algebra and the eigenvalue construction of integer orthogonal bases extends to the group algebra of any finite group with rational characters.