We investigate the expressive power of $\mathsf{MATLANG}$, a formal language for matrix manipulation based on common matrix operations and linear algebra. The language can be extended with the operation $\mathsf{inv}$ of inverting a matrix. In $\mathsf{MATLANG}+\mathsf{inv}$ we can compute the transitive closure of directed graphs, whereas we show that this is not possible without inversion. Indeed we show that the basic language can be simulated in the relational algebra with arithmetic operations, grouping, and summation. We also consider an operation $\mathsf{eigen}$ for diagonalizing a matrix, which is defined so that different eigenvectors returned for a same eigenvalue are orthogonal. We show that $\mathsf{inv}$ can be expressed in $\mathsf{MATLANG}+\mathsf{eigen}$. We put forward the open question whether there are boolean queries about matrices, or generic queries about graphs, expressible in $\mathsf{MATLANG} + \mathsf{eigen}$ but not in $\mathsf{MATLANG}+\mathsf{inv}$. The evaluation problem for $\mathsf{MATLANG} + \mathsf{eigen}$ is shown to be complete for the complexity class $\exists \mathbf{R}$.