The starting point of this work is a theorem due to Maxwell characterizing the distribution of a Gaussian vector with at least two coordinates. We define the Gaussian orthogonal, unitary and symplectic tensor ensembles for notions of real symmetric, hermitian and self-dual hermitian tensors which recover the classical vector and matrix Gaussian ensembles when the order is one and two. We give a complete family of invariant polynomials for orthogonal, unitary and symplectic transformations and we prove a Maxwell-type theorem for these Gaussian tensor distributions unifying and extending the ones known for vectors and matrices.