The factor analysis model is a statistical model where a certain number of hidden random variables, called factors, affect linearly the behaviour of another set of observed random variables, with additional random noise. The main assumption of the model is that the factors and the noise are Gaussian random variables. This implies that the feasible set lies in the cone of positive semidefinite matrices. In this paper, we do not assume that the factors and the noise are Gaussian, hence the higher order moment and cumulant tensors of the observed variables are generally nonzero. This motivates the notion of kth-order factor analysis model, that is the family of all random vectors in a factor analysis model where the factors and the noise have finite and possibly nonzero moment and cumulant tensors up to order k. This subset may be described as the image of a polynomial map onto a Cartesian product of symmetric tensor spaces. Our goal is to compute its dimension and we provide conditions under which the image has positive codimension.