We propose two new conformity scores for conformal prediction, in a general multivariate regression framework. The underlying score functions are based on a covariance analysis of the residuals and the input points. We give theoretical guarantees on the prediction sets, which consist in explicit ellipsoids. We study the asymptotic properties of the ellipsoids, and show that their volume is reduced compared to that of classic balls, under ellipticity assumptions. Finally, we illustrate the effectiveness of all our results on an in-depth numerical study, including heavy-tailed as well as non-elliptical distributions.