Despite the renewed interest in the Newey and Powell (1987) concept of expectiles in fields such as econometrics, risk management, and extreme value theory, expectile regression---or, more generally, M-quantile regression---unfortunately remains limited to single-output problems. To improve on this, we introduce hyperplane-valued multivariate M-quantiles that show strong advantages, for instance in terms of equivariance, over the various point-valued multivariate M-quantiles available in the literature. Like their competitors, our multivariate M-quantiles are directional in nature and provide centrality regions when all directions are considered. These regions define a new statistical depth, the halfspace M-depth, whose deepest point, in the expectile case, is the mean vector. Remarkably, the halfspace M-depth can alternatively be obtained by substituting, in the celebrated Tukey (1975) halfspace depth, M-quantile outlyingness for standard quantile outlyingness, which supports a posteriori the claim that our multivariate M-quantile concept is the natural one. We investigate thoroughly the properties of the proposed multivariate M-quantiles, of halfspace M-depth, and of the corresponding regions. Since our original motivation was to define multiple-output expectile regression methods, we further focus on the expectile case. We show in particular that expectile depth is smoother than the Tukey depth and enjoys interesting monotonicity properties that are extremely promising for computational purposes. Unlike their quantile analogs, the proposed multivariate expectiles also satisfy the coherency axioms of multivariate risk measures. Finally, we show that our multivariate expectiles indeed allow performing multiple-output expectile regression, which is illustrated on simulated and real data.