We study robust estimators of the mean of a probability measure $P$, called robust empirical mean estimators. This elementary construction is then used to revisit a problem of aggregation and a problem of estimator selection, extending these methods to not necessarily bounded collections of previous estimators. We consider then the problem of robust $M$-estimation. We propose a slightly more complicated construction to handle this problem and, as examples of applications, we apply our general approach to least-squares density estimation, to density estimation with Küllback loss and to a non-Gaussian, unbounded, random design and heteroscedastic regression problem. Finally, we show that our strategy can be used when the data are only assumed to be mixing.