Current methods for regularization in machine learning require quite specific model assumptions (e.g. a kernel shape) that are not derived from prior knowledge about the application, but must be imposed merely to make the method work. We show in this paper that regularization can indeed be achieved by assuming nothing but invariance principles (w.r.t. scaling, translation, and rotation of input and output space) and the degree of differentiability of the true function. Concretely, we derive a novel (non-Gaussian) stochastic process from the above minimal assumptions, and we present a corresponding Bayesian inference method for regression. The mean posterior turns out to be a polyharmonic spline, and the posterior process is a mixture of t-processes. Compared with Gaussian process regression, the proposed method shows equal performance and has the advantages of being (i) less arbitrary (no choice of kernel) (ii) potentially faster (no kernel parameter optimization), and (iii) having better extrapolation behavior. We believe that the proposed theory has central importance for the conceptual foundations of regularization and machine learning and also has great potential to enable practical advances in ML areas beyond regression.