This paper aims to lay the foundations for statistics over local fields, such as the field of $p$-adic numbers. Over such fields, we give characterizations for maximum likelihood estimation and conditional independence for multivariate Gaussian distributions. We also give a bijection between the tropicalization of such Gaussian measures in dimension 2 and supermodular functions on the 2-dimensional discrete cube. Finally, we introduce the Bruhat-Tits building as a parameter space for Gaussian distributions and discuss their connections to conditional independence statements as an open problem.