Various approaches to stochastic processes exist, noting that key properties such as measurability and continuity are not trivially satisfied. We introduce a new theory for Gaussian processes using improper linear functionals. Using a collection of i.i.d. standard normal variables, we define Gaussian white noise and discuss its properties. This is extended to general Gaussian processes on Hilbert space, where the variance is allowed to be any suitable operator. Our main focus is $L^2$ spaces, and we discuss criteria for Gaussian processes to be continuous in this setting. Finally, we outline a framework for statistical inference using the presented theory with focus on the special case of $L^2[0,1]$. We introduce the Fredholm determinant into the functional log-likelihood. We demonstrate that the naive functional log-likelihood is not consistent with the multivariate likelihood. A correction term is introduced, and we prove an asymptotical result.