Linear second order elliptic boundary value problems (BVP) on bounded Lipschitz domains are studied in the case of Gaussian white noise loads. Especially, Neumann and Robin BVPs are considered. The main obstacle for applying the usual variational approach is that the Gaussian white noise has irregular realizations. In particular, the corresponding Neumann boundary values are not well-defined in the ordinary sense. In this work, the BVP is reformulated by replacing the continuity of the boundary trace mapping with measurability. Instead of using variational methods alone, the reformulation of the BVP derives also from Cameron-Martin space techniques. The reformulation essentially returns the study of irregular white noise loads to study of $L^2$-loads. Admissibility of the reformulation is demonstrated by showing that usual finite element approximations of the BVP with discretized white noise loads converge to the solution of the reformulated problem. For Neumann and Robin BVPs, the finite dimensional approximations have been utilized before. However, here also the infinite-dimensional limit is considered.