Approximation of elliptic PDEs with random diffusion coefficients typically requires a representation of the diffusion field in terms of a sequence $y=(y_j)_{j\geq 1}$ of scalar random variables. One may then apply high-dimensional approximation methods to the solution map $y\mapsto u(y)$. Although Karhunen-Loève representations are commonly used, it was recently shown, in the relevant case of lognormal diffusion fields, that they do not generally yield optimal approximation rates. Motivated by these results, we construct wavelet-type representations of stationary Gaussian random fields defined on bounded domains. The size and localization properties of these wavelets are studied, and used to obtain polynomial approximation results for the related elliptic PDE which outperform those achievable when using Karhunen-Loève representations. Our construction is based on a periodic extension of the random field, and the expansion on the domain is then obtained by simple restriction. This makes the approach easily applicable even when the computational domain of the PDE has a complicated geometry. In particular, we apply this construction to the class of Gaussian processes defined by the family of Matérn covariances.