We investigate the probabilistic and analytic properties of Volterra processes constructed as pathwise integrals of deterministic kernels with respect to the Hölder continuous trajectories of Hilbert-valued Gaussian processes. To this end, we extend the Volterra sewing lemma from \cite{HarangTindel} to the two dimensional case, in order to construct two dimensional operator-valued Volterra integrals of Young type. We prove that the covariance operator associated to infinite dimensional Volterra processes can be represented by such a two dimensional integral, which extends the current notion of representation for such covariance operators. We then discuss a series of applications of these results, including the construction of a rough path associated to a Volterra process driven by Gaussian noise with possibly irregular covariance structures, as well as a description of the irregular covariance structure arising from Gaussian processes time-shifted along irregular trajectories. Furthermore, we consider an infinite dimensional fractional Ornstein-Uhlenbeck process driven by Gaussian noise, which can be seen as an extension of the volatility model proposed by Rosenbaum et al. in \cite{ElEuchRosenbaum}.