We introduce a non-Markovian model for electricity markets where the spot price of electricity is driven by several Gaussian Volterra processes, which can be e.g., fractional Brownian motions (fBms), Riemann-Liouville processes or Gaussian-Volterra driven Ornstein-Uhlenbeck processes. Since in energy markets the spot price is not a tradeable asset, due to the limited storage possibilities, forward contracts are considered as traded products. We ensure necessary and sufficient conditions for the absence of arbitrage that, in this kind of market, reflects the fact that the prices of the forward contracts are (Gaussian) martingales under a risk-neutral measure. Moreover, we characterize the market completeness in terms of the number of forward contracts simultaneously considered and of the kernels of the Gaussian-Volterra processes. We also provide a novel representation of Ornstein-Uhlenbeck (OU) processes driven by Gaussian Volterra processes. Also exploiting this result, we show analytically that, for some kinds of Gaussian-Volterra processes driving the spot prices, under conditions ensuring the absence of arbitrage, the market is complete. Finally, we formulate a portfolio optimization problem for an agent who invests in an electricity market, and we solve it explicitly in the case of CRRA utility functions. We also find closed formulas for the price of options written on forward contracts, together with the hedging strategy.