Although for a number of semilinear stochastic wave equations existence and uniqueness results for corresponding solution processes are known from the literature, these solution processes are typically not explicitly known and numerical approximation methods are needed in order for mathematical modelling with stochastic wave equations to become relevant for real world applications. This, in turn, requires the numerical analysis of convergence rates for such numerical approximation processes. A recent article by the authors proves upper bounds for weak errors for spatial spectral Galerkin approximations of a class of semilinear stochastic wave equations. The findings there are complemented by the main result of this work, that provides lower bounds for weak errors which show that in the general framework considered the established upper bounds can essentially not be improved.