We establish formulae for the moments of the moments of the characteristic polynomials of random orthogonal and symplectic matrices in terms of certain lattice point count problems. This allows us to establish asymptotic formulae when the matrix-size tends to infinity in terms of the volumes of certain regions involving continuous Gelfand-Tsetlin patterns with constraints. The results we find differ from those in the unitary case considered previously