We study the Gorenstein property for phylogenetic group-based models. We prove that for the groups $\mathbb Z_3$ and $\mathbb Z_2\times \mathbb Z_2$ and trivalent trees the associated polytopes are always Gorenstein extending the results of Buczyńska and Wiśniewski for the group $\mathbb Z_2$.