We use the tropical trigonal construction to calculate the second moment of the tropical Prym variety of all double covers $π:\widetildeΓ\to Γ$ of tropical curves of genus $g(Γ)\leq 4$. The answer is expressed in terms of the signed graphic matroid of the double cover and consists a polynomial and piecewise-polynomial term. We relate the latter term, which does not occur in the analogous formula for the tropical Jacobian, to the problem of extending the Prym--Torelli map from the moduli space of admissible double covers to the second Voronoi compactification of the moduli space of principally polarized abelian varieties.