We associate a matroid $M(\widetildeΓ/Γ)$ to a harmonic double cover $π:\widetildeΓ\to Γ$ of metric graphs. The matroid $M(\widetildeΓ/Γ)$ is a geometric interpretation of Zaslavsky's signed graphic matroid. We show that the principalization $\mathrm{Prym}_p(\widetildeΓ/Γ)$ of the tropical Prym variety of the double cover can be reconstructed from $M(\widetildeΓ/Γ)$, equipped with certain additional decorations. We describe the simplification of the matroid $M(\widetildeΓ/Γ)$ and show that the Prym variety does not change under simplification.