We show that the non-Archimedean skeleton of the Prym variety associated to an unramified double cover of an algebraic curve is naturally isomorphic (as a principally polarized tropical abelian variety) to the tropical Prym variety of the associated tropical double cover. This confirms a conjecture by Jensen and the first author. We prove a new upper bound on the dimension of the Prym-Brill-Noether locus for generic unramified double covers of curves with fixed even gonality on the base. Our methods also give a new proof of the classical Prym-Brill-Noether Theorem for generic unramified double covers that is originally due to Welters and Bertram.