This paper concerns the Vertex Reinforced Jump Process (VRJP) and its representations as a Markov process in random environment. We show that all possible representations of the VRJP as a mixture of Markov processes can be expressed in a similar form, using a random potential and harmonic functions for an associated operator. This allows to show that the VRJP on $\mathbb{Z}^d$ (with certain initial conditions) has a unique representation, by proving that an associated Martin boundary is trivial. Moreover, on infinite trees, we construct a family of representations, that are all different when the VRJP is transient and the tree is $d$-regular (with $d\geq 3$).