\noindent Consider an infinite collection of particles on the real line moving according to independent Brownian motions and such that the $i$-th particle from the left gets the drift $g_{i-1}$. The case where $g_0=1$ and $g_{i}=0$ for all $i \in \mathbb{N}$ corresponds to the well studied infinite Atlas model. Under conditions on the drift vector $\boldsymbol{g} = (g_0, g_1, \ldots)'$ it is known that the Markov process corresponding to the gap sequence of the associated ranked particles has a continuum of product form stationary distributions $\{π_a^{\boldsymbol{g}}, a \in S^{\boldsymbol{g}}\}$ where $S^{\boldsymbol{g}}$ is a semi-infinite interval of the real line. In this work we show that all of these stationary distributions are extremal and ergodic. We also prove that any product form stationary distribution of this Markov process that satisfies a mild integrability condition must be $π_a^{\boldsymbol{g}}$ for some $a \in S^{\boldsymbol{g}}$. These results are new even for the infinite Atlas model. The work makes progress on the open problem of characterizing all the invariant distributions of general competing Brownian particle systems interacting through their relative ranks. Proofs rely on synchronous and mirror coupling of Brownian particles and properties of the intersection local times of the various particles in the infinite system.