We consider stochastic differential games with a large number of players, with the aim of quantifying the gap between closed-loop, open-loop and distributed equilibria. We show that, under two different semi-monotonicity conditions, the equilibrium trajectories are close when the interactions between the players are weak. Our approach is non-asymptotic in nature, in the sense that it does not make use of any a priori identification of a limiting model, like in mean field game (MFG) theory. The main technical step is to derive bounds on solutions to systems of PDE/FBSDE characterizing the equilibria that are independent of the number of players. When specialized to the mean field setting, our estimates yield quantitative convergence results for both open-loop and closed-loop equilibria without any use of the master equation. In fact, our main bounds hold for games in which interactions are much sparser than those of MFGs, and so we can also obtain some "universality" results for MFGs, in which we show that games governed by dense enough networks converge to the usual MFG limit. Finally, we use our estimates to study a joint vanishing viscosity and large population limit in the setting of displacement monotone games without idiosyncratic noise.