We study the convergence of an $N$-particle Markovian controlled system to the solution of a family of stochastic McKean-Vlasov control problems, either with a finite horizon or Schrödinger type cost functional. Specifically, under suitable assumptions, we prove the convergence of the value functions, the fixed-time probability distributions, and the relative entropy of their path-space probability laws. These proofs are based on a Benamou-Brenier type reformulation of the problem and a superposition principle, both of which are tools from the theory of optimal transport.