In this article, we establish precise convergence rates of a general class of $N$-Player Stackelberg games to their mean field limits, which allows the response time delay of information, empirical distribution based interactions, and the control-dependent diffusion coefficients. All these features makes our problem nonstandard, barely been touched in the literature, and they complicate the analysis and therefore reduce the convergence rate. We first justify the same convergence rate for both the followers and the leader. Specifically, for the most general case, the convergence rate is shown to be $\mathcal{O}\left(N^{-\frac{2(q-2)}{n_1(3q-4)}}\right)$ when $n_1>4$ where $n_1$ is the dimension of the follower's state, and $q$ is the order of the integration of the initial; and this rate has yet been shown in the literature, to the best of our knowledge. Moreover, by classifying cases according to the state dimension $n_1$, the nature of the delay, and the assumptions of the coefficients, we provide several subcases where faster convergence rates can be obtained; for instance the $\mathcal{O}\left(N^{-\frac{2}{3n_1}}\right)$-convergence when the diffusion coefficients are independent of control variable. Our result extends the standard $o(1)$-convergence result for the mean field Stackelberg games in the literature, together with the $\mathcal{O}(N^{-\frac{1}{n_1+4}})$-convergence for the mean field games with major and minor players. We also discuss the special case where our coefficients are linear in distribution argument while nonlinear in state and control arguments, and we establish an $\mathcal{O}(1/\sqrt{N})$ convergence rate, which extends the linear quadratic cases in the literature.