We prove that any semi-streaming algorithm for $(1-ε)$-approximation of maximum bipartite matching requires \[ Ω(\frac{\log{(1/ε)}}{\log{(1/β)}}) \] passes, where $β\in (0,1)$ is the largest parameter so that an $n$-vertex graph with $n^β$ edge-disjoint induced matchings of size $Θ(n)$ exist (such graphs are referred to as RS graphs). Currently, it is known that \[ Ω(\frac{1}{\log\log{n}}) \leqslant β\leqslant 1-Θ(\frac{\log^*{n}}{\log{n}}) \] and closing this huge gap between upper and lower bounds has remained a notoriously difficult problem in combinatorics. Under the plausible hypothesis that $β= Ω(1)$, our lower bound result provides the first pass-approximation lower bound for (small) constant approximation of matchings in the semi-streaming model, a longstanding open question in the graph streaming literature. Our techniques are based on analyzing communication protocols for compressing (hidden) permutations. Prior work in this context relied on reducing such problems to Boolean domain and analyzing them via tools like XOR Lemmas and Fourier analysis on Boolean hypercube. In contrast, our main technical contribution is a hardness amplification result for permutations through concatenation in place of prior XOR Lemmas. This result is proven by analyzing permutations directly via simple tools from group representation theory combined with detailed information-theoretic arguments, and can be of independent interest.