Physical layer secret key generation exploits the reciprocal channel randomness for key generation and has proven to be an effective addition security layer in wireless communications. However, static or scarcely random channels require artificially induced dynamics to improve the secrecy performance, e.g., using intelligent reflecting surface (IRS). One key challenge is that the induced random phase from IRS is also reflected in the direction to eavesdroppers (Eve). This leakage enables Eve nodes to estimate the legitimate channels and secret key via a globally known pilot sequence. To mitigate the secret key leakage issue, we propose to exploit random matrix theory to inform the design of a new physical layer secret key generation (PL-SKG) algorithm. We prove that, when sending appropriate random Gaussian matrices, the singular values of Alice's and Bob's received signals follow a similar probability distribution. Leveraging these common singular values, we propose a random Gaussian matrix based PL-SKG (RGM PL-SKG), which avoids the usages of the globally known pilot and thereby prevents the aforementioned leakage issue. Our results show the following: (i) high noise resistance which leads to superior secret key rate (SKR) improvement (up to 300%) in low SNR regime, and (ii) general improved SKR performance against multiple colluded Eves. We believe our combination of random matrix theory and PL-SKG shows a new paradigm to secure the wireless communication channels.