We present a secret key agreement (SKA) protocol for a multi-user time-division duplex system where a base-station (BS) with a large antenna array (LAA) shares secret keys with users in the presence of non-colluding eavesdroppers. In the system, when the BS transmits random sequences to legitimate users for sharing common randomness, the eavesdroppers can attempt the pilot contamination attack (PCA) in which each of eavesdroppers transmits its target user's training sequence in hopes of acquiring possible information leak by steering beam towards the eavesdropper. We show that there exists a crucial complementary relation between the received signal strengths at the eavesdropper and its target user. This relation tells us that the eavesdropper inevitably leaves a trace that enables us to devise a way of measuring the amount of information leakage to the eavesdropper even if PCA parameters are unknown. To this end, we derive an estimator for the channel gain from the BS to the eavesdropper and propose a rate-adaptation scheme for adjusting the length of secret key under the PCA. Extensive analysis and evaluations are carried out under various setups, which show that the proposed scheme adequately takes advantage of the LAA to establish the secret keys under the PCA.