Millimeter wave is a promising technology for the next generation of wireless systems. As it is well-known for its high path loss, the systems working in this spectrum tend to exploit the shorter wavelength to equip the transceivers with a large number of antennas to overcome the path loss issue. The large number of antennas leads to large channel matrices and consequently a challenging channel estimation problem. The channel estimation algorithms that have been proposed so far either neglect the probability of estimation error or require a high feedback overload from receivers to ensure the target probability of estimation error. In this paper, we propose a multi-stage adaptive channel estimation algorithm called robust adaptive multi-feedback (RAF). The algorithm is based on using the estimated channel coefficient to predict a lower bound for the required number of measurements. Our simulations demonstrate that compared with existing algorithms, RAF can achieve the desired probability of estimation error while on average reducing the feedback overhead by 75.5% and the total channel estimation time by 14%.