Millimeter wave (mmWave) based multiple-input multiple-output (MIMO) capable user-centric (UC) ultra-dense (UD) networks are suggested to facilitate high throughput requirements of future networks. Due to the high blockage susceptibility of mmWave, the connections may drop frequently. Hence efficient and fast beam management in initial access (IA) is essential. Current cellular systems use beam sweeping based IA mechanisms. UC UD concept requires all of its access points (APs) to perform IA. This leads to a shortage of orthogonal radio resources. Nonorthogonal resource allocation causes interference which leads to a higher misdetection probability. In this paper, we propose a novel deep contextual bandit (DCB) based approach to perform fast and efficient IA in mmWave based UC UD networks. The DCB model uses one reference signal from the user to predict the IA beam. The reduced use of reference signals improves beam discovery delay and relaxes the requirement for radio resources. Ray-tracing and stochastic channel model-based simulations show that the suggested system outperforms its beam sweeping counterpart in terms of probability of beam misdetection and beam discovery delay in mmWave based UC UD networks.