Robustly tracking a person of interest in the crowd with a robotic platform is one of the cornerstones of human-robot interaction. The robot platform which is limited by the computational power, rapid movements, and occlusions of the target requires an efficient and robust framework to perform tracking. This paper proposes a deep learning framework for tracking a person using a mobile robot with a stereo camera. The proposed system detects a person based on its head, then utilizes the low-cost, high-speed regression network-based tracker to track the person of interest in real-time. The visual servoing of the mobile robot has been designed using a PID controller which utilizes tracker output and depth estimation of the person in subsequent frames, hence providing smooth and adaptive movement of the robot based on target movement. The proposed system has been tested in a real environment, thus proving its effectiveness.