This paper looks into the problem of grasping region localization along with suitable pose from a cluttered environment without any a priori knowledge of the object geometry. This end-to-end method detects the handles from a single frame of input sensor. The pipeline starts with the creation of multiple surface segments to detect the required gap in the first stage, and eventually helps in detecting boundary lines. Our novelty lies in the fact that we have merged color based edge and depth edge in order to get more reliable boundary points through which a pair of boundary line is fitted. Also this information is used to validate the handle by measuring the angle between the boundary lines and also by checking for amy potential occlusion. In addition, we also proposed an optimizing cost function based method to choose the best handle from a set of valid handles. The method proposed is tested on real-life datasets and is found to out form state of the art methods in terms of precision.