Currently, the improvement of LiDAR poses estimation accuracy is an urgent need for mobile robots. Research indicates that diverse LiDAR points have different influences on the accuracy of pose estimation. This study aimed to select a good point set to enhance accuracy. Accordingly, the sensitivity and uncertainty of LiDAR point residuals were formulated as a fundamental basis for derivation and analysis. High-sensitivity and low -uncertainty point residual terms are preferred to achieve higher pose estimation accuracy. The proposed selection method has been theoretically proven to be capable of achieving a global statistical optimum. It was tested on artificial data and compared with the KITTI benchmark. It was also implemented in LiDAR odometry (LO) and LiDAR inertial odometry (LIO), both indoors and outdoors. The experiments revealed that utilizing selected LiDAR point residuals simultaneously enhances optimization accuracy, decreases residual terms, and guarantees real-time performance.