In this paper, an automatic calibration algorithm is proposed to reduce the depth error caused by internal stray light in amplitude-modulated continuous wave (AMCW) coaxial scanning light detection and ranging (LiDAR). Assuming that the internal stray light generated in the process of emitting laser is static, the amplitude and phase delay of internal stray light are estimated using the Gaussian mixture model (GMM) and particle swarm optimization (PSO). Specifically, the pixel positions in a raw signal amplitude map of calibration checkboard are segmented by GMM with two clusters considering the dark and bright image pattern. The loss function is then defined as L1-norm of difference between mean depths of two amplitude-segmented clusters. To avoid overfitting at a specific distance in PSO process, the calibration check board is actually measured at multiple distances and the average of corresponding L1 loss functions is chosen as the actual loss. Such loss is minimized by PSO to find the two optimal target parameters: the amplitude and phase delay of internal stray light. According to the validation of the proposed algorithm, the original loss is reduced from tens of centimeters to 3.2 mm when the measured distances of the calibration checkboard are between 1 m and 4 m. This accurate calibration performance is also maintained in geometrically complex measured scene. The proposed internal stray light calibration algorithm in this paper can be used for any type of AMCW coaxial scanning LiDAR regardless of its optical characteristics.