Unmanned aerial vehicle (UAV) system is vulnerable to the control signal spoofing attack due to the openness of the wireless communications. In this correspondence, a physical layer approach is proposed to combat the control signal spoofing attack, i.e,. to determine whether the received control signal packet is from the ground control station (GCS) or a potential malicious attacker (MA), which does not need to share any secret key. We consider the worst case where the UAV does not have any prior knowledge about the MA. Utilizing the channel feature of the angles of arrival, the distance-based path loss, and the Rician-$κ$ factor, we construct a generalized log-likelihood radio (GLLR) test framework to handle the problem. Accurate approximations of the false alarm and successful detection rate are provided to efficiently evaluate the performance.