High impedance faults (HIFs) in distribution grids may cause wildfires and threaten human lives. Conventional protection relays at substations fail to detect more than 10\% HIFs since over-currents are low and the signatures of HIFs are local. With more $μ$PMU being installed in the distribution system, high-resolution $μ$PMU datasets provide the opportunity of detecting HIFs from multiple points. Still, the main obstacle in applying the $μ$PMU datasets is the lack of labels. To address this issue, we construct a physics-informed convolutional auto-encoder (PICAE) to detect HIFs without labeled HIFs for training. The significance of our PICAE is a physical regularization, derived from the elliptical trajectory of voltages-current characteristics, to distinguish HIFs from other abnormal events even in highly noisy situations. We formulate a system-wide detection framework that merges multiple nodes' local detection results to improve the detection accuracy and reliability. The proposed approaches are validated in the IEEE 34-node test feeder simulated through PSCAD/EMTDC. Our PICAE outperforms the existing works in various scenarios and is robust to different observability and noise.