Timely and accurate detection of defects and contaminants in solar panels is critical for maintaining the efficiency and reliability of photovoltaic (PV) systems. While recent studies have applied deep learning to PV inspection, fair benchmarking across detector architectures and unbiased handling of class imbalance remain limited. This work presents a comprehensive benchmark of convolutional and transformer-based object detectors on UAV-captured RGB imagery of solar panels. It introduces a class-targeted augmentation strategy applied exclusively to the training split to mitigate imbalance without compromising evaluation integrity. Faster R-CNN with ResNet50 and MobileNetV3 backbones, RetinaNet with ResNet50, YOLOv5, YOLOv8, and Swin Transformer backbones integrated with Faster R-CNN (Tiny, Small, and Base variants) are evaluated. Performance is assessed using mean Average Precision (mAP) across multiple IoU thresholds, precision, recall, F1 score, and inference throughput to enable accuracy-throughput tradeoff analysis relevant to UAV deployment. Experimental results show that Faster R-CNN with a ResNet50 backbone achieves the highest localization accuracy, with mAP@0.5 of 0.893 and mAP@0.5:0.95 of 0.759, whereas the MobileNetV3 variant provides the best overall reliability balance, achieving recall of 0.745, F1-score of 0.809, and accuracy of 0.679 on the test set. The dataset and code will be released upon acceptance of the paper.