Port scanning refers to the systematic exploration of networked computing systems. The goal of port scanning is to identify active services and associated information. Although this technique is often employed by malicious actors to locate vulnerable systems within a network, port scanning is also a legitimate method employed by IT professionals to troubleshoot network issues and maintain system security. In the case of the latter, cybersecurity practitioners use port scanning catalog exposed systems, identify potential misconfigurations, or test controls that may be running on a system. Existing literature has thoroughly established a taxonomy for port scanning. The taxonomy maps the types of scans as well as techniques. In fact, there are several tools mentioned repeatedly in the literature. Those are Nmap, Zmap, and masscan. Further, the presence of multiple tools signals that how a port scanner interacts with target systems impacts the output of the tool. In other words, the various tools may not behave identically or produce identical output. Yet, no work has been done to quantify the efficacy for these popular tools in a uniform, rigorous manner. Accordingly, we used a comparative experimental protocol to measure the accuracy, false positive, false negative, and efficiency of Nmap, Zmap, and masscan. The results show no difference between port scanners in general performance. However, the results revealed a statistically significant difference in efficiency. This information can be used to guide the selection of port scanning tools based on specific needs and requirements. As well, for researchers, the outcomes may also suggest areas for future work in the development novel port scanning tools.