Nearly all modern software suffers from bloat that negatively impacts its performance and security. To combat this problem, several automated techniques have been proposed to debloat software. A key metric used in many of these works to demonstrate improved security is code reuse gadget count reduction. The use of this metric is based on the prevailing idea that reducing the number of gadgets available in a software package reduces its attack surface and makes mounting a gadget-based code reuse exploit such as return-oriented programming (ROP) more difficult for an attacker. In this paper, we challenge this idea and show through a variety of realistic debloating scenarios the flaws inherent to the gadget count reduction metric. Specifically, we demonstrate that software debloating can achieve high gadget count reduction rates, yet fail to limit an attacker's ability to construct an exploit. Worse yet, in some scenarios high gadget count reduction rates conceal instances in which software debloating makes security worse by introducing new, useful gadgets. To address these issues, we propose a set of four new metrics for measuring security improvements realized through software debloating that are quality-oriented rather than quantity-oriented. We show that these metrics can identify when debloating negatively impacts security and be efficiently calculated using our static binary analysis tool, the Gadget Set Analyzer. Finally, we demonstrate the utility of these metrics in two realistic case studies: iterative debloating and debloater evaluation.