Embedded systems are ubiquitous. However, physical access of users and likewise attackers makes them often threatened by fault attacks: a single fault during the computation of a cryptographic primitive can lead to a total loss of system security. This can have serious consequences, e.g., in safetycritical systems, including bodily harm and catastrophic technical failures. However, countermeasures often focus on isolated fault models and high layers of abstraction. This leads to a dangerous sense of security, because exploitable faults that are only visible at machine code level might not be covered by countermeasures. In this work we present ARMORY, a fully automated open source framework for exhaustive fault simulation on binaries of the ubiquitous ARM-M class. It allows engineers and analysts to efficiently scan a binary for potential weaknesses against arbitrary combinations of multi-variate fault injections under a large variety of fault models. Using ARMORY, we demonstrate the power of fully automated fault analysis and the dangerous implications of applying countermeasures without knowledge of physical addresses and offsets. We exemplarily analyze two case studies, which are highly relevant for practice: a DFA on AES (cryptographic) and a secure bootloader (non-cryptographic). Our results show that indeed numerous exploitable faults found by ARMORY which occur in the actual implementations are easily missed in manual inspection. Crucially, most faults are only visible when taking machine code information, i.e., addresses and offsets, into account. Surprisingly, we show that a countermeasure that protects against one type of fault can actually largely increase the vulnerability to other fault models. Our work demonstrates the need for countermeasures that, at least in their evaluation, are not restricted to isolated fault models and consider low-level information [...].