Low-power wide-area networks (LPWANs) are becoming an integral part of the Internet of Things. As a consequence, businesses, administration, and, subsequently, society itself depend on the reliability and availability of these communication networks. Released in 2015, LoRaWAN gained popularity and attracted the focus of security research, revealing a number of vulnerabilities. This lead to the revised LoRaWAN 1.1 specification in late 2017. Most of previous work focused on simulation and theoretical approaches. Interoperability and the variety of implementations complicate the risk assessment for a specific LoRaWAN network. In this paper, we address these issues by introducing ChirpOTLE, a LoRa and LoRaWAN security evaluation framework suitable for rapid iteration and testing of attacks in testbeds and assessing the security of real-world networks.We demonstrate the potential of our framework by verifying the applicability of a novel denial-of-service attack targeting the adaptive data rate mechanism in a testbed using common off-the-shelf hardware. Furthermore, we show the feasibility of the Class B beacon spoofing attack, which has not been demonstrated in practice before.