The concept of Internet of Things (IoT) has become more popular in the modern era of technology than ever before. From small household devices to large industrial machines, the vision of IoT has made it possible to connect the devices with the physical world around them. This increasing popularity has also made the IoT devices and applications in the center of attention among attackers. Already, several types of malicious activities exist that attempt to compromise the security and privacy of the IoT devices. One interesting emerging threat vector is the attacks that abuse the use of sensors on IoT devices. IoT devices are vulnerable to sensor-based threats due to the lack of proper security measurements available to control use of sensors by apps. By exploiting the sensors (e.g., accelerometer, gyroscope, microphone, light sensor, etc.) on an IoT device, attackers can extract information from the device, transfer malware to a device, or trigger a malicious activity to compromise the device. In this survey, we explore various threats targeting IoT devices and discuss how their sensors can be abused for malicious purposes. Specifically, we present a detailed survey about existing sensor-based threats to IoT devices and countermeasures that are developed specifically to secure the sensors of IoT devices. Furthermore, we discuss security and privacy issues of IoT devices in the context of sensor-based threats and conclude with future research directions.