The consumer Internet of Things (IoT) space has experienced a significant rise in popularity in the recent years. From smart speakers, to baby monitors, and smart kettles and TVs, these devices are increasingly found in households around the world while users may be unaware of the risks associated with owning these devices. Previous work showed that these devices can threaten individuals' privacy and security by exposing information online to a large number of service providers and third party analytics services. Our analysis shows that many of these Internet connections (and the information they expose) are neither critical, nor even essential to the operation of these devices. However, automatically separating out critical from non-critical network traffic for an IoT device is nontrivial, and requires expert analysis based on manual experimentation in a controlled setting. In this paper, we investigate whether it is possible to automatically classify network traffic destinations as either critical (essential for devices to function properly) or not, hence allowing the home gateway to act as a selective firewall to block undesired, non-critical destinations. Our initial results demonstrate that some IoT devices contact destinations that are not critical to their operation, and there is no impact on device functionality if these destinations are blocked. We take the first steps towards designing and evaluating IoTrimmer, a framework for automated testing and analysis of various destinations contacted by devices, and selectively blocking the ones that do not impact device functionality.