A new development of smart-home systems is to use mobile apps to control IoT devices across a Home Area Network (HAN). Those systems tend to rely on the Wi-Fi router to authenticate other devices; as verified in our study, IoT vendors tend to trust all devices connected to the HAN. This treatment exposes them to the attack from malicious apps, particularly those running on authorized phones, which the router does not have information to control, as confirmed in our measurement study. Mitigating this threat cannot solely rely on IoT manufacturers, which may need to change the hardware on the devices to support encryption, increasing the cost of the device, or software developers who we need to trust to implement security correctly. In this work, we present a new technique to control the communication between the IoT devices and their apps in a unified, backward-compatible way. Our approach, called Hanguard, does not require any changes to the IoT devices themselves, the IoT apps or the OS of the participating phones. Hanguard achieves a fine-grained, per-app protection through bridging the OS-level situation awareness and the router-level per-flow control: each phone runs a non-system userspace Monitor app to identify the party that attempts to access the protected IoT device and inform the router through a control plane of its access decision; the router enforces the decision on the data plane after verifying whether the phone should be allowed to talk to the device. Hanguard uses a role-based access control (RBAC) schema which leverages type enforcement (TE) and multi-category security (MCS) primitives to define highly flexible access control rules. We implemented our design over both Android and iOS (>95% of mobile OS market share) and a popular router. Our study shows that Hanguard is both efficient and effective in practice.