Smart home IoT systems and devices are susceptible to attacks and malfunctions. As a result, users' concerns about their security and safety issues arise along with the prevalence of smart home deployments. In a smart home, various anomalies (such as fire or flooding) could happen, due to cyber attacks, device malfunctions, or human mistakes. These concerns motivate researchers to propose various anomaly detection approaches. Existing works on smart home anomaly detection focus on checking the sequence of IoT devices' events but leave out the temporal information of events. This limitation prevents them to detect anomalies that cause delay rather than missing/injecting events. To fill this gap, in this paper, we propose a novel anomaly detection method that takes the inter-event intervals into consideration. We propose an innovative metric to quantify the temporal similarity between two event sequences. We design a mechanism to learn the temporal patterns of event sequences of common daily activities. Delay-caused anomalies are detected by comparing the sequence with the learned patterns. We collect device events from a real-world testbed for training and testing. The experiment results show that our proposed method achieves accuracies of 93%, 88%, 89% for three daily activities.