With the increased popularity of smart phones, there is a greater need to have a robust authentication mechanism that handles various security threats and privacy leakages effectively. This paper studies continuous authentication for touch interface based mobile devices. A Hidden Markov Model (HMM) based behavioral template training approach is presented, which does not require training data from other subjects other than the owner of the mobile device and can get updated with new data over time. The gesture patterns of the user are modeled from multiple sensors - touch, accelerometer and gyroscope data using a continuous left-right HMM. The approach models the tap and stroke patterns of a user since these are the basic and most frequently used interactions on a mobile device. To evaluate the effectiveness of the proposed method a new data set has been created from 42 users who interacted with off-the-shelf applications on their smart phones. Results show that the performance of the proposed approach is promising and potentially better than other state-of-the-art approaches.