Cyberbullying in online social networks has become a critical problem, especially among teenagers who are social networks' prolific users. As a result, researchers have focused on identifying distinguishing features of cyberbullying and developing techniques to automatically detect cyberbullying incidents. While this research has resulted in developing highly accurate classifiers, two key practical issues related to identifying cyberbullying have largely been ignored, namely scalability of cyberbullying detection services and timeliness of raising alerts whenever a cyberbullying incident is suspected. These two issues are the subject of this paper. We propose a multi-stage cyberbullying detection solution that drastically reduces the classification time and the time to raise cyberbullying alerts. The proposed solution is highly scalable, does not sacrifice accuracy for scalability, and is highly responsive in raising alerts. The solution is comprised of three novel components, an initial predictor, a multilevel priority scheduler, and an incremental classification mechanism. We have implemented this solution and utilized data obtained from the Vine online social network to demonstrate the utility of each of these components via a detailed performance evaluation. We show that our complete solution is significantly more scalable and responsive than the current state-of-the-art.