The use of social media as a means of communication has significantly increased over recent years. There is a plethora of information flow over the different topics of discussion, which is widespread across different domains. The ease of information sharing has increased noisy data being induced along with the relevant data stream. Finding such relevant data is important, especially when we are dealing with a time-critical domain like disasters. It is also more important to filter the relevant data in a real-time setting to timely process and leverage the information for decision support. However, the short text and sometimes ungrammatical nature of social media data challenge the extraction of contextual information cues, which could help differentiate relevant vs. non-relevant information. This paper presents a novel method to classify relevant social media posts during disaster events by ensembling the features of both domain-specific word embeddings as well as more generic domain-agnostic word embeddings. Therefore, we develop and evaluate a hybrid feature engineering framework for integrating diverse semantic representations using a combination of word embeddings to efficiently classify a relevant social media post. The application of the proposed classification framework could help in filtering public posts at large scale, given the growing usage of social media posts in recent years.