Crude oil, a critical component of the global economy, has its prices influenced by various factors such as economic trends, political events, and natural disasters. Traditional prediction methods based on historical data have their limits in forecasting, but recent advancements in natural language processing bring new possibilities for event-based analysis. In particular, Language Models (LM) and their advancement, the Generative Pre-trained Transformer (GPT), have shown potential in classifying vast amounts of natural language. However, these LMs often have difficulty with domain-specific terminology, limiting their effectiveness in the crude oil sector. Addressing this gap, we introduce CrudeBERT, a fine-tuned LM specifically for the crude oil market. The results indicate that CrudeBERT's sentiment scores align more closely with the WTI Futures curve and significantly enhance price predictions, underscoring the crucial role of integrating economic principles into LMs.