Distance-based anomaly detection methods rely on compact in-distribution (ID) embeddings that are well separated from anomalies. However, conventional contrastive learning strategies often struggle to achieve this balance, either promoting excessive variance among inliers or failing to preserve the diversity of outliers. We begin by analyzing the challenges of representation learning for anomaly detection and identify three essential properties for the pretext task: (1) compact clustering of inliers, (2) strong separation between inliers and anomalies, and (3) preservation of diversity among synthetic outliers. Building on this, we propose a structured contrastive objective that redefines positive and negative relationships during training, promoting these properties without requiring explicit anomaly labels. We extend this framework with a patch-based learning and evaluation strategy specifically designed to improve the detection of localized anomalies in industrial settings. Our approach demonstrates significantly faster convergence and improved performance compared to standard contrastive methods. It matches or surpasses anomaly detection methods on both semantic and industrial benchmarks, including methods that rely on discriminative training or explicit anomaly labels.