High-entropy alloys have attracted attention for their exceptional mechanical properties and thermal stability. However, the combinatorial explosion in the number of possible elemental compositions renders traditional trial-and-error experimental approaches highly inefficient for materials discovery. To solve this problem, machine learning techniques have been increasingly employed for property prediction and high-throughput screening. Nevertheless, highly accurate nonlinear models often suffer from a lack of interpretability, which is a major limitation. In this study, we focus on local data structures that emerge from the greedy search behavior inherent to experimental data acquisition. By introducing a linear and low-dimensional mixture regression model, we strike a balance between predictive performance and model interpretability. In addition, we develop an algorithm that simultaneously performs prediction and feature selection by considering multiple candidate descriptors. Through a case study on high-entropy alloys, this study introduces a method that combines anchor-guided clustering and sparse linear modeling to address biased data structures arising from greedy exploration in materials science.