This article identifies and addresses a fundamental bottleneck in data-driven 360-degree image quality assessment (IQA): the lack of intelligent, sample-level data selection. Hence, we propose a novel framework that introduces a critical refinement step between patches sampling and model training. The core of our contribution is an embedding similarity-based selection algorithm that distills an initial, potentially redundant set of patches into a compact, maximally informative subset. This is formulated as a regularized optimization problem that preserves intrinsic perceptual relationships in a low-dimensional space, using residual analysis to explicitly filter out irrelevant or redundant samples. Extensive experiments on three benchmark datasets (CVIQ, OIQA, MVAQD) demonstrate that our selection enables a baseline model to match or exceed the performance of using all sampled data while keeping only 40-50% of patches. Particularly, we demonstrate the universal applicability of our approach by integrating it with several state-of-the-art IQA models, incleasy to deploy. Most significantly, its value as a generic,uding CNN- and transformer-based architectures, consistently enabling them to maintain or improve performance with 20-40\% reduced computational load. This work establishes that adaptive, post-sampling data refinement is a powerful and widely applicable strategy for achieving efficient and robust 360-degree IQA.