The sixth-generation (6G) mobile network is envisioned to incorporate sensing and edge artificial intelligence (AI) as two key functions. Their natural convergence leads to the emergence of Integrated Sensing and Edge AI (ISEA), a novel paradigm enabling real-time acquisition and understanding of sensory information at the network edge. However, ISEA faces a communication bottleneck due to the large number of sensors and the high dimensionality of sensory features. Traditional approaches to communication-efficient ISEA lack awareness of semantic relevance, i.e., the level of relevance between sensor observations and the downstream task. To fill this gap, this paper presents a novel framework for semantic-relevance-aware sensor selection to achieve optimal end-to-end (E2E) task performance under heterogeneous sensor relevance and channel states. E2E sensing accuracy analysis is provided to characterize the sensing task performance in terms of selected sensors' relevance scores and channel states. Building on the results, the sensor-selection problem for accuracy maximization is formulated as an integer program and solved through a tight approximation of the objective. The optimal solution exhibits a priority-based structure, which ranks sensors based on a priority indicator combining relevance scores and channel states and selects top-ranked sensors. Low-complexity algorithms are then developed to determine the optimal numbers of selected sensors and features. Experimental results on both synthetic and real datasets show substantial accuracy gain achieved by the proposed selection scheme compared to existing benchmarks.