We focus on collaborative edge inference over wireless, which enables multiple devices to cooperate to improve inference performance in the presence of corrupted data. Exploiting a key-query mechanism for selective information exchange (or, group formation for collaboration), we recall the effect of wireless channel impairments in feature communication. We argue and show that a disjoint approach, which only considers either the semantic relevance or channel state between devices, performs poorly, especially in harsh propagation conditions. Based on these findings, we propose a joint approach that takes into account semantic information relevance and channel states when grouping devices for collaboration, by making the general attention weights dependent of the channel information. Numerical simulations show the superiority of the joint approach against local inference on corrupted data, as well as compared to collaborative inference with disjoint decisions that either consider application or physical layer parameters when forming groups.