We analyse the effect of agent-dependent heavy-tailed waiting times in the voter model on the complete graph with $N$ vertices. We derive a novel scaling limit and show the existence of a limiting infinite voter model on the slowest updating sites. We further derive the consensus probabilities in the limit model explicitly. In the mean-field setting, the limit is determined by the extreme-value landscape of the waiting times and depends only on the tail index. To obtain these results, we study the coalescing system of random walks that is dual to the limit voter model and prove, among other auxiliary results, that it comes down from infinity.