This work is a continuation of the manuscript "the structure of extreme level sets in branching Brownian motion", in which the same authors studied the fine structure of the extreme level sets of branching Brownian motion, namely the sets of particles whose height is within a finite distance from the global maximum. It is well known that such particles congregate at large times in clusters of order-one genealogical diameter around local maxima which form a Cox process in the limit. Our main finding here is that most of the particles in an extreme level set come from only a small fraction of the clusters, which are atypically large.