The Fields Medal, often referred as the Nobel Prize of mathematics, is awarded to no more than four mathematician under the age of 40, every four years. In recent years, its conferral has come under scrutiny of math historians, for rewarding the existing elite rather than its original goal of elevating mathematicians from under-represented communities. Prior studies of elitism focus on citational practices and sub-fields; the structural forces that prevent equitable access remain unclear. Here we show the flow of elite mathematicians between countries and lingo-ethnic identity, using network analysis and natural language processing on 240,000 mathematicians and their advisor-advisee relationships. We found that the Fields Medal helped integrate Japan after WWII, through analysis of the elite circle formed around Fields Medalists. Arabic, African, and East Asian identities remain under-represented at the elite level. Through analysis of inflow and outflow, we rebuts the myth that minority communities create their own barriers to entry. Our results demonstrate concerted efforts by international academic committees, such as prize-giving, are a powerful force to give equal access. We anticipate our methodology of academic genealogical analysis can serve as a useful diagnostic for equality within academic fields.