8 hours → minutes | Processing time for a 300-million-record data job after moving to Databricks

1/2 day | To complete a student segmentation analysis that previously took two analysts two weeks

The Office for Students regulates more than 400 higher education providers across England and manages data spanning millions of student records over decades. As the scale and complexity of analysis grew, legacy systems could no longer keep pace. By moving to Databricks, the organisation transformed how its teams access, analyse and act on data, dramatically accelerating insight generation while creating a more flexible foundation for AI-driven decision support.

When the tools couldn't keep up with the work

The Office for Students is focused on ensuring a high quality of higher education for all students across England through data-informed regulation that supports the quality, fairness and accountability of the higher education system. The team examines student and provider data, including student outcomes, provider reporting, enrollment patterns, student continuation data and indicators that may signal risks to education quality or student experience across higher education providers.

However, the limitations of a legacy analytics platform had become impossible to work around. Their data team managed data on every student who had touched higher education in England, up to 3 million records per year, drawn from the JISC, Department for Education, Universities and Colleges Admissions Service (UCAS), the Student Loans Company and other sources spanning 15 to 20 years. The system had originally been designed for analysis of quantitative data, but the demands on the organisation had evolved far beyond what the legacy platform could support efficiently.

One of the clearest examples was a data wrangling process used to create the infrastructure for monitoring student outcomes. The workflow processed approximately 300 million records and took 8 hours to complete on the legacy environment. Beyond performance limitations, incorporating unstructured and qualitative data required manual workarounds that slowed analysis and limited the organisation’s ability to work with emerging data sources.

The aging platform also created operational challenges for the team itself. Specialised skills were increasingly difficult to hire for, making it harder to scale analytical capabilities and modernise workflows. Analysts spent a disproportinate time navigating tooling limitations rather than generating insight.

“We had reached the point where the platform simply wasn’t aligned with the kind of analytical work we needed to do,” says Mark Gittoes, Head of Analytical Innovation at the Office of Students. “We needed an environment that could support both the scale of the data and the pace of decision-making.”

One platform, one source of truth

The Office for Students moved to Databricks to consolidate data, analytics and AI workflows onto a single governed platform. Bringing structured datasets, qualitative information and near-live data into one environment fundamentally changed how analysis could be performed for assessing risks, understanding provider performance and supporting decisions that impact student outcomes at scale.

Instead of working through disconnected systems sequentially, teams can analyse multiple sources simultaneously within a unified architecture to surface actionable insights more quickly and consistently across the sector.

Databricks helps us turn complex higher education data into faster, more trusted insights that support better decisions for students and providers across England. —Mark Gittoes, Head of Analytical Innovation, Office of Students

The platform also improved governance and collaboration across teams. Unity Catalog provides the data lineage, consistent access controls and security patterns that a regulated environment requires, while giving analysts greater confidence that appropriate guardrails were in place when working with high-stakes education data across hundreds of providers. This has created a more scalable foundation for experimentation and AI adoption without compromising governance requirements and ensured that insights used in regulatory decisions could be traced, validated, and trusted.

With all data centralised, analysts are able to iterate more quickly and focus on higher-value work rather than maintaining fragmented pipelines or manually stitching together outputs from multiple systems, shifting effort from data preparation to interpreting what the data reveals about higher education quality and risk. The modernisation effort also broadened the organisation’s hiring flexibility by aligning its tooling with more widely adopted modern data skills.

“Having everything in one place changes how quickly you can move from a question to meaningful analysis,” says Gittoes. “It allows us to spend less time preparing data and more time understanding what it’s telling us about risks, trends and outcomes across higher education providers.”

From faster analysis to better-informed decisions

For the Office for Students, the value of AI is not about replacing human judgment. It is about reducing the friction that slows analysis and helping teams surface relevant information more quickly so they can better understand risks to higher education quality and student experience across England. “As a regulator, humans are always in the loop on this,” says Gittoes. “It’s always decision support, not decision making.”

At the Office for Students, Genie Code reduced the time and cost of complex analytical tasks. A student segmentation analysis that would have taken two analysts at least two weeks was completed in half a day. A proof of concept for provider registration triage, previously requiring two to three colleagues reading documents manually over a month, now supports the flagging of missing submissions before a full assessment begins, helping teams identify potential issues with provider compliance earlier in the regulatory process and reducing delays in assessing institutional readiness.

The impact extends beyond efficiency gains. Faster access to trusted information enables the Office for Students to identify issues earlier, better understand trends across higher education providers that may indicate emerging risks to student outcomes or institutional performance, and make more informed regulatory decisions that help protect student outcomes and strengthen accountability across the sector. With a modern data and AI foundation in place, the organisation continues to shorten the distance between a question and a confident answer while ensuring human oversight and accountability remain central to every decision to support higher standards across English higher education.

Get regular updates about how Databricks helps public sector organizations unify data, govern AI and turn knowledge into action at global scale by following Databricks for Public Sector on LinkedIn.