30,000 tables, zero context: Why legacy data architecture remains AI’s biggest enemy

Enterprise AI ambitions are stalling not because models are hard to build, but because the data foundations underneath them were never designed to support intelligent workloads at scale — and a unified data lakehouse architecture might be the solution.

The problem is especially acute for legacy organizations carrying decades of accumulated data infrastructure built in silos. As companies scramble to prepare their data estates for agentic AI, those that fail to address technical debt at the data layer risk building AI on a foundation that will buckle under real production pressure, according to Debopriyo Nag (pictured, right), global lead for data and analytics at Quantiphi Inc. That perspective is perhaps best illustrated by Quantiphi’s work with John Wiley & Sons Inc., where consolidating fragmented data became the foundation for a broader AI-ready modernization effort.

“When we started with the journey we realized that there is huge amounts of data across 30,000 tables lying around different business units and they all were running in their own fashion,” Nag said. “We were not able to contextualize the data for downstream AI or [business intelligence]. We were not able to bring in the connections across data from different domains.”

Nag and Mehul Trivedi (left), group vice president of technology at Wiley, spoke with theCUBE’s Rebecca Knight at Phi Moments @ Next, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how Wiley is reshaping its approach to publishing through a unified data lakehouse architecture. (* Disclosure below.)

Unified data lakehouse strategy powers Wiley’s AI-ready modernization

Wiley, a 219-year-old publishing company, found itself carrying a fragmented data ecosystem built over a decade — one in which individual business units maintained separate data warehouses with inconsistent schemas across tens of thousands of tables. With vendor renewals approaching, the company saw an opportunity to rethink the entire stack rather than simply renew, Trivedi noted.

“[We asked if] the ecosystem that we have today — is it the right ecosystem? Is this something that we really should be investing [in] or should we be preparing ourselves for the AI, ML-based new era?” Trivedi said. “We started thinking that we should be looking for a holistic data ecosystem where we have the trusted ecosystem [and] the right way of garnering our data.”

Google Cloud’s BigQuery became the foundation of choice on three grounds: economics, technology integration and open-source flexibility. Quantiphi’s proprietary AI migration tool, Codeaira, enabled the team to automate query translation and pipeline migration across all 300 terabytes within a six-to-nine-month window — a timeline that typically runs one to two years, Nag noted. AI agents drove accuracy and speed throughout discovery, execution and validation stages of the migration.

“The people who are going to be [winning this battle] are the organizations preparing themselves for the long game. ‘How am I going to build my organization for the next 10 years to compete in this kind of environment?'” Trivedi said. “We really need to make sure that we are investing in our talent, investing in our people and making sure that the people are being guided in the right way.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the Phi Moments @ Next event:

(* Disclosure: TheCUBE is a paid media partner for the Phi Moments @ Next event. Neither Quantiphi, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE