Keeping pace with AI startups “running around with scissors” — to use Steve Santana’s metaphor — can be unnerving for established enterprises. But the CIO of Princeton-based Education Testing Service (ETS) believes companies can’t stand still.

In this installment of the IT Leaders Fast-5 — InformationWeek's column for IT professionals to gain peer insights — Santana acknowledges that AI presents major security and data management concerns for enterprises. Still, he said he is a firm believer that when used within proper guardrails on a foundation of customer trust, AI will be critical to scaling ETS and improving customer experience.

He also forecasts that AI will fundamentally reshape how enterprises approach cybersecurity and the size of their security teams. 

ETS provides academic assessment tests, such as the GRE. The organization conducts 50 million academic tests a year, and Santana's goal is to use AI to scale test distribution to exceed 100 million tests a year.

Related:What AI must learn from Roosevelt, conservation and 1929

Prior to his role at ETS, Santana was senior vice president and CTO of Pearson, held director-level positions at Ciena and Nortel, and was a manager at Apple.

This column has been edited for clarity and space.

The Decision That Mattered

What decision — technical or organizational — made the biggest difference recently, and why?

For us, it's the rapid adoption of AI. We have been using machine learning models for 20 years. All of our core scoring engines, the e-rater and SpeechRater applications, we built based off our data. They score just as well as the humans and are very reliable, but they're expensive to maintain because you're constantly updating those models and cleaning the data. 

Large language models   are much less expensive to integrate with, so we want to move in that direction — not just on scoring, but also on item generation, which are the test questions. 

A large language model can really help you in running the test by taking the test-taker through a process. Instead of just multiple-choice questions, I can start having a dialogue with you, and then I can have agents inside that model interact with you in the ways that somebody would in a real-world situation. We can process that you understand the approach, the process, the concepts, the competencies behind it.  

We wanted to use this technology, and we saw a great opportunity — but also a tremendous amount of risk . Like at every company, there were a lot of their security issues. We don't know where this data came from. How do we manage and work with it? 

Related:AI on trial: The Workday case that CIOs can't ignore

I very quickly just said that we don't have a choice — we have to do this. There's so many startups that are just running around with scissors, meaning they're going to do this without us, and they're going to start breaking trust with people. We should adopt it, and we should show people the right way to go about doing it. 

We take the 78 years of expertise that ETS has and the decades of data, and use that as a mechanism to leverage that trust. We already had high trust in our assessments. How do we build it in a way that people will trust what we do with AI? We don't use any data that comes from the large language models. It is always using reference data that is first-party data that we understand and utilize. 

About a year and a half ago, we decided to centralize this effort back under the technology organization. We have a seven-point plan of how we're implementing this technology, and many things are actually already in production today. 

We took the learnings of what everyone was doing through experimentation, and then we considered how we implement it at scale. The most important thing that we had was trust — our customers trusted us. We need to maintain that as we deploy AI in these new tests and models, and they're going to help us drastically reduce costs.

Related:Why AI teams treat training data like capital

The Hard-Won Lesson

What didn’t go as planned recently — and what did it force you to rethink?

A few years ago, at a different company, we were pushing the business to deploy a new CRM that was core to their business. We pushed them pretty hard on making the decision to move forward because they wasted millions of dollars on failed initiatives because they didn't move [quickly enough].

But, we had some decent-sized problems with the CRM when we rolled it out. Most of them were things that were missed — business requirements that we didn't gather. If we had taken some more time and had not pushed the business so hard on that, I think it would have gone smoother. When I pushed them, they weren't on my side. And when things went wrong, they were still not on my side.

Moving forward, I spend a lot more time with the business team, and I try to make sure that we're on the same side. I was focusing too much on the requirements of the technology and not enough time focusing on the business change management side of things. 

The Talent Trade-Off

Where are you investing in talent right now — and what are you consciously not investing in?

The utility player is even more important than they used to be. A lot of people are very focused on being strong in a specific domain. A utility player is someone who knows a little about a lot of things and can use agentic AI to round out the rest of their delivery capabilities.

If you have a very strong working knowledge of a number of domains, AI can help give you depth in those areas that maybe you didn't have access to before. You'll have engineers that want to do business analyst work. 

If you're an engineer and you need to convert business requirements into user stories, then you can have the AI do that. AI can help pull all of that together for you and write your stories. You've just now expanded your ability to deliver more and your value to the organization has exponentially grown. 

I'm not investing in one-dimensional player database administrators [DBAs] that just know everything about database performance management — those things are going away. We've encouraged our DBAs at ETS to learn more pieces of the business and build out their [roles as] utility player across those things. But we don't seek out specific individual domain knowledge.

The External Signal

What recent external development is most likely to change how your organization operates, even indirectly?

The easy answer is agentic AI   and how it will impact our security. Agentic AI can certainly help identify vulnerabilities and improve monitoring, but it also massively increases the risk profile. We're a decent-sized organization, so we have a big security team, but there are many small and medium businesses that don't have that security management.

That keeps me up at night — I was worried before Anthropic Mythos  came out. I'm glad that Anthropic is providing organizations with the tooling to try to get rid of their vulnerabilities. But if you have software that's more than 5 years old, you probably still have a large number of vulnerabilities that you're having to manage. I worry about organizations that just can't scale up to handle that.

But it's also a big opportunity. I think security operation centers are going to be a fraction of the size they are today. In  the next two or three years, you're going to see a lot of kind of changing of those [security] resources, because the AI can do the monitoring and learning to take action on things much faster than people can.

The Perspective Shift 

What have you read, watched or listened to recently that changed how you think about leadership or technology — even slightly?

I watch a lot of YouTube channels and send things off to my staff about a new Microsoft feature or a frontier AI beta release. It's cool spending a little time to be familiar with how companies are deploying their technology and how we can deploy this to our customers. 

I also like the podcast Masters of Scale, which talks about how startups scale their companies. You learn how their business manages growth and what they're prioritizing. 

I also like the YouTube channel Ship it Weekly. It's about the lowest-level technology change update — like different ways to configure and manage GitHub flows. It gives me little nuggets of information, so when I'm in an operations review or talking to developers, I sound more informed, and I can ask more questions. The developers sometimes start to think about things a little differently because they know their leaders are paying attention to those details. That gives us a more reliable product on the back end.

Get more IT leadership updates and insights three times a week direct to your inbox with the InformationWeek newsletter.

About the Author