What should Chief AI Officers actually be investing in as AI models and agents rapidly evolve?

In this episode of CIO Connect, Sanjay Puri sits down with Vijay Krishnan, Founder and CTO of Turing, to explore one of the biggest challenges facing enterprise AI leaders today: how to evaluate AI systems rigorously enough to make the right decisions.

Vijay explains why he believes evaluations—or “evals”—are the skill Chief AI Officers most underinvest in, and why enterprises need to build disciplined evaluation frameworks before deciding which models, agents, or AI systems to deploy. 

The conversation goes deep into the changing AI data landscape, the role of human versus synthetic data, reinforcement learning environments (RL Gyms), AI agents, enterprise-specific evaluation, and how companies can safely train and test agents without exposing production systems to unnecessary risk.

Vijay also explains why enterprises should focus on their three to five most important business use cases, build robust evals around them, and compare models based on quality, cost, speed and business value rather than simply choosing the most powerful frontier model. 

In this episode:

  • Why AI evaluations are becoming essential for every enterprise
  • The difference between synthetic data and expert human data
  • Has the AI industry already hit the data wall?
  • What RL Gyms are and how they train AI agents
  • How enterprises can build simulated environments for AI agents
  • Why automated evaluation rubrics matter
  • How to evaluate frontier models against smaller, cheaper models
  • Why proprietary enterprise data can become a competitive moat
  • How regulated industries can safely train AI agents
  • Why enterprises should avoid prematurely investing in unsolvable AI use cases
  • Why long-horizon AI agents are likely to become much more capable
  • How Chief AI Officers should think about AI costs
  • When to build internal evaluation capabilities vs. work with an external partner
  • Why AI-forward employees will become increasingly important
  • The three stages of enterprise AI adoption
  • Human data vs. synthetic data
  • Frontier models vs. fine-tuned small models
  • The one skill Chief AI Officers most underinvest in
  • What enterprises should do this quarter to prepare for the next phase of AI

Vijay’s advice for enterprise AI leaders is simple: identify your top three to five business use cases, create rigorous evals for them, and understand the trade-offs between quality, cost and speed across different models. 

Chapters

00:00 The AI skill Chief AI Officers underinvest in

00:32 Welcome to CIO Connect

01:10 Meet Vijay Krishnan, Founder & CTO of Turing

02:04 Has AI hit the data wall?

03:43 Human data vs. synthetic data

05:17 The enterprise AI data challenge

06:30 Finding the right AI use cases

07:33 What are RL Gyms?

08:19 How AI agents learn in simulated environments

09:10 Training computer-use agents

10:07 Building automated AI verifiers

11:18 Why agents need thousands of iterations

12:35 Can enterprises build their own RL environments?

13:44 Why enterprise evals matter

15:50 Can proprietary AI data become a competitive moat?

16:19 AI agents in regulated industries

17:54 Choosing the right enterprise AI problems

18:51 Why disciplined evals will become table stakes

20:29 Can RL environments be reused?

21:37 The economics of AI training and evals

23:10 How to choose expert human reviewers

25:06 Build vs. buy AI evaluation capabilities

26:59 What Chief AI Officers should prepare for

28:35 Frontier models vs. cost-efficient AI

29:00 How enterprises should optimize AI costs

30:07 The three stages of enterprise AI adoption

32:42 The future of enterprise AI economics

32:51 Lightning Round

33:14 The one skill Chief AI Officers underinvest in

34:10 Data vs. compute: what’s the bigger constraint?

34:33 Vijay’s advice for AI leaders this quarter

35:03 The CIO Connect AI playbook 

Follow CIO Connect for conversations with Chief AI Officers, CIOs, CTOs and enterprise technology leaders shaping the future of AI.

#EnterpriseAI #ArtificialIntelligence #AIAgents #ChiefAIOfficer #AILeadership #AIEvals #GenerativeAI #ReinforcementLearning #AIData #Turing #CIOConnect #SanjayPuri #VijayKrishnan