For any kind of business, survival is tough. You have the commonly cited statistic, in America, that 75% of new businesses fail within five years. But then, new numbers came along.

It turns out that to the extent we can measure AI implementation, a full 95% of pilots fail to provide the ROI and results desired. So that’s one fifth of the 25% of traditional businesses that can keep the doors open. Just for context – that’s not related. But it does show you how hard it is for business leaders to address AI goals.

At this year’s The Next Endeavor event, in Mountain View, CA, I saw a panel discuss the realities around beating those odds. What’s the special sauce for companies that actually get a working pilot off the ground, and go on to develop it some more? (Disclaimer: With Imagination in Action, I help to put on these events.)

It was a great lineup: we had Paul Baier, co-founder of GAI Insights, and Andrew Lau, co-founder of Jellyfish. We had Pankaj Thakkar, CEO and co-founder of Kloudfuse, and Rahul Todkar, founder and CEO of Nasik, as well, to talk about agentic business, the challenge for humans, and much more.

There wasn’t an actual table, per se, but there were introductions.

Lau explained what Jellyfish does:

“Very few companies are able to get the full yield of the productivity that they actually aspire to,” he said. “And so we help companies through the transformation, through observability, telemetry, and transformation.”

He also gave this thesis statement:

“The main message is actually, I think AI now outpaces the challenges of human transformation,” he said. “Most companies aren’t seeing this yield, because it’s hard to get your teams, your roles, your people, and your processes to change. That’s actually the harder lift that’s moving through here.”

“We do production observability,” Thakkar said. “(Kloudfuse) runs the whole observability stack, metrics, logs, traces, across your whole infrastructure, applications, and now agents, with a twist. And the twist is that all the data stays in the customer’s environment.”

We started Nasik with the idea that building agents has become super-easy now,” Todkar said. “I mean, we all know this, especially at this point in time, where it just takes a few seconds, two minutes, to build an agent. But running agents in production is extremely hard and challenging, especially if you’re an enterprise. And so, what we’ve built is an open runtime platform for all your agents, coding harnesses, frameworks, and tools that brings it all together.”

“Everybody’s pushing toward AGI, which is fantastic,” he said. “I Love that. You know, I have a lot of research background myself. But now, I would say the idea of AGI has to also really take into account ‘IGA,’ which is, I would say, intelligence that is governed and administered. So agents without access, without governance and control, is a very, very dangerous situation. And so that’s what I would love for us to think about.”

“We do a fair amount of presentations and discussions at the board level,” Baier added of GAI Insights, “with executives, and this stuff is so confusing and moving so fast.”

Some Navigation of the AI Business World

The panel then talked about use cases, and related measurements of how successful companies can be. Lau explained how software development uses can be distinct from other more generalized pilots.

“Everyone’s talking about ROI,” he said. “We actually just look at the denominator, the ‘I,’ the token cost. Right now, very little is actually talked about the return. I think engineering is actually one of the places where you can start seeing the return—the acceleration.”

“There’s another dimension to this,” he said. “It’s based on persona, and how these AI products and agents are used in enterprises …. because, to me, there’s so much dependent on that, where the usage pattern becomes so important. There are verticalized solutions which are geared toward certain functions, which could be right from product engineering, finance, HR. Every single function has their specific usage patterns, and that becomes very important.”

He also mentioned general usage, and other specializations that might require their own approaches. But the one pillar that presenters kept coming back to is governance.

“Something needs to bring it all together across all three buckets,” Todkar said. “That’s something to think about as well.”

Thakkar had some additional insights on this.

“The question I always ask is, ‘okay now it’s in production - who’s responsible, who’s going to react when things don’t work?” he said. “Because systems are complex.”

More on Business Success and Responsibility

As the group discussed metrics and more, Todkar pointed out some of the difficulties of getting ahead of ROI measurement.

“It just becomes quite hard, and so, again, more and more conversation needs to happen on that,” he said.

Lau, for his part, suggested that, in general, as a business community, we’re getting there.

“The world keeps changing every three months on this stuff, right?” he said. “But at least there’s a second derivative of normalcy here that we’re starting to see, and patterns as industries.”

Other discussion points included how to get information on productivity gains, how to get transparency through patterns and repetition, and how to cut through the opaque layers of implementations that can resist this kind of analysis. But Lau was ebullient.

“We’re starting to see what good looks like,” he suggested.

Observability and Bird’s Eye View

The panel also talked about observability, which Thakkar defined this way:

“Ever since the beginning of computing, this whole idea of performance monitoring, or why the systems are doing something, that question has been asked, so we are kind of fortunate that the layers have been built in as people have built in the different systems.”

“What’s the impediment there?” Baier asked the group. “Is it technology? Is it leadership? Is it business case and ROI?”

In response, Lau noted that in the beginning, in his view, people were doing things out of FOMO.

“As we’ve matured, we actually start (to improve),” he added. “When you initiate these, you actually have a clear picture of: what do I want out of this?”

Going back to business solutions, Todkar had two commandments for applications: articulate the why, with a clear process, and look out for the limitations of sandboxing, where something might not be fully fleshed out into a production life cycle.

“Your agentic stack looks so different in production,” he warned, also noting that debugging takes a long time. “There are layers of opaqueness stacked on each other, and that, in production, is a whole different thing. So that’s why you need a full understanding of what’s going on.”

Thoughts on Open and Closed Source Models

Nearer the end of the segment, the panel talked about the outlook for open source models in a future world of AI advancement. All agreed that open source models doesn’t mean Chinese models, necessarily, and that it’s crucial for business people to think of model design beyond just a geopolitical context.

“You have a criteria with which you select from them, and this is why I think you’re seeing it make the most dent today in production workloads,” Lau said. “There’ll be a huge diversity of costs and utility, once we understand the return on each of these things.”

“I think everybody wants open models,” Thakkar said. “It just makes sense. It’s just so much easier. You can just adapt it to your use case. I just don’t think as an industry we are there yet.”

There was a lot more in this talk! Check out the video for audience questions and everything else. And keep these tips in mind if your company is venturing out onto the ice.