The question every executive and VP is dreading right now is "What are we actually getting from our AI investment?” Many leaders are investing heavily in new tools and subscriptions for their teams, launching pilots across departments and announcing company wide AI mandates, only to quickly find out that measuring value and ROI is difficult in practice.

The roll out of any emerging technology is inherently chaotic. Experimental phases are difficult to control, and many are experiencing the wake up call that their legacy systems and data were never appropriately configured in the first place, making scale impossible from an unstable foundation.

Still, as Artificial Intelligence continues to evolve, creating the right infrastructure for measuring adoption and value can be possible by taking a few initial steps that can put your organization on the right track to predictably measured outcomes.

Why Most AI Measurement Attempts Fail

Let’s start with why most attempts at measuring the impact of AI fail. In many cases current implementations of AI are entirely disconnected, with different teams and individual contributors using the tech to varying degrees without much direction. It’s true that standardizing something that’s changing so quickly can be challenging, but making some decisions on best practices early on that are then periodically adjusted can make a big difference on setting the right expectations for the team.

As an extension of this, most teams don’t have clearly defined success criteria for what good usage actually looks like, and therefore no explicit KPIs tied to either P&L cost savings or revenue growth activities. With no criteria and specificity around desired results in place, accountability becomes impossible, further obscuring the measurement problem.

Start With Closing the Knowledge Gap

The good news is that those that are creating a culture that encourages experimentation are already ahead of many other industry laggards. The next step is to start filling knowledge gaps on your team to be able to then define where and how value is being created.

Many non-technical teams are now being asked to become systems level thinkers that can scale their work, and while some people on your team are inherent tinkerers that are comfortable with pseudo technical work like setting up small automations, everyone else has varying degrees of knowledge gaps that haven’t been addressed. Fortunately, the learning curve here is not insurmountable and it starts with just a bit of context and awareness building.

Learning by seeing how technology is leveraged in the real world, and in the functional areas related to the team in question can immediately provide clarity where there’s currently too much ambiguity. Encouraging participation in free resources like expert led AI seminars is one way to bridge critical skills gaps while providing your team with guidance from people already successfully implementing these tools.

From there, understanding where and how value is created requires having somebody responsible for tracking various initiatives within the organization, and tying them to specific business outcomes that connect to a line item on the P&L.

What Good Measurement Actually Looks Like

For example, the Client Success team is charged with optimizing how data is surfaced on high risk clients that are about to churn. The team builds an automation that ensures all client interactions are stored in Salesforce, with a daily cron job that surfaces clients that have not been contacted or heard from in at least two months that have a contact renewal coming up in 30-60 days.

Any client that meets this requirement gets a custom email drafted by a well crafted prompt in Claude that’s scheduled by a reliable orchestration tool like Make or Zapier, and each rep is trained to check their email drafts folder each morning to see if new drafts were created before they’re manually verified and sent.

From there, the team extrapolates several metrics tied to the P&L. First, number of clients flagged per week by the automated system compared to the manual work previously required to pull the data and craft the email message, multiplied by the cost of the client manager’s time. Second, the number of clients that reply and renew, directly tied to either an improved retention metric or direct revenue expansion.

Building Your AI Scorecard

Working from a granular definition such as this, each team’s initiatives can now be tied to an AI score card that clearly shows time related costs savings, other direct expenses reduced, and revenue influenced. By coupling this with supervisor ownership, once achievable metrics are clearly defined, and a regular review cadence, teams can start to have clear visibility into the impact of their investments.

Over time your teams will be able to extract their highest ROI use-cases which will allow them to focus on fewer initiatives with the biggest impact, stripping away everything else that’s a distraction. And once this happens your engineering resources can be confidently dedicated to appropriately scale what works, while simplifying the decision making process for executives that finally have a reliable source of truth on what’s happening with their systems and on their teams.

Remember that almost every early experiment feels uncomfortably disorganized at first. However, by investing in the talent that you’re entrusting to help you work through the current evolution of how businesses operate and creating clear expectations about what’s required of each team, your organization will start to naturally work through the friction brought on by any large change such as this. Eventually this will lead to clarity of what needs to be measured, and what’s actually working well enough to be scaled.