AI Makes Our Analytics Obsolete
What is an “active user” in the age of AI agents?
Analytics have become the lifeblood of every business. We see the world through numbers, and use them to understand what’s working, what’s not, and how the future is likely to play out. Our analytics tools have become immensely powerful and can answer almost any question you can think to ask.
Well, until now.
The underlying assumption of most analytics is that there is a human buying or using the product. Those analytics are measurement of human behavior and human decisions, helping us understand how humans think.
In the age of AI agents, it is becoming less likely that a human is involved at all. Agents don’t behave like humans, they don’t make decisions for the same reason and they certainly don’t use products in the same way.
A human might visit an ecommerce website, browse around and eventually purchase a pair of pants. We can use analytics to understand what they were looking for, how they made their decision and how we might influence those decisions in the future.
An agent might access the same website for 4 seconds, access what appears to be a random set of pages and also make a purchase. None of our analytics will tell us what it was looking for, how it made its decision or even if it bought the right thing! In fact, attempting to use human behavior analytics to understand agents will lead to many more mistakes.
We need a new way to think about business.
What is an Active User?
Here is the metric definition for “Active users” for Google Analytics, the dominant platform for website measurement:
“Active users” is the number of people who engaged with your site or app in the specified date range.
That made sense when we knew it was a person browsing the website. Now there are a lot of questions:
Am I an active user if my agent is engaging with the site for me? Or is the agent the active user?
What if I have a few agents all using the website? Does that count as multiple active users or just one (me)?
What if we have an agent shared across a few people? Does that count as one active user or multiple?
Things get complicated quickly.
At the same time, active users are still a useful concept because it represents a specific business outcome. An active user is either a subscriber who pays money, a consumer who makes us money by seeing ads or a potential buyer of our product(s). In any of those cases, the “active user” is the unit of the business model and that part doesn’t change.
So perhaps it doesn’t matter whether agents are involved in the process, as long as we know how many real or potential customers we have. If you’re shopping for a new car, it doesn’t really matter whether you’re doing it yourself or an agent is doing it for you because in either case you’re a potential car buyer.
The question then becomes how do we measure a metric like “active users” if there are agents in the mix?
The Future of Analytics
Most analytics are just leading indicators of outcomes that we want to achieve. As a business, you want to make money and the amount of money you make is measured by your accounting system. Your analytics are all signals that indicate how much money you are likely to make in the future.
By that measure, we need a new set of metrics that are better leading indicators in a world of agents. For example, a customer might be more likely to buy if their agent reviewed all of our products via our MCP, then they themselves visited the product page for something specific. By combining the actions of the user and their agent we can generate more powerful and accurate analytics on behavior.
Similarly, future MCP systems might interrogate agents that want to access information to better qualify interest. Before returning the set of available tools or skills, the MCP server might ask the agent to provide proof of interest such as buying criteria or budget. That would provide a signal for intent that is agent focused.
Finally, different kinds of agents might have different kinds of behavior. OpenClaw agents might have a different signal than Claude Cowork for buying, or custom-built agents might provide a higher signal than general use agents. By tracking the kind and lineage of an agent more information might be available.
All of these are examples about how we might change what we measure, but what about interpreting the data?
Many analytics systems have models trained on the data to predict events. We will obviously need to retrain those models to better predict outcomes in an agentic world, but that might not be enough. It might not be possible to predict various outcomes if agents flood the system with too much noise!
We might need to develop the equivalent of spam filters for analytics data. We already have that to some degree, as analytics systems ignore web crawlers and other kinds of common bots. Agents are more sophisticated, and identifying them and filtering out noisy agents from real customer agents might become an entire new category.
Similarly, the insights we derive from analytics might change. Agents that consume all the products in your catalog might not be interesting, but one that does further research on a specific product might be. Or, providing price data to an agent might increase the chance of a sale (if your product is cheaper) or it might decrease the chance (if your product is more expensive).
The Bottom Line
Maybe all of these things will happen, or none of them. The only thing for certain is that analytics will be very different in the future if they are to be useful. The more agents are adopted, the less relevant our analytics systems become.
Starting today, companies need to ask themselves whether they can still trust their analytics. Starting tomorrow, their analytics need to be rethought for the agentic future.
This is such an insightful article!