One Does Not Simply Sprinkle AI on Top

The Call

It starts with a message. Teams or Slack, it doesn't matter which; they all make the same little sound. Someone has put two dashboards side by side. Both have a tile called "Active customers", and the two tiles show two different numbers.

"How can we not know how many active customers we have? This is ridiculous."

Before you've finished reading, the video call starts ringing. You know the tone. Somewhere far away, something with a very good view of your calendar has turned its attention toward you. You wonder, briefly, whether it would be rude to let it ring out. Then you see who's calling, and you understand that letting it ring out stopped being an option the moment those two tiles ended up in the same screenshot.

The uncomfortable part comes later in the call. Both numbers are probably correct. Each one answers a question somebody asked once, and they were two different questions wearing the same name.

How We Got Here

Nobody sets out to build two versions of "active customer". It happens one deadline at a time.

A stakeholder is under pressure. The board meets on Thursday and they need the number of active customers, fast. A heroic data engineer shifts up a gear and finds the tables. On the way they make a dozen small decisions (test accounts out, paused subscriptions in, anyone who paid in the last 30 days counts), and the number lands on Wednesday night. The meeting goes well. Everybody is grateful.

Months pass, maybe years. The dashboard is forgotten. The hero changes jobs, and the dozen small decisions leave with them.

Then a new stakeholder needs the same number for a different meeting. A new hero shifts up the same gear, makes their own dozen decisions (anyone who logged in during the last 30 days counts, paid or not), and delivers in the nick of time. There's cake in the kitchen.

The lost dashboard was never lost, though. Adam in controlling still opens it every month, because it's the number his forecast is built on. Now there's a second number, newer and shinier, and it contradicts his.

Nobody's at fault, and it's everybody's fault at the same time. Both heroes did good work under real pressure. The trouble is where the definition lived: in one person's head and one SQL file. The second hero couldn't read the first hero's head, and nobody knew which folder the file was in. Nextory's data team lived inside this pattern before they rebuilt their platform, and The Work Before the Answer tells how they got out. You'll recognize it in any company with dashboards older than its newest employee.

Hero Number Three

Now the agent arrives.

It's fast. It's eager. It delivers in seconds, with a nicely formatted table and an encouraging tone. It also has no memory, and it doesn't know about Adam's dashboard either. Ask it how many active customers you have and it reads the tables, picks a definition that looks plausible, and hands you a third number. Same failure as the heroes, at machine speed, billed per token.

The tempting move, when leadership asks for AI, is to put the agent on top of whatever is already there and hope the smart model papers over the mess. One does not simply sprinkle AI on top. Gustaf Sköld, COO of Nextory, put it better than I can when we wrote up their work:

Sprinkling LLMs on top of a weak foundation does not yield great results.

You can do it, of course. You just pay for it. Every question sends the agent back to rediscover what "active" means: reading schemas, sampling rows, guessing at status codes, spending tokens on meaning that was never written down. And after all that, Adam still has a number that disagrees with the new one.

The clearest public account of what that costs comes from a team that took it seriously. In Context Isn't Enough, Lars walks through Anthropic's own write-up of their internal analytics agent. Claude now handles about 95% of their internal business analytics queries, with roughly 95% accuracy. Without skills (documents that tell the agent how the data model works), accuracy never got above 21%. And within a month, that 95% had slid to 65%, because the documents described a data model that changed daily.

Anthropic fixed the drift with real engineering, and they published the drop along with the 95%. That candor is what makes the account useful. It shows that the meaning had to be written down for the agent anyway, and that keeping the writing in step with the platform became a job of its own.

One Cause: The Model Was Never Declared

Two dashboards and an agent look like two different problems. One is an organizational mess, the other is an AI accuracy problem. They have one cause: the meaning of "active customer" was never declared. It only ever existed as a side effect of a query.

Declaring the model means writing down, in plain words, the things the business talks about (customers, subscriptions, orders), what each one means, how they relate, and how they change over time. Write it once, in one place, and people and agents read the same thing: one declaration, which is what "one version of the truth" was always supposed to mean.

It helps to split the work in two. The model is the meaning, and people decide it. What makes a customer active is a business decision, and it belongs in a room with Adam in it. Mappings are the plumbing: this column in the billing system is when a subscription started, that field in the CRM is when it ended. Mappings are mechanical once the meaning is settled, so agents are good at writing them, and a check can validate what they wrote.

That changes the agent's job. Hero number three stops guessing what "active" means, because it can read the declaration. Its speed goes into the work it's good at: connecting new sources to a meaning that people already agreed on. And when one of its mappings is broken, the check says so before Adam does.

If that sounds like "write better documentation", or like the semantic layer you already bought, it isn't. Documentation, catalogs and semantic layers all sit next to tables that were built separately and describe them. When the tables change and the description doesn't, you get Anthropic's slide from 95% to 65%. A declared model works the other way round: the SQL and the tables come from it, since they are generated, not hand-crafted, so there's no description on the side to fall behind. And when finance changes what "active" means, there's one place to change it.

This is the idea we built Daana around. You declare the model in YAML and daana-cli generates, deploys and loads the SQL, and agents write mappings that daana-cli check validates (the AI-assisted mapping guide shows the workflow).

It's Still Work

None of this makes the argument about "active customer" go away. Someone still has to get finance, product and controlling to agree on what the word means. That conversation is slow and occasionally loud. It's also the most valuable hour the data team will spend that quarter. It's the same conversation both heroes had alone with a SQL editor at midnight. Declaring the model just moves it into daylight and makes its result stick.

It's also overkill sometimes. If you have three tables and one stakeholder who sits next to you, declare nothing. Write the query, ask them what they meant, and go home on time. The case for declaring grows with the number of people asking and the number of years the answers have to survive, and an agent counts as one more person asking, a very fast one.

Start Small

You don't need to model the whole company to find out whether this works. Trying to model everything at once is the classic way information modeling stalls, so don't.

Pick the one concept your organization argues about. It's probably "active customer", or whatever your business calls the people who pay you. Get the people who own that number together, Adam included. Describe what the thing is before you fight about its name. Declare it in one place that people and agents both read, and point the dashboards and the agent at that declaration.

Then watch what changes. Does the next "why don't these match?" message still arrive? Does the agent give the same answer twice? Does Adam's forecast still hold? You'll know within a few weeks, and the answer costs one concept's worth of effort.

It's a small thing to do. Small hands have moved bigger things than a dashboard tile, though, and the next time that ringtone goes off, you might even pick up with a smile.