Ask your data anything. Can you trust the answer?

They’re not wrong. “Ask your data anything” has stopped being a promise. Since 1 July 2026 BigQuery Conversational Analytics is generally available: Gemini turns natural language into SQL, reasons across multiple tables and calls functions like ML.FORECAST and anomaly detection on its own. Looker has a comparable Conversational Analytics offering. And external LLMs can now connect straight to the GA4 API through the Model Context Protocol, read-only for the time being.

So yes, the demo works. But the demo works because the demo data is clean. Yours probably isn’t.

The real problem isn’t the model

At SDIM we keep bumping into the same misconception. Companies ask themselves which AI tool they should pick. That’s the wrong question to start with. An AI agent on your analytics data doesn’t fail because the model is too weak. It fails because the data underneath it is a mess.

An LLM that queries GA4 or BigQuery inherits all of it. Every inconsistent naming convention. Every metric that’s defined slightly differently in three separate places. Every gap that opens up the moment consent is missing. AI amplifies your foundation in both directions. Get the foundation right and you get an agent that analyses faster and broader than any human could. Get it wrong and you get nonsense, but nonsense that’s phrased so convincingly nobody questions it.

Here’s the part most people underestimate. What you give up is the human sanity check. An analyst who sees a spike in conversions thinks: hang on, that’s a tracking bug, I was wrestling with that exact thing last month. An agent has none of that reflex. It reports the spike as an insight, neatly worded, with a recommendation thrown in for good measure.

And then there’s the trust cliff, which is merciless. The moment a stakeholder catches one wrong figure in an AI answer, you’ve lost them. They walk straight back to the dashboard or, worse, to their own Excel. Budget was never the bottleneck. Trust is. And trust evaporates on a single miss. None of this is new, by the way. Bad data was already expensive back when a human was reading it. Now it just scales.

Why AI raises the bar

You’d think humans make mistakes too, so what’s the difference? Fair. But humans hesitate. An LLM has no domain knowledge of your schema and no gut feeling for what looks off. Ask it something ambiguous and it guesses. With full conviction.

Three things make this genuinely hard:

• Semantic ambiguity. “Revenue” can mean ten different things. The agent picks one, and not necessarily yours.
• No domain understanding. It has no idea you switched tag implementations in Q3.
• Confidently wrong. The dangerous one. A wrong answer that looks professional gets believed.

The rigour that used to make human analysis reliable now counts for even more. AI has far less tolerance for mess than your colleague does.

Five layers between a question and a trustworthy answer

Now for the core of it. One idea changes the whole picture: your measurement plan used to have exactly one reader, the human. Now it has two. The AI agent is a second reader that’s completely literal, has no institutional memory and never wanders over to a colleague to ask what you meant.

Between a question and a trustworthy answer sit five layers.

Collection. It starts with a documented measurement plan. Events and parameters that carry real business meaning, one naming convention, one single source of truth. Consent belongs here too, which people tend to forget. Say 30% of your conversions are modelled because users didn’t give consent. An agent that doesn’t know that will happily draw conclusions about figures that are partly estimated. Wrong start, wrong end.

Structure and modelling. Raw data in BigQuery, cleaned and modelled. Enriched where it matters with cost data, CRM data and margin. Deduplicated, so one transaction doesn’t get counted twice. And derived metrics you define once, instead of reinventing them every time in yet another report.

Semantic layer. This is the layer the AI grounds itself on, and by far the biggest factor in whether your agent gets things right. Here it’s nailed down that “revenue” means exactly one thing. That “conversion” means exactly one thing. Metric definitions, entity resolution, business logic. Get this layer right and the rest tends to fall into place. Leave it out and the agent improvises, which is the last thing you want.

Governance and access. Row and column level security, so not everyone sees everything. Version control on your definitions, because definitions do change. And a feedback loop that catches drift, because data shifts more quietly than you’d expect.

Human in the loop. The agent proposes, a human validates anything that touches a decision or a budget. That only works when answers are inspectable: visible reasoning, the SQL that was actually used, source citations. You can validate a glass box. You can’t validate a black one.

The plan the AI reads

Now the layer that, honestly, makes the biggest difference of all. And that almost nobody is working on yet.

Your measurement plan has been written for humans for decades. Prose, a few slides, here and there a tab tucked away in a spreadsheet. That worked fine, because the reader filled in the gaps themselves. A colleague reads “revenue” and knows from experience whether VAT is in there. The agent doesn’t know that. It’s literal, it has no memory, and it never asks a follow-up question.

So write the plan for the machine as well. Concretely:

• Put your definitions in a machine-readable, structured form. A semantic or metrics layer like the dbt Semantic Layer or LookML, or metric definitions in YAML or JSON. Not as prose buried on slide 14.
• Give every metric a formula, a grain, allowed dimensions, filters, units, synonyms and aliases (so “turnover” resolves cleanly to “revenue”) plus the known caveats.
• Bake in entity resolution. Canonical names with aliases for products, campaigns and channels. Otherwise “Google” and “google / cpc” get counted as two separate things.
• Set up trusted query templates. Verified example questions with the correct query attached, so the agent reuses proven patterns instead of improvising from scratch.
• Treat data caveats as first-class context. Tell the agent what a human simply knows: sampling, consent gaps and modelled conversions, periods where tracking changed, weeks with known bugs, deprecated tables.
• Write down your guardrails and instructions. Which tables are authoritative, when the agent should flag uncertainty rather than guess, always show the SQL or the reasoning, and where the security boundaries sit.
• Keep one source of truth. The human plan and the AI-facing context come from the same definitions, so they can’t quietly drift apart.

And this is the line I’d want you to take away: the AI-facing plan is how you encode the sanity check that used to live inside the analyst’s head. All that knowledge nobody ever bothered to write down, because everyone just knew it, has to come out now. On paper. Or better, in YAML or JSON.

An example from practice

We worked with a company that wanted an AI assistant to answer “why did revenue drop last week”. Sounds simple enough. Before a single line of AI came anywhere near it, we ran an audit.

What did we find? Three different definitions of revenue in circulation: one in GTM, one in GA4, one in the warehouse. They disagreed on VAT and on whether returns were counted.

There was no plan the agent could read at all. Set an AI loose on that and it would have blended the three definitions together and averaged out the difference. Convincingly phrased, completely wrong.

Most of the work went into repairing the measurement plan, defining the metrics once, and writing them in a form the agent could actually use. The AI layer itself turned out to be the easy last step. As it so often is.

Are you ready?

A short checklist to hold up against your own situation:

• Is your collection clean and is consent set up properly, including insight into what portion is modelled?
• Are your metrics defined once, and the same everywhere?
• Do you have a machine-readable semantic layer, and not just prose in slides?
• Are there trusted query templates for your most common questions?
• Are your data caveats documented somewhere the agent can read them?
• Do you have guardrails: which tables lead, when the agent should hesitate, and where the boundaries lie?
• Does a human validate everything that touches a decision or a budget?

Anyone answering “no” to most of these isn’t ready for an AI agent yet. And that’s fine. Better to know now than after the first mistake lands in the boardroom.

The unglamorous truth

The winners of the AI-analytics era won’t be the companies with the smartest models. They’ll be the ones that did the boring measurement work first and wrote it down in a form their AI can read. AI doesn’t replace the discipline. It rewards it, and it punishes the absence of it harder than anything did before.

So the real question isn’t “which AI tool”. It’s whether your measurement is set up so an AI can be trusted with it.

If you want to gauge how far along you are, we put together a piece on whether your organisation is ready for AI, with an AI-readiness checklist and a measurement-plan template you can use for exactly this. Want us to look at your setup with you? Get in touch for a no-strings chat and we’ll walk through it together.

SDIM is sponsor of Experimentation Heroes 2026. More information and tickets for Experimentation Heroes taking place on 1 October at Hotel Casa in Amsterdam.

Jesse Wieleman

Team Lead Data & Analytics } SDIM

• LinkedIn