AI on-call is confident. That's exactly the problem.

There's a new wave of AI-SRE tools that sit next to your on-call rotation, read your live systems, and tell you what's wrong. Some of them are genuinely good. I run assistants like this against real clusters. The technology isn't the thing I want to argue about.

What I want to argue about is a number from a study that came out this month, because it describes the exact failure mode these tools walk you into if you let them.

Researchers gave people access to AI advice and measured what happened to their answers. Accuracy dropped by roughly three times. Confidence went up by roughly two. And the share of people willing to say "I don't know" collapsed from 44% to 3%. The researchers deliberately used questions where the model tends to be wrong, so the point isn't "the AI was bad." The point is that people got more certain while getting less right.

If you do incident response, that last number should stop you cold.

The number that matters isn't accuracy

Everyone focuses on accuracy, because accuracy is easy to argue about. Your AI-SRE tool got the root cause right, or it didn't. Fine. But every tool is wrong sometimes, and a team that's paying attention catches wrong answers.

The failure that doesn't get caught is a different one. It's when the tool is wrong and fluent, and it quietly convinces you to stop checking. That's what the 44% to 3% collapse describes. It isn't that people trusted a specific wrong answer. It's that the mere presence of AI advice suppressed the habit of noticing what they didn't know.

On-call is built on that habit. The whole discipline is a series of "I'm not sure yet" moments held open long enough to look at the evidence. Remove the "I'm not sure," and you've removed the part that works.

Confidence is not calibration

Here's the thing a fluent summary hides: the words carry no signal about whether they're true.

A model that's guessing writes with the same fluency as a model that's right. There's no tremor in the prose when it's making things up. Human experts leak uncertainty through hedges, pauses, and "let me double-check." A confident paragraph of generated text leaks nothing. You cannot read calibration off fluency, and that's precisely the read your tired 3am brain wants to make.

The 3am cascade

The dangerous path isn't exotic. It's the most natural thing in the world when you're woken up by a pager.

Notice that nothing in this chain requires the tool to be bad. A good tool that's right 80% of the time still produces this exact cascade on the other 20%, because the 20% arrives wearing the same confident voice as the 80%.

What actually protects you

The guardrails I put around an on-call assistant have almost nothing to do with how much I trust the model. They exist to protect my own doubt, by making sure the model can inform a decision without quietly making it.

Three rules, in order:

Read-only by default means the assistant can read status, logs, and metrics, and nothing else. Diagnostics are safe because they're cheap to undo. Any state-changing action is a separate class of thing.

Approval-gated means every action that touches production stops at a human. The assistant proposes "restart this deployment" and waits. Nothing runs on prod without someone saying yes. That single gate is what converts a confident wrong answer from an outage into a rejected suggestion.

Made to cite means an answer without a source I can open is worth less than an answer with one. "Latency is up because the cache is cold, see this Grafana panel" is checkable. "Latency is up because the cache is cold" is a vibe. The citation is what lets me re-insert the doubt the study says I'll otherwise skip.

In practice that looks less like trusting a black box and more like giving it a narrow, typed set of read-only capabilities and routing everything else through a person:

assistant:
  mode: read-only            # diagnostics only, no write-back
  capabilities:
    - kube:get               # pods, events, describe
    - logs:read
    - metrics:query          # Prometheus / CloudWatch, read
  actions:
    restart, scale, rollback, delete:
      requires: human-approval   # proposed, never auto-executed
      audit: true                # who asked, what ran

The new generation of read-only AI-SRE agents is built on exactly this shape: a tool-calling agent driving a typed allowlist of read-only capabilities, building a root-cause hypothesis from live evidence, and proposing classified fixes that a human approves. That constraint isn't the model being humble. Humility is a property the model doesn't have. It's the design supplying the doubt the study shows the human will stop supplying.

What AI may do, and what it may only suggest

The line I draw is about how expensive the action is to undo. Reading is reversible. Restarting a production service at 3am, on a hunch, is not.

A confident wrong answer is only dangerous if it can reach production without passing a human. The gate is the whole point.

Use it, don't outsource your doubt

I'm not anti-AI on-call. The triage speed is real: clustering an alert storm into one incident, pulling the three relevant log lines out of ten thousand, drafting a first hypothesis while you're still finding your glasses. That's a real time-saver, and I use it.

The mistake is letting the tool's confidence stand in for your own judgment, because the study is clear about the cost. You accept a wrong answer, sure. Worse, you lose the reflex to ask whether it's wrong at all. Read-only defaults, approval gates, and forced citations aren't friction bolted on out of paranoia. They're the mechanism that keeps you in the loop the AI would otherwise smooth you out of.

Use AI on-call. Just don't let it talk you out of "I don't know."

What does your team let an AI actually do during an incident, versus only suggest? I'm curious where people are drawing the read-only line right now.