The Reliability Iceberg

Problem: You shipped an internal AI tool with Claude Code over a weekend, the demo landed, everyone was impressed, and then you spent the next three months keeping it alive. The build was the easy 20% you could see. The upkeep was the 80% underwater that nobody quoted you.

Quick Win: Before you build an internal AI tool, price the whole iceberg, not just the tip. The reliability iceberg is the gap between the weekend build everyone sees and the months of maintenance nobody plans for: silent failures, messy real-world inputs, prompt drift (when the same prompt quietly gets worse because the model provider updated the model underneath you), and the evals (automated tests that check the AI's output is still correct) you skipped and later regretted. MIT studied real deployments and found 95% of company AI pilots delivered no measurable profit, and the cause was integration and upkeep, not model quality (MIT NANDA, State of AI in Business 2025). This is a build log about the part below the waterline.

The Weekend It Shipped (The Easy 20%)

Here is the honest version, because I lived it.

A Friday night, an idea, and Claude Code. By Sunday I had an internal tool that read from our systems, ran the work a person used to do by hand, and produced output that looked done. Monday morning I demoed it. It worked. People were impressed. Somebody said "ship it."

That whole arc, idea to working demo, is the part everyone talks about when they talk about building with AI. And it is real. Connecting a capable model to a few of your own tools genuinely does produce something convincing in days. The speed is not a lie. The speed is the trap.

Because the demo runs on inputs you chose, in conditions you controlled, on a model version that existed that weekend. None of those three things stay still. The moment the tool meets real work, the iceberg starts to show.

Week 3: The First Silent Failure

The failure you plan for is the loud one: the tool crashes, throws an error, someone gets an alert. That is the easy kind. You see it, you fix it.

The one that gets you is silent. Around week three, the tool started returning answers that were confidently, completely wrong, and it returned them the exact same way it returned the right ones. No error. No red flag. Just a plausible wrong answer, dressed identically to a plausible right one.

This is the defining failure mode of anything built on a large language model. The model is built to produce a convincing next word, not to know when it is wrong. So when it strays, it strays with total confidence, and it can repeat the same mistake across dozens of transactions before a human happens to notice (the research on evolving model behavior). A traditional program that breaks stops. An AI tool that breaks keeps talking.

I found the week-three failure by accident, because someone downstream questioned a number that felt off. That is the scary part. Not that it failed, but that nothing told me it had. The tool had been quietly wrong for days.

The Maintenance Work Nobody Puts In The Quote

Once you have been burned by a silent failure, you see the real shape of the job. It is not a build. It is an operation. Here is what filled those three months, none of which was in the weekend estimate.

Every row on that list is invisible on day one and unavoidable by day thirty. This is the exact pattern MIT's researchers found across 300 real deployments: the tools that failed were "slick enough for demos, brittle in workflows," and they collapsed not because the model was weak but because nobody built the memory, the checks, and the upkeep the real work demanded (Forbes on the MIT report).

The demo eliminates friction. Production is friction. That is the whole gap.

When A Model Update Broke My Prompts

This is the one I did not see coming, and it is the one every technical founder should understand before they ship.

I did not change anything. I did not touch a line of code or a word of a prompt. And one week the tool got measurably worse. Answers that had been reliable started missing. Formatting that had been clean started breaking.

The cause was the model provider. They updated the model behind the same name I was calling, and my prompts, which had been tuned against the old behavior, no longer landed the same way. This has a name: prompt drift, the gradual change in an AI tool's output when the prompt is identical but the model underneath it changed (Comet on prompt drift).

It is not rare. A study of evolving model APIs found that 58.8% of prompt-and-model setups lost accuracy after an update, and of those, more than 70% lost over 5 percentage points, often on small hidden updates the provider never announced (arXiv, (Why) Is My Prompt Getting Worse?). Nearly six in ten of the setups they tested got worse the next time the provider shipped an update nobody was told about.

Think about what that means for an internal tool with no tests. The floor can move under you, at a time you don't choose, and the only way you find out is that the output got worse and someone eventually complained. Which brings us to the part I skipped.

Evals: The Part You Skip And Regret

An eval is an automated test for an AI tool. You give it a set of inputs where you already know the right answer, and it checks whether the tool still produces the right answer. Run it before any change, before any model update, on a schedule, and it tells you the instant quality drops.

I skipped evals on the first build. Of course I did. Nothing had changed yet, the tool worked, and writing tests for output that was currently correct felt like busywork. That is the single most common and most expensive mistake in this whole category.

Without evals, prompt drift is invisible until it is a complaint. A version update is a surprise instead of a caught regression. A silent failure runs for days. The entire reason the week-three failure and the model-update break hurt so much is that I had no automated way to catch either one. The tool had no way to tell me it had gotten worse.

The people who do this well test before they deploy. The research on managing model updates is blunt about it: you keep a regression test suite covering the full range of expected behavior, and you run the whole thing before anything ships, including before you accept a new model version (arXiv, Test Before You Deploy). Evals are not the polish you add at the end. They are the thing that makes the tool a system instead of a demo. Skip them and you are not running a tool, you are hoping.

What It Actually Cost To Keep Alive

Add up the real bill, and the weekend build is a rounding error.

The direct cost is ongoing maintenance. Across the software industry, annual maintenance runs a commonly cited 15 to 25% of the original build cost every year, and can hit 20 to 30% for anything complex (Abbacus Technologies). AI tools sit at the top of that range for one reason ordinary software does not have: the model underneath them keeps changing, so the upkeep never stops. Over five years, total maintenance often equals or exceeds what the software cost to build in the first place.

But the invoice is the smaller cost. The bigger one has no line item: your best engineer's attention. Every hour spent babysitting the tool, catching a silent failure, chasing a model update, patching an edge case, is an hour not spent on the product that makes you money. The tool that saved a person's afternoon can quietly eat your best builder's month.

This is why MIT found what it found. It was never that the models were not good enough. 95% of the pilots delivered no measurable profit, and the divide between them and the 5% that worked was integration, upkeep, and the boring infrastructure of keeping a thing reliable (MIT NANDA, State of AI in Business 2025). The model was the cheap part. It always is.

When To Build It Anyway, And When To Buy The Outcome

None of this means don't build. It means price the iceberg before you commit, and be honest about which side of the line your tool sits on.

The question that matters is not "can we build this." A weekend with Claude Code answers that, and the answer is almost always yes. The real question is "do we want to run this for the next three years." Building commits you to a permanent reliability operation. If that operation is not your core business, you are volunteering your best people for someone else's already-solved problem.

That is the honest trade, and it is the same one we walk through in why so many companies fail at AI department automation and in the case for buying the outcome instead of building the internal tool.

Where This Reasoning Breaks Down

To be fair to the other side, buying the outcome is not free of risk either.

You give up control. When the tool is plumbing you don't own, you can't reach in and change it on a whim. For genuine internal plumbing that is a feature. For anything close to your core, it is a real cost.

Not every tool is worth outsourcing. If the internal tool is where your actual edge lives, handing it to anyone else is a mistake no cost math can justify.

A weekend build can be the right call. Sometimes you need a throwaway tool that runs for a month and then dies. If it never touches a customer, never reports a number that matters, and never needs to survive a model update, the iceberg does not apply. Ship it and move on. The iceberg only matters when the tool has to keep being right.

The mistake is not building. The mistake is building something that has to stay reliable, pricing only the tip, and discovering the other 80% one silent failure at a time.

Frequently Asked Questions

Is Claude Code bad for building internal tools?

No, the opposite. It is so good at the build that it hides how much work comes after. The tool being easy to make is exactly what makes people underprice the maintenance. The weekend build is real and impressive. The mistake is treating the weekend as the whole cost when it is the visible tip of a much larger iceberg of upkeep.

How do I know if my internal AI tool will need heavy maintenance?

Ask one question: does it have to stay correct over time. If a wrong answer would cost you a customer, a compliance problem, or a number you report, it needs evals, drift monitoring, and ongoing upkeep, and the maintenance will be significant. If it is a throwaway that runs for a month and never touches anything that matters, the iceberg does not apply.

What is prompt drift and can I prevent it?

Prompt drift is when your AI tool's output quietly gets worse even though you never changed the prompt, because the model provider updated the model underneath you. You cannot prevent the provider from updating. What you can do is catch it: keep a set of automated tests, evals, that check the output is still correct, and run them on a schedule and before you accept any new model version, so drift shows up as a caught test failure instead of a customer complaint.

---

If you have been burned by an internal AI tool that was easy to build and hard to keep alive, that is the pattern, not bad luck. We build and run internal automation as an operation, with the silent-failure checks, the evals, and the model-update discipline that keep it reliable, so your best engineers stay on the product that earns your revenue instead of babysitting the plumbing. See what we install for companies →