Is AI Silently Disappearing Your Features?

We have been shipping features with AI for a while now, and somewhere along the way we noticed something strange. Features were not just getting built faster. Some of them were quietly going away.

Not all at once, and never with a commit message that said "remove this feature." A small, less-trafficked capability would work one week, behave oddly the next, and a sprint or two later it was just gone. No one decided to cut it. It simply eroded until a customer or a teammate stumbled onto the gap and asked, "wait, didn't this used to work?"

When we looked at which features were disappearing, the pattern was obvious in hindsight: they were the features that did not have end-to-end tests.

The opposite of scope creep

Everyone knows scope creep, where a product slowly accumulates features nobody asked for. What we were watching was the inverse. Call it feature decay, or silent scope shrink. The surface area of the product was quietly contracting while everyone felt like they were moving fast.

Here is the mechanism. When an AI coding agent works across your codebase, it refactors, renames, rewires data flows, and reshapes components to satisfy the task in front of it. That is exactly what you want it to do. But the agent optimizes for the signal it can see. If a feature has a test that turns red when it breaks, the agent treats that behavior as load-bearing and protects it. If a feature has no test and nothing in the prompt points at it, that feature is invisible. To the model, code with no test and no mention is indistinguishable from dead code.

Code with no test and no mention is, to an AI agent, indistinguishable from dead code. So it gets treated like dead code.

So the hidden, less important, "we'll get to documenting that later" features are precisely the ones at risk. They break in a refactor, the break ships because nothing caught it, and over a few iterations the now-broken path gets simplified away entirely. The feature does not crash loudly. It disappears quietly.

How to stop AI from disappearing your features

The fix is not to slow the AI down. It is to give it the signal it is missing and to verify its work somewhere you can trust. Three things have worked for us.

1. Have end-to-end tests for the features that matter

An end-to-end test is the clearest possible statement of "this behavior is real and it needs to keep working." It does double duty in an AI workflow:

• It gives the agent context. A passing E2E suite is a map of what your product actually does. The agent can read it, respect it, and avoid quietly deleting the paths it covers.
• It shows you the truth. When a feature regresses, a red test tells you immediately, while the change is still fresh and cheap to fix, instead of months later when a customer finds it.

You do not need 100 percent coverage. You need your important and your easy-to-forget features represented by a test, so the AI can see them and so you get a signal the moment one breaks.

2. Do not blindly trust AI, run your tests where you get an audit trail

Having tests is only half of it. If those tests run inconsistently, only on someone's laptop, or with results that vanish into a terminal, you are still flying blind. You want every run recorded, every result attributable, and a history you can scroll back through to answer "when did this stop working, and what changed?"

That is the case for running your suite on an AI-centric test orchestration platform like Testery. Testery runs your end-to-end tests in the cloud at scale and keeps a durable audit trail of every run: what passed, what failed, when, and against which build. When AI is writing a large share of your code, that record is how you keep it honest. You are not taking the agent's word that the feature still works. You have the receipts.

3. Level up your Claude plugins and skills files

The agent only protects what it knows about. So make sure it knows about your testing workflow. Spell it out in your CLAUDE.md and your skills files: which features are critical, that E2E tests are the source of truth, and that test runs go through Testery. The clearer those instructions, the less likely the agent is to wander off and quietly prune something important.

The fastest way to wire this in is the Testery Claude plugin, which turns Claude Code into a Testery teammate. It adds slash commands and skills so Claude can create and monitor test runs, manage environments and schedules, and author playwright-bdd tests, all from natural language. Install it from inside a Claude Code session:

/plugin marketplace add testery/testery-skills
/plugin install testery@testery

Then authenticate and wire up a project:

/testery-onboard      # sign up or log in, persist your API key
/testery-init         # scaffold playwright-bdd and register the project on Testery

Now when Claude works on your codebase, it has a first-class way to run your tests and confirm what is actually still working, instead of guessing. Full setup details live in the Claude plugin docs.

Ship fast, keep what you built

AI coding agents are extraordinary at building. Left unsupervised, they are also happy to quietly un-build the things you forgot to protect. End-to-end tests are how you tell the agent what matters, and an audit trail is how you verify it listened. Do both, and you get the upside of AI velocity without watching your product silently shrink.

Want to stop trusting and start verifying? Start a free trial of Testery and give your AI a test suite it cannot ignore.