Letting AI into our systems is not exactly risk-free, as this story from Pocket OS clearly shows.

No warning lights were flashing when Claude was asked to do routine work on a staging database. The agent ran into a problem and started looking for a solution. Along the way, it found an API token that gave access to production. Not a token it was explicitly supposed to have, but a token no one knew still existed. The token had no expiration date and broad access, so Claude used it to call Railway’s APIs and ended up deleting the production database AND the backups.

Claude’s explanation?
“I violated every principle I was given.”

It could have gone very wrong. Fortunately, Railway was able to restore the database.

So what should we learn from this story? It was not obvious that the developer was heading into trouble. The work was being done on staging, separate from production. There were backups. The castle had been cleared of every needle Sleeping Beauty could prick herself on. Except for one.

With AI, the small forgotten needles we can prick ourselves on start to matter. The token that gave the agent access to production should have been better scoped and should have expired long before it was forgotten. That is easy to see in hindsight.

Could this have happened without AI? Humans are also unpredictable from time to time, but at least things usually move more slowly when we go off track, and there is a chance to correct course. AI is such a powerful tool that a lot of damage can happen in a very short time.

As Spider-Man taught us: With great power comes great responsibility.

We need to invest just as seriously in cleaning up our systems and access rights as we do in bringing AI into them. Because AI doesn’t just amplify productivity; it also amplifies the weaknesses we forgot to fix.