API Keys And App Secrets May Have Been Leaked At Railway – How Can Teams Protect Their Apps On Cloud Platforms?

Environment variables are the sticky note on the monitor of app development. They’re convenient, everywhere and hold the keys to the entire house. These external values keep sensitive data out of codebase repositories, which is why many developers use them to store credentials for databases, payment gateways, email tools and cloud storage.

Cloud platform Railway, which developers use to host their apps, recently detected suspicious activity in the systems storing these user variables. The platform says an unauthorised party may have accessed encrypted variables and, for some of them, obtained the decrypted contents.

In a preliminary statement, Railway said that up to 1.85% of workspaces, about 100,000, were potentially exposed. Because the team couldn’t pinpoint which specific variables were decrypted, the company advised anyone who signed up before 21 October to rotate their keys.

Live applications stayed online during the incident, although new builds and code releases were paused while engineers investigated. By 26 October, the company said the issue was resolved for existing users and that builds had been restored.

Why Credentials End Up In Environment Variables

The appeal comes down to convenience. Environment variables keep configuration data separate from core code, which lets teams tweak settings for different environments without altering any of the codebase. On Railway, developers set them in the dashboard and assign them to specific services. The application then pulls them in as standard variables at runtime.

This incident hinges on a key difference in data security. Encrypted variables are still protected at rest, but decrypted keys can be used by whoever holds them. That makes the mention of decrypted contents the most serious line in Railway’s statement.

Hosting variables on a platform also changes who holds what. The provider’s systems then hold the keys to a team’s databases, payment accounts and cloud storage, so an incident at the provider reaches every service those credentials unlock.

We asked experts: how can teams protect the API keys and app secrets their apps rely on, and what are the alternatives to storing them in environment variables?

Our Experts

• Callum Duncan, Co-Founder and Senior Penetration Tester, Sencode
• Bapuji Koraganti, Principal Engineer, Independent Identity Security Researcher
• Lisa Loud, CEO, DEPH Labs
• Kevin M. Coles, Fractional CIO, Coles Technical Group
• Vladimir Beskorovainyi, Enterprise AI Architect and CTO, Besk Tech
• Devarshan Patel, Founder, Build with DEV

Callum Duncan, Co-Founder and Senior Penetration Tester, Sencode

“Good secret management requires encrypting environment variables at rest. Services like Amazon’s KMS let users fetch and decrypt only the secrets they need, so even if someone else reads them, they can’t use them.

“Setup for these systems is trivial and has been considered best practice for many years. When hosting providers don’t allow this, Full Disk Encryption and runtime decryption of secrets mean these can only be accessed through memory or a full device compromise. Secret Management is a solved problem. If a hosting provider has a breach which allows environment variables to be read, they ought to be benign dynamic strings which are used during runtime but aren’t secrets.”

Bapuji Koraganti, Principal Engineer, Independent Identity Security Researcher

“Railway’s incident forces an uncomfortable admission: environment variables are plaintext secrets waiting for the wrong process to find them. Encryption at rest offers limited comfort when the platform storing the ciphertext also holds the means to decrypt it.

“And the risk is growing. Teams now feed those same .env files to AI coding assistants and wire API keys straight into autonomous agents. Anything an agent can read, a prompt injection can make it leak. A long-lived payment or cloud key in an agent’s context is one malicious instruction away from a breach.

“The fix is to stop treating secrets as things apps hold, and start treating access as something they’re granted. With workload identity federation, an app or agent proves who it is with a short-lived token and swaps it for credentials that expire in minutes. If nothing static sits in a file, there’s nothing to steal.

“For AI agents, go further. Route their calls through a credential broker that attaches keys outside the agent, so it can use an API without ever seeing the secret. Issue tokens scoped to one user and one task, never the master key.

“Keys you truly can’t eliminate belong in a secrets manager, encrypted with keys you control and fetched at runtime with minimum permissions.

“Then rehearse rotation. If rotating a critical payment key takes your team a week, you don’t have a security strategy. You have a prayer. Affected Railway customers should rotate today.”

Lisa Loud, CEO, DEPH Labs

“A key management system is best security practice for storing secrets that are required for coding. Environment variables are typically unencrypted and can easily be exposed by searching memory. With AI the situation is even more vulnerable because AI often vibecodes environment variables by default. It takes a security-conscious developer to explicitly instruct AI coding otherwise.”

Kevin M. Coles, Fractional CIO, Coles Technical Group

“Environment variables were never meant to be a vault. They’re convenient, which is exactly why every credential a company has ends up in one place, readable by anything that runs. When the host gets breached, that’s the whole kingdom in one file.

“First, use a real secrets manager: AWS Secrets Manager, Azure Key Vault, Google Secret Manager or HashiCorp Vault. The app pulls the secret at runtime using its own identity, and every access is logged.

“Second, get rid of long-lived keys wherever you can. Workload identity and OIDC let your app prove who it is to your cloud provider without a stored key at all. Nothing stored means nothing to steal.

“Third, scope and rotate. A payment key that can only create charges and expires in 30 days is a bad day. A full-access key that hasn’t changed in three years is a catastrophe.

“Fourth, assume your hosting provider will have a bad day. When a host announces a breach, you should be able to rotate every secret within an hour. Most teams can’t, because nobody knows where every key is used.

“The hardest part isn’t the tools. It’s that nobody owns the list.”

Vladimir Beskorovainyi, Enterprise AI Architect and CTO, Besk Tech

“Encryption at rest protects the store, not the secret. Any process running as the application can read whatever the application can read, so a platform compromise and an application compromise end in the same place.

“The useful question is not where a key lives. It is how long it stays valid and how far it can reach. A credential that never expires turns one bad day into a door that stays open until somebody happens to notice.

“The strongest alternative is to stop holding static secrets at all. Workload identity lets a running service prove what it is to the provider and receive a short-lived token at runtime, so there is nothing durable sitting in the environment to steal. Most major cloud and database providers support this today.

“Where a static secret is genuinely unavoidable, three constraints do most of the work. Scope it to one action rather than one account. Give it an expiry and rotate on a schedule rather than after an incident, because rotation that has never run does not work. And close outbound traffic by default with destinations named, since a stolen key is only valuable if it can reach the service it unlocks.

“Then assume disclosure anyway and shorten the window. Alert on credential use from an unfamiliar network or outside normal patterns. You cannot reliably stop a secret escaping. You can make it expire before it is worth anything.”

Devarshan Patel, Founder, Build with DEV

“I ship products fast, and the stage where I see API keys leak most is the prototype. Three rules I hold every build to:

“1. Keys never live in the repo or the client bundle. They sit in the hosting platform’s secret manager and get injected at runtime, with a separate set per environment.

“2. Every key is scoped to one service with the least permissions it needs, so a leak has a small blast radius.

“3. Secret scanning runs in pre-commit hooks and CI, and keys rotate on a schedule and immediately when anyone leaves the project.

“The common mistake is treating security as a cleanup task after launch. Setting this up on day one costs an hour and saves a breach.”