Railway's breach report: a build escape, 100,000 workspaces and $740,000 in stolen AI usage

On November 10, cloud hosting platform Railway published its full incident report on the October 20-21 intrusion it first disclosed on October 22. The headline numbers: environment variables for up to 1.85% of workspaces (about 100,000) may have been decrypted, 1,852 users had linked-account tokens reset, and stolen credentials were used to run up an estimated $740,000 in unauthorized AI usage.

The report matters beyond Railway’s customers because it is the most detailed public account yet of a campaign that security firm Wiz has named “Black October”: a run of breaches against developer platforms between early October and early November in which the attackers’ main target was not source code or customer data but AI provider API keys. The attacker in Railway’s case queried the encrypted variables database with regular expressions matching AI provider key names. That is a new kind of loot, and it changes what “secrets hygiene” has to mean for anyone shipping AI features.

The attack chain, step by step

Railway’s timeline (all times UTC) shows an attacker who prepared for days and then moved through five distinct weaknesses:

1. Preparation. Dummy accounts were being created from October 18. Earlier attempts were caught by automated abuse systems.
2. Build isolation escape (October 20, 07:28). After about 40 minutes of probing, the attacker used a previously unknown vulnerability in Railway’s multi-tenant build system to partially escape filesystem isolation, read files on the host VM and obtain a build server credential used to push images.
3. Temporal trust flaw. Railway’s internal Temporal workflow cluster accepted that build credential as a trusted client. Between 13:50 and 15:30 the attacker read the stored histories of 353 builds, which contained build variables (including sealed ones) and GitHub app tokens.
4. Source code theft. Access to cache snapshots gave the attacker a copy of Railway’s internal monorepo, which they used to look for further escalation paths.
5. OIDC misconfiguration and credential chaining (October 21). At 03:26 an erroneous OIDC binding in a cloud environment handed over a short-lived production credential. From there, internal credentials, an SSH key used by the deployment pipeline and a long-lived decryption credential left on an orchestration machine gave access to the variables database and the cloud key management service.

The decryption step is the one with the widest lesson. Railway encrypts each environment’s variables with a per-environment key, wrapped by a root key in cloud KMS. That is textbook envelope encryption. It did not help, because a long-lived credential with permission to call the decrypt operation was sitting on a machine the attacker could reach. Between 04:42 and 07:15 the attacker decrypted variables through KMS in two windows. Encryption at rest protects against a stolen disk; it does nothing against a stolen identity that is allowed to decrypt.

Detection, and a defence fought in parallel

Railway’s first signal came early. At 13:58 on October 20, an alert on anomalous container registry API calls paged the on-call engineer, who traced it to the compromised build credential. Builds were halted at 17:03 and the credential was revoked at 17:28. But the attacker had already read build histories and the monorepo, and returned through a different door overnight.

What followed was roughly 26 hours, from the first probe at 06:50 on October 20 to eviction at 09:25 on October 21, in which engineers patched and isolated paths while the attacker probed others. “This was not a single breach followed by a single fix,” the report says. At 07:14 on October 21 Railway took the blunt option: it disabled every active cloud credential, revoked static credentials, turned off the decryption service and shut down parts of the platform. Hashed lists of potentially exposed keys went to AI providers at 09:25, all user sessions and tokens were invalidated that evening, and builds came back tier by tier through October 22, with Enterprise first and Free and Trial last. New signups stayed paused until November 2.

Running services mostly stayed up, but the cleanup had real costs for customers: revoked credentials broke some services, redeploys were blocked while builds were down, and commits pushed during the pause were never deployed.

Why the attacker wanted AI keys

The regex filtering on AI provider key names is the clearest statement of intent in the report, though Railway notes that some queries also targeted cloud, database and payment credentials. Railway believes the stolen keys were sold to “distributed transfer stations” for model distillation, and it observed them being used for coding agents, AI training pipelines and checks against training scoring systems. It has not established whether the original attacker also ran the resale services.

Wiz’s research supplies the market side. It links stolen keys from the campaign to two resale operations, one taking payment in USDT, and found that one reseller’s site hosted a “distillation detection” page, which Wiz reads as an attempt to dodge provider account bans. Its two Bedrock case studies show how fast a stolen key is spent: about 800,000 calls in 30 hours in one case, about three quarters of them to Opus models, and 1.7 million calls in 13 hours in the other, with the first unauthorized call arriving within 30 minutes of the credential being taken. Wiz estimates those two keys alone ran up more than $100,000 at list price.

Railway is not an isolated case. Wiz’s list of affected platforms includes Fireworks AI, Lightning AI, OpenCode, Warp, Northflank, Zeabur (which says it will shut down) and others, with root causes ranging from leaked credentials to an SGLang remote code execution flaw. We covered Fireworks AI’s second breach on November 10. The common thread is that these are platforms where developers park the API keys their apps use, which makes them a single place to harvest keys from thousands of accounts at once.

Who pays for the $740,000

The report breaks down refunds by provider, and the policies differ:

Provider	Refund position
AWS Bedrock, Google Gemini	Existing fraud refund policies apply; refunds initiated
OpenAI, SpaceXSI	Reviewed case by case through a shared bridge
Anthropic	Discussions ongoing; Railway is covering breached-key costs meanwhile
OpenRouter	No-refund policy enforced; Railway covering the costs

Railway says it will cover any charges a provider declines to refund. That is generous, and it also shows a gap: a customer whose key leaked through a platform breach is, by default, liable for usage they never made, and recovery depends on each provider’s fraud policy. The provider-side notification at 09:25 on October 21, sent as hashed key lists, is the mechanism that limited the bill. Providers continued revoking keys into October 23.

What changes, and what to do

Railway’s remediation list maps onto each link in the chain: every build now runs on its own ephemeral VM on top of the original isolation, build credentials can push only one specific image, the shared systems that were exploited are retired, service accounts were reviewed for least privilege, credentials are short-lived and just-in-time where possible, every encrypt and decrypt call is audit-logged, and only the secrets service can reach the keys that unwrap data encryption keys. The CI pipeline was rebuilt with 2FA approval, biometrics for critical areas and SIEM alerting. Railway has filed a complaint with the FBI’s IC3, engaged a forensics firm whose investigation is ongoing, and says it will publish indicators of compromise after law enforcement signs off.

For teams with AI keys on any hosting platform, the practical steps from this incident are:

• Rotate any AI provider key that lived in a Railway environment variable during October, and report unexpected charges through Railway’s form linked in the report.
• Set hard spend limits and per-key budgets at the provider. A stolen key with no cap is a blank cheque; Wiz’s cases show it can be drained in hours.
• Use one key per service and environment, so a leak can be revoked without taking everything down and so usage anomalies point to a source.
• Alert on usage, not just access. A sudden jump in calls, a new model family or traffic at odd hours is often the first visible sign.
• Prefer short-lived credentials (workload identity, OIDC federation) over static keys where the provider supports them, while auditing those OIDC bindings, since one misconfigured binding was a key step here.

The open question is the rest of the campaign. Wiz’s list was still growing on November 10, the forensics on Railway are unfinished, and none of the affected platforms has yet publicly tied the intrusions to a named group.