OpenAI's Models Jumped From 10.9% to 72.1% on COBOL in One Generation

OpenAI announced on October 19 that its latest models deliver step-change performance on low-resource and legacy programming languages, and the before-and-after numbers on its newly released 20-language benchmark make the case bluntly: COBOL rose from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%, per OpenAI’s announcement and coverage of the release. The Astra-class model exceeded 83% on each of the other 19 languages, with reports of up to 97.9% accuracy on software modernization tasks. The target customer is unmistakable: banks, insurers, and governments still running core operations on mainframe COBOL.

Why legacy languages were the hardest test

The gap existed for a structural reason, not a modeling one. Pretraining corpora are dominated by modern languages, so a model sees orders of magnitude more Python than COBOL, and the COBOL that exists online is disproportionately old, dialect-dependent (IBM Enterprise COBOL versus Micro Focus versus GnuCOBOL differ meaningfully), and embedded in mainframe conventions like CICS and JCL that rarely appear in public code. Evaluation was equally broken: there was no shared benchmark for this space, which is why OpenAI shipped one alongside the results, covering 20 languages. Independent data confirms the gap was real rather than measured: research from the Factory team has shown the best models solve only about 42.5% of legacy-language tasks against over 70% on modern benchmarks, so OpenAI’s numbers, while self-reported on its own benchmark, are movement in the direction everyone’s instrumentation already indicated.

The market this targets is enormous and stuck

The modernization backlog is one of the enterprise software industry’s oldest unsolved problems: decades of attempts to migrate mainframe COBOL have failed expensively, because the business logic embedded in those systems is undocumented, the original authors are retired, and a translation bug in a payments core is a catastrophic event. This is the same demand Anthropic targeted with Claude Code automating million-line migrations, and the competitive read is straightforward: whoever solves understanding and translation for these stacks unlocks maintenance contracts worth far more than model subscriptions. Two sobering notes belong next to the enthusiasm. A 72.1% task-completion rate on COBOL still means roughly one in four tasks failing, an unacceptable rate for autonomous migration of production financial systems; and this week’s Postgres 19 delay showed what happens when AI review finds more defects than humans can triage, a dynamic that compounds when the code in question has no surviving test suites.

What to watch

Three things will tell you whether this is a real capability shift or a benchmark-shaped one. First, independent reproduction on non-OpenAI evaluation sets, particularly compilation-and-run tests on real dialects rather than sanitized tasks; the related COBOL-Coder research line, which achieved 73.95% compilation success on COBOLEval, suggests the metric definitions matter enormously. Second, whether OpenAI productizes this for the mainframe world (migration tooling with verification harnesses, not just API access), because the buyers here purchase outcomes, not tokens. Third, the safety-critical angle: COBOL runs payment rails, tax systems, and benefits infrastructure, and the industry’s recent lesson, from AI code review flooding Postgres with findings faster than humans could fix them, is that verification capacity, not generation capacity, is the bottleneck. A model that writes COBOL 7x better is only valuable alongside review machinery that can check COBOL 7x better, and nobody has announced that yet.