A Dermatologist's 10-Year-Old CNN Beat GPT-6 Astra and Fable 5.1 at Skin Cancer

Dr Magnus Lynch, a consultant dermatologist at Guy’s Hospital, published a benchmark study in October 2026 that lands harder than its modest presentation suggests: a fine-tuned Inception v3, the Google architecture from 2015, trained on 68,011 dermoscopic images from the ISIC Archive on a single H100, outperformed GPT-6 Astra and Claude Fable 5.1, the most capable models available as of October, at diagnosing 200 randomly drawn, biopsy-proven skin lesions. The CNN got 125 of 200 first-choice diagnoses right (62%); the LLMs managed 84 and 85 (42% each). On melanoma, the diagnosis that matters most: the CNN caught 13 of 30, GPT-6 Astra 8, Fable 5.1 11.

The experiment, done properly

The methodology is what makes the result citable rather than anecdotal. The test set came from MILK10k, a public ISIC dataset, restricted to lesions that were actually biopsied, all hard cases: 100 unsafe-to-discharge (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe-to-discharge. The LLMs received a clinical photo plus a dermoscopic image and picked from six diagnoses; the author’s scoring was scripted against histology; data, prompts, and full model replies are publicly downloadable. Lynch, who ran the analysis using Claude Code, is explicit about the caveats: 30 melanomas is a small sample with wide uncertainty, each LLM case ran once, every lesion was hard enough to biopsy (the CNN’s AUC drops from 0.96 to 0.86 on this set versus the general population), training contamination from the public dataset is possible, and genuine state-of-the-art medical systems beat this deliberately baseline CNN.

The error patterns are as interesting as the score

The three models failed differently, and the differences are diagnostic of architecture. GPT-6 Astra over-called basal cell carcinoma: 75 BCC diagnoses, only 25 true. Fable 5.1 over-called actinic keratosis: 53 calls, 20 true. The CNN’s misses skewed toward calling melanomas benign, the error pattern classical CNNs are known for. Both LLMs did excel at the safety-critical question as a patient would ask it: asked “should I see a doctor,” they appropriately referred 99 and 98 of the 100 melanoma patients. And the threshold analysis cuts both ways: at its preset operating point the CNN kept 77 of the 200 lesions for review against the LLMs’ 96-97, but matched to LLM thresholds it kept 98-99 while still discharging 18 of 30 melanomas at its default. Specialization wins accuracy; generality wins calibrated caution, at least as prompted.

The finding generalizes, and that is the point

This is not the first such result, but it is the cleanest demonstration during the exact month when the opposite assumption, frontier generality subsumes everything, dominated the discourse, from agent harnesses making five small models match a frontier one to formal verification at assembly scale. The consistent lesson of October’s results: task-specific systems, small models, narrow verifiers, purpose-built pipelines, keep beating frontier generalists inside their domain, at a fraction of the cost, while the frontier wins on breadth, language, and calibrated hedging. What to watch: whether specialized medical AI teams treat this as validation to press into deployment with the LLMs as a second opinion, since the patient-framed referral numbers suggest the right architecture is CNN for triage plus LLM for communication; and whether LLM vendors start routing sub-tasks to fine-tuned specialists, because the 20-point gap on their own images is not a prompt-engineering problem.