36% of ICML 2026 Papers Fail Hugging Face Reproducibility Audit

Hugging Face has published the complete results of its ICML 2026 Open Reproduction Challenge, a community-driven evaluation of all 2,214 papers accepted to the main track of the 43rd International Conference on Machine Learning. Utilizing the Hugging Face Evaluate library and a dedicated cluster of 1,000 NVIDIA H200 GPUs, the initiative attempted independent verification of every claim.

The final data shows 64% of submissions were fully reproducible, meaning results matched the published findings within a 5% margin of error using the provided code and data. The verified code and logs are now indexed in the public ICML 2026 Reproducibility Map.

Artifacts and Environment Constraints

Another 21% of papers were categorized as partially reproducible. These submissions required significant debugging by the Hugging Face team or produced inconsistent metrics during testing. The remaining 15% were entirely not reproducible due to missing code, proprietary datasets, or excessive compute requirements.

While 88% of accepted papers included a link to a code repository on platforms like GitHub, only 42% provided a working “one-click” environment. Submissions lacking a Docker container or a specific Conda environment routinely failed to execute without manual intervention.

The Compute and Data Bottleneck

Replicating the top 5% of State of the Art (SOTA) papers required an average of 4,096 GPU hours on H100-equivalent hardware. This extreme resource requirement effectively locks most academic labs out of verifying frontier research, pushing a growing portion of AI compute budgets exclusively toward evaluation.

Data access presented an even harder limit. Papers relying on closed corporate or restricted medical datasets achieved a reproduction success rate below 12%, even when researchers published the complete model architecture.

Hosting infrastructure directly correlated with verification success. Papers that hosted their models and datasets directly on the Hugging Face Hub during submission were 3.5 times more likely to pass independent review compared to those using personal or university web servers.

Changing Publication Standards

Based on preliminary data shared at the Vienna conference in July, ICML 2027 program chairs announced a new mandate requiring “Artifact Appendices” for all future submissions. The policy forces researchers to document specific metadata for their hardware and software environments.

Industrial researchers have criticized the focus on pure reproducibility, arguing the metric penalizes large-scale foundation models where single training runs exceed $500,000. Representatives from major labs propose that the community should accept verification of methodology in place of strict output reproduction for high-compute research.

If you submit research to upcoming machine learning conferences, expect strict environment validation requirements. Package your model evaluations in containerized environments and ensure your external dependencies are durably hosted before the review period begins.