BBH
BIG-Bench Hard · a curated subset of 23 challenging tasks from BIG-Bench where language models previously failed to outperform average humans.
The Frontier
Best score over time · one chart, every benchmark
Full rankings
28 models tested · sorted by score
| # | Model | Score |
|---|---|---|
| 1 | 85.6 | |
| 2 | 83.3 | |
| 3 | 78.7 | |
| 4 | 77.2 | |
| 5 | 75.2 | |
| 6 | 73.1 | |
| 7 | 72.1 | |
| 8 | 71.7 | |
| 9 | 66.8 | |
| 10 | 62.3 | |
| 11 | U Stable Beluga 2 | 59.1 |
| 12 | 48.8 | |
| 13 | 45.9 | |
| 14 | U Nemotron-4 15B | 44.9 |
| 15 | 44.3 | |
| 16 | 41.5 | |
| 17 | 40.0 | |
| 18 | U internlm-20b | 36.7 |
| 19 | U Yi 6B | 29.6 |
| 20 | U vicuna-13b-v1.1 | 24.1 |
| 21 | U Baichuan 2-7B | 22.1 |
| 22 | U MPT-30B | 17.3 |
| 23 | 17.2 | |
| 24 | 16.1 | |
| 25 | 13.6 | |
| 26 | U INTELLECT-1 | 13.1 |
| 27 | U Baichuan1-7B | 10.0 |
| 28 | 4.3 |
Score distribution
Where models cluster
Correlated benchmarks
Pearson r · original research
Benchmarks that track with BBH
Pearson correlation across models scored on both benchmarks. Closer to 1 = strongly predictive.
Frequently asked
About BBH
What does BBH measure?
BIG-Bench Hard · a curated subset of 23 challenging tasks from BIG-Bench where language models previously failed to outperform average humans. 28 AI models have been tested on it. Scores range from 4.3 to 85.6 out of 100.
Which model leads on BBH?
Gemini 1.5 Pro (May 2024) from Google DeepMind leads BBH with a score of 85.6. The median score across 28 tested models is 44.6.
Is BBH saturated?
No · the top score is 85.6 out of 100 (86%). There is still meaningful room for improvement on BBH.
Does BBH predict performance on other benchmarks?
Yes · BBH scores correlate 0.98 with CMMLU across 5 shared models. Models that do well on BBH tend to do well on CMMLU.
How often is BBH data refreshed?
BenchGecko pulls updates daily. New model scores on BBH appear as soon as they are published by Epoch AI or the model provider.
- Category
- Reasoning
- Max score
- 100
- Models
- 28
- Updated
- 2024-12-26
Top on BBH
Gemini 1.5 Pro (May 2024) · 85.6DeepSeek V3 · 83.3Gemini 1.5 Pro (Feb 2024) · 78.7Llama 3.1 405B · 77.2phi-3-medium 14B · 75.2More reasoning benchmarks
Same category · related evaluations