Which model leads on WeirdML?

GPT-6 Astra from OpenAI leads WeirdML with a score of 93.6. The median score across 115 tested models is 45.4.

Is WeirdML saturated?

No · the top score is 93.6 out of 100 (94%). There is still meaningful room for improvement on WeirdML.

Does WeirdML predict performance on other benchmarks?

Yes · WeirdML scores correlate 0.99 with Osworld 2 0 across 6 shared models. Models that do well on WeirdML tend to do well on Osworld 2 0.

How often is WeirdML data refreshed?

BenchGecko pulls updates daily. New model scores on WeirdML appear as soon as they are published by Epoch AI or the model provider.

Benchmark · CodeSettled

WeirdML

Name: WeirdML Benchmark
Creator: BenchGecko
License: https://creativecommons.org/licenses/by/4.0/

WeirdML · tests models on unusual and adversarial machine learning tasks that require creative problem-solving beyond standard patterns.

Updated 2026-09-04

Models tested

115

Top score

93.6

GPT-6 Astra

Median

45.4

min 1.7

Top-5 spread

σ 1.4

Settled

The Frontier

Best score over time · one chart, every benchmark

Chart type

Frontier on WeirdML rose from 25.1 to 93.6 in 22 months · +68.4 points · latest leader GPT-6 Astra from OpenAI.

Pink dots = frontier records · 15 totalClick to open model page

Full rankings

115 models tested · sorted by score

#	Model	Score	Price
1	GPT-6 Astra· OpenAI	93.6	$10.00
2	Claude Fable 5.1· Anthropic	92.9	$10.00
3	Claude Fable 5· Anthropic	91.9	$10.00
4	Claude Opus 5· Anthropic	91.8	$5.00
5	GPT-5.6 Sol· OpenAI	89.4	$2.00
6	Claude Opus 4.8· Anthropic	82.9	$5.00
7	Kimi K3· moonshotai	82.6	$0.67
8	GPT-5.3-Codex· OpenAI	79.3	$1.75
9	GPT-5.6 Terra· OpenAI	78.3	$2.00
10	Claude Opus 4.6· Anthropic	78.0	$5.00
11	GPT-5.4· OpenAI	77.7	$2.50
12	Claude Opus 4.7· Anthropic	76.4	$5.00
13	GLM 5.3· z-ai	75.4	$1.40
14	GPT-5.2· OpenAI	72.2	$1.75
15	Gemini 3.1 Pro Preview· Google DeepMind	72.1	$2.00
16	GLM 5.2· z-ai	70.1	$1.00
17	Gemini 3 Pro· Google DeepMind	69.9	n/a
18	Claude Sonnet 5· Anthropic	68.8	$2.00
19	Grok 4.6· xAI	67.3	$2.00
20	DeepSeek V4 Pro 0813· DeepSeek	66.2	$0.55
21	Claude Sonnet 4.6· Anthropic	66.1	$3.00
22	Claude Opus 4.5· Anthropic	63.7	$5.00
23	DeepSeek V4 Flash 0731· DeepSeek	63.0	$0.02
24	Gemini 3.5 Flash· Google DeepMind	62.6	$1.50
25	Gemini 3 Flash Preview· Google DeepMind	61.6	$0.50
26	GPT-5.6 Luna· OpenAI	60.9	$0.20
27	GPT-5.1· OpenAI	60.8	$1.25
28	GPT-5· OpenAI	60.7	$1.25
29	GPT-5 Pro· OpenAI	60.4	$15.00
30	GPT-5.4 Mini· OpenAI	60.3	$0.75
31	o3 Pro· OpenAI	58.2	$20.00
32	GPT-5.4 Pro· OpenAI	57.4	$30.00
33	GLM 5.1· z-ai	57.1	$0.97
34	Gemini 3.6 Flash· Google DeepMind	56.1	$0.75
35	Kimi K2.6· moonshotai	55.9	$0.95
36	Kimi K2.7 Code· moonshotai	54.1	$0.61
37	Gemini 2.5 Pro· Google DeepMind	54.0	$1.25
38	GPT-5 Mini· OpenAI	52.7	$0.25
39	o4 Mini· OpenAI	52.6	$1.10
40	o3· OpenAI	52.4	$2.00
41	Gemma 4 31B· Google DeepMind	52.3	$0.09
42	Grok 4.20· xAI	52.3	$1.25
43	Gemini 3.1 Flash Lite· Google DeepMind	52.2	$0.25
44	Grok 4.3· xAI	49.9	$1.25
45	GPT-5.4 Nano· OpenAI	49.2	$0.20
46	DeepSeek V4 Pro· DeepSeek	48.9	$0.21
47	GLM 5· z-ai	48.2	$0.60
48	gpt-oss-120b· OpenAI	48.2	$0.04
49	Claude Sonnet 4.5· Anthropic	47.7	$3.00
50	o1-preview· OpenAI	47.6	n/a
51	Grok 4.5· xAI	46.4	$2.00
52	Claude Sonnet 4· Anthropic	46.1	$3.00
53	o1· OpenAI	46.1	$15.00
54	Claude Opus 4.1· Anthropic	45.9	$15.00
55	Grok 4· xAI	45.7	n/a
56	DeepSeek V4 Flash· DeepSeek	45.6	$0.03
57	Kimi K2.5· moonshotai	45.6	$0.45
58	Claude Haiku 4.5· Anthropic	45.4	$1.00
59	Claude Opus 4· Anthropic	43.7	$15.00
60	Mistral Medium 3.5· Mistral AI	43.7	$1.50
61	o3 Mini· OpenAI	43.7	$1.10
62	Nemotron 3 Ultra· NVIDIA	43.5	$0.50
63	Grok 4 Fast· xAI	42.9	n/a
64	Kimi K2 Thinking· moonshotai	42.8	$0.60
65	Grok-3 mini· xAI	42.6	n/a
66	R1 0528· DeepSeek	41.6	$0.50
67	Qwen3 235B A22B Thinking 2507· Alibaba Qwen	41.0	$0.23
68	Gemini 2.5 Flash· Google DeepMind	41.0	$0.30
69	gpt-oss-20b· OpenAI	40.9	$0.02
70	DeepSeek V3.2 Exp· DeepSeek	39.5	$0.27
71	GPT-4.5· OpenAI	39.4	n/a
72	Kimi K2 0711· moonshotai	39.4	$0.57
73	GPT-4.1· OpenAI	39.0	$2.00
74	Gemini 3.5 Flash Lite· Google DeepMind	39.0	$0.30
75	Qwen3 235B A22B Instruct 2507· Alibaba Qwen	38.7	$0.09
76	DeepSeek V3.1· DeepSeek	38.4	$0.25
77	GPT-5 Nano· OpenAI	38.1	$0.05
78	GPT-4.1 Mini· OpenAI	37.6	$0.40
79	Qwen3 235B A22B· Alibaba Qwen	37.3	$0.46
80	Grok 3· xAI	37.2	n/a
81	MiniMax M2.7· minimax	37.0	$0.21
82	R1· DeepSeek	36.5	$0.70
83	o1-mini· OpenAI	36.3	n/a
84	DeepSeek V3· DeepSeek	36.1	$0.26
85	Gemini 2.5 Flash Lite· Google DeepMind	35.2	$0.10
86	Gemma 4 26B A4B · Google DeepMind	35.2	$0.07
87	Qwen3.6 35B A3B· Alibaba Qwen	34.5	$0.15
88	Claude 3.5 Sonnet· Anthropic	31.0	n/a
89	Claude 3.5 Haiku· Anthropic	30.7	n/a
90	Qwen3 30B A3B· Alibaba Qwen	29.8	$0.12
91	Gemini 2.0 Flash· Google DeepMind	25.8	$0.10
92	Gemini 2.0 Flash (Dec 2024)· Google DeepMind	25.8	n/a
93	GPT-4o (2024-11-20)· OpenAI	25.1	$2.50
94	Gemini 1.5 Flash (May 2024)· Google DeepMind	24.9	n/a
95	Llama 4 Maverick· Meta	24.5	$0.19
96	Grok-2 (Dec 2024)· xAI	22.2	n/a
97	Gemini 1.5 Pro (Feb 2024)· Google DeepMind	22.2	n/a
98	Gemini 1.5 Pro (May 2024)· Google DeepMind	22.2	n/a
99	Llama 3.1 405B· Meta	21.4	n/a
100	Claude 3 Opus· Anthropic	19.2	n/a
101	GPT-4.1 Nano· OpenAI	19.0	$0.10
102	Qwen2.5 72B Instruct· Alibaba Qwen	16.0	$0.36
103	Llama 3.3 70B Instruct· Meta	14.4	$0.10
104	Llama 3.3 70B Instruct (free)· Meta	14.4	$0.00
105	GPT-4 Turbo· OpenAI	12.4	$10.00
106	GPT-4o-mini· OpenAI	11.8	$0.15
107	GPT-4o-mini (2024-07-18)· OpenAI	11.8	$0.15
108	Qwen2-72B· Alibaba Qwen	11.3	n/a
109	Claude 3 Sonnet· Anthropic	10.2	n/a
110	Claude 3 Haiku· Anthropic	9.8	$0.25
111	Llama 3.1 70B Instruct· Meta	9.0	$0.40
112	Claude 2.1· Anthropic	7.1	n/a
113	GPT-3.5 Turbo (older v0613)· OpenAI	3.5	$1.00
114	Mixtral 8x22B Instruct· Mistral AI	3.2	$2.00
115	Llama 3.1 8B Instruct· Meta	1.7	$0.02