Compare · ModelsLive · 2 picked · head to head
Llama 3.1 8B Instruct vs StarCoder 2 15B
Side by side · benchmarks, pricing, and signals you can act on.
Winner summary
Llama 3.1 8B Instruct wins on 7/8 benchmarks
Llama 3.1 8B Instruct wins 7 of 8 shared benchmarks. Leads in math · general · knowledge.
Category leads
math·Llama 3.1 8B Instructgeneral·Llama 3.1 8B Instructknowledge·Llama 3.1 8B Instructlanguage·Llama 3.1 8B Instructreasoning·Llama 3.1 8B Instruct
Hype vs Reality
Attention vs performance
Llama 3.1 8B Instruct
#275 by perf·#18 by attention
StarCoder 2 15B
#274 by perf·no signal
Best value
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
968.0 pts/$
$0.03/M
StarCoder 2 15B
n/a
no price
Vendor risk
Who is behind the model
Meta AI
$1.87T·Tier 1
U
Unknown
private · undisclosed
Head to head
8 benchmarks · 2 models
Llama 3.1 8B InstructStarCoder 2 15B
GSM8K
Llama 3.1 8B Instruct leads by +24.7
Grade School Math 8K · 8,500 linguistically diverse grade-school math word problems that require multi-step reasoning to solve.
Llama 3.1 8B Instruct
82.4
StarCoder 2 15B
57.7
BBH (HuggingFace)
Llama 3.1 8B Instruct leads by +8.8
Llama 3.1 8B Instruct
29.2
StarCoder 2 15B
20.4
GPQA
Llama 3.1 8B Instruct leads by +6.4
Llama 3.1 8B Instruct
9.5
StarCoder 2 15B
3.1
IFEval
Llama 3.1 8B Instruct leads by +22.8
Llama 3.1 8B Instruct
50.6
StarCoder 2 15B
27.8
MATH Level 5
Llama 3.1 8B Instruct leads by +9.5
Llama 3.1 8B Instruct
15.5
StarCoder 2 15B
6.0
MMLU-PRO
Llama 3.1 8B Instruct leads by +15.8
Llama 3.1 8B Instruct
30.9
StarCoder 2 15B
15.0
MUSR
Llama 3.1 8B Instruct leads by +5.6
Llama 3.1 8B Instruct
8.5
StarCoder 2 15B
2.9
MMLU
StarCoder 2 15B leads by +10.7
Massive Multitask Language Understanding · 57 subjects spanning STEM, humanities, social sciences, and more. The standard benchmark for broad knowledge.
Llama 3.1 8B Instruct
41.5
StarCoder 2 15B
52.1
Full benchmark table
| Benchmark | Llama 3.1 8B Instruct | StarCoder 2 15B |
|---|---|---|
GSM8K Grade School Math 8K · 8,500 linguistically diverse grade-school math word problems that require multi-step reasoning to solve. | 82.4 | 57.7 |
BBH (HuggingFace) | 29.2 | 20.4 |
GPQA | 9.5 | 3.1 |
IFEval | 50.6 | 27.8 |
MATH Level 5 | 15.5 | 6.0 |
MMLU-PRO | 30.9 | 15.0 |
MUSR | 8.5 | 2.9 |
MMLU Massive Multitask Language Understanding · 57 subjects spanning STEM, humanities, social sciences, and more. The standard benchmark for broad knowledge. | 41.5 | 52.1 |
Pricing · per 1M tokens · projected $/mo at 10M tokens
| Model | Input | Output | Context | Projected $/mo |
|---|---|---|---|---|
| $0.02 | $0.03 | 131K tokens (~66 books) | $0.23 | |
U StarCoder 2 15B | — | — | — | — |