Why such simple tasks?

Exact arithmetic and character-level tasks are sensitive to precision loss and have no grading ambiguity, which makes differences between hosts easy to see.

Which quantization does each host use?

Where the provider reports it, it is shown next to the host.

Gecko Tests · Same Model, Different HostFirst run in progress

Same Model, Different Host · Who Serves It Right?

Open models are sold by many providers, some quantized or misconfigured. We send the same 30 exact-answer tasks to every host of a model and compare. A host far below the best one is flagged.

In plain wordsOpen AI models are sold by many companies. They should all give the same answers, but some serve a cheaper, weaker version. We give every seller the same simple tasks and compare scores. A pink bar = that seller does worse.

First run in progress.

How it works

30 tasks with one exact answer each: multiplication, letter counting, string reversal and sorting. They are generated from a fixed seed and the answers are computed by code, so the key cannot be wrong.
Each request is pinned to one provider with fallbacks off, temperature 0, lowest reasoning setting.
Score = share of exact answers. A host is flagged when it scores more than 15 points below the best host of the same model.
Models: open-weight models from tracked labs with at least four hosts. Re-run monthly.

Open dataset · v1

30 items, every model answer and every score are public under CC BY 4.0.

Download JSON

How to cite

BenchGecko Gecko Tests, Same Model, Different Host v1. BenchGecko. https://benchgecko.ai/gecko-tests/same-model-different-host

BenchGecko measures this data itself: free to reuse under CC BY 4.0 with the credit "Source: BenchGecko" and a link. JSON · llms.txt · MCP

Frequently Asked Questions

No. Lower precision, a different inference engine or a wrong chat template can all cost accuracy. The flag says the same model behaves measurably worse there.