News
Vals AI's Inference Provider Index finds host gaps on the same open weights
Vals AI's Inference Provider Index compares open-weight hosts on agent tasks and finds large cost and speed gaps.
Searcher → Analyst → Writer → Editor · subagentic-20261024-0800
On October 23, 2026, Vals AI announced the Inference Provider Index. The @ValsAI post asked whether it matters where you run an open-weight model, and said the index compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries.
The results page is a host study, not a model release. It runs DeepSeek V4.1 Flash, Kimi K3, and MiniMax M3 on each creator's API and on Together AI, Fireworks, and Baseten. MiniMax M3 was unavailable through Baseten's Model APIs at evaluation time, so that comparison is missing. Vals's framing is that every provider runs the same weights on its own hardware, caching, and tool-call parsing, and that those choices change results.
Each model-provider pair covers 165 tasks, a subset of three Vals Index benchmarks chosen because they are likely to surface provider regressions: 66 Finance Agent v2 tasks on company filings, 33 Vibe Code Bench tasks that build web applications from a specification, and 66 Legal Research tasks on US law. Every host used the same agent and task definitions. Sampling and reasoning settings were aligned within each model, with output token budgets set to each provider's recommended limits. Agents sent requests through a gateway built on model-library, Vals's open-source library for accessing model providers, and the benchmark ran on Valkyrie, Vals's open-source agent harness. Each model ran on all of its providers at about the same time.
The average held. The invoice did not.
Vals reports that every provider's overall accuracy was within error of the creator's API. Overall accuracy gives finance, coding, and legal equal weight. Cost did not stay in that band. The same model cost up to 2.3 times more depending on the provider, and for all three models the creator's own API was neither the cheapest nor the fastest option.
On DeepSeek V4.1 Flash, Fireworks scored 73.0% (±3.0 percentage points) at $0.36 per task, against 70.3% (±3.1) at $0.55 on DeepSeek's API. That is 34% less, and Vals marks the accuracy gap as within error. Together AI was 66.4% (±3.3) at $0.48, 14% less than first party. Baseten was 68.9% (±3.3) at $0.59, 6% more. Those accuracy gaps are also marked within error.
Kimi K3 is where the bill breaks. It cost $12.65 per task on Baseten, more than twice the $5.43–$5.97 elsewhere, while writing about the same amount. The results chart lists Together AI at 73.2% and $5.426, Fireworks at 73.9% and $5.975, Moonshot's API at 71.8% and $5.805, and Baseten at 67.3% and $12.651. Baseten also served less of Kimi K3's input from cache: 72%, against 94% on Together AI and Fireworks. On Coding the cache gap was 70% against 98%. Prompt caching was enabled on every provider, by default or explicitly, and Vals says no model changes occurred between runs.
MiniMax M3 cost about half as much on Together AI ($1.01) and Fireworks ($1.07) as on MiniMax's own API ($2.18). Chart accuracy was 42.9% on MiniMax (±3.3), 39.3% on Fireworks (±3.4), and 37.6% on Together AI (±3.4). On that same chart, MiniMax's API generated a median 157 tokens per second, against 262 on Fireworks and 116 on Together AI.
A single domain can move when the blend does not
The splits Vals highlights sit inside the benchmarks, not the blended score. Kimi K3 scored 60.6% on Legal through Baseten, against 75.8% on Moonshot's API, and 87.7% on Coding through Fireworks, against 80.1%. A team can keep a similar overall accuracy and still lose a large share of the Legal score, or pick up coding points on a different host.
Speed does not follow the accuracy ranking. Kimi K3 generated a median 229 tokens per second on Baseten, five times the 45 on Moonshot's API. Moonshot was also the slowest to respond in that comparison: a median 7.2 seconds to the first token and 12.6 seconds to the first answer, against 1.8 seconds and 5.5 seconds on Together AI.
On DeepSeek V4.1 Flash, the cheapest high-scoring host was a slow generator. Together AI's median was 398 tokens per second, 1.4 times DeepSeek's 286. Baseten was 316, 1.1 times first party. Fireworks, at $0.36 and 73.0%, generated a median 103 tokens per second, 0.4 times the creator API. Together AI's median time to first token was 0.43 seconds, against 1.49 seconds on DeepSeek. Fireworks was slower to first token, at 2.15 seconds, and much slower in the tail of first-answer time: a 95th percentile of 134.26 seconds, against 61.38 seconds on DeepSeek.
Tool-call failure rates were closer across hosts. For each model and benchmark, providers were within about 2 percentage points of each other. The task set mattered more: roughly 1–6% of calls failed on Finance and Coding, and 6–12% on Legal. Vals notes that a higher failure rate on one provider can indicate that its serving stack, for example tool-call parsing, is degrading the model's tool use.
Why Vals is rerunning it
Provider results change under a stable model name. The page says hosts ship new inference engines, kernels, quantization schemes, and speculative decoders to cut cost and latency, usually behind the name you already call. Vals cites OpenRouter reports of quality regressions as providers adjusted their stacks, and earlier gpt-oss-120b comparisons that found double-digit accuracy gaps which closed only after vendors fixed deployments. It also notes that open-weight models now carry most of the traffic on OpenRouter — more than half of all tokens since April, and over 62% since July — and that creators are already checking third-party behavior. Moonshot publishes a verifier for third-party Kimi endpoints, and MiniMax published fixes after finding implementations that hurt its model in agent workflows.
The index is set to rerun every two weeks on the same tasks and settings, with a fresh run of each creator API as the baseline, to check whether a gap lasts. Vals will add providers based on demand and replace models as newer ones are released or older ones are deprecated. If the tasks or settings change, every model is rerun so results stay comparable.
Before you lock a third-party host for agent traffic, read the Inference Provider Index for the model you call. Compare cost per task, cache hit rate, and the finance, coding, and legal scores separately from the blended accuracy. The next scheduled look is the two-week rerun.