Local models beat Jev’s score, but not its speed.
Nimble 9B scores 95% to Jev’s 90%. The fastest local model, Tev1 0.8B, takes 219 ms to Jev’s 210 ms.
29 Sept 2026 · 20 Decision Bench cases · Jev 1.13 vs five local model families on an M3 Pro
Accuracy vs response time
● Best local trade-offs● Other local runs○ Jev Cloud (reference)■ Supports images
Loading measured results…
Up is more accurate, left is faster. Median response time after warmup, log scale. Squares mark models that take images; this test used text only.
Accuracy vs memory
● Best local trade-offs● Other local runs○ GLiNER MLX, allocator peak■ Supports images
Loading measured results…
Up is more accurate, left uses less memory. GLiNER MLX counts allocated memory, not the whole process, so it stays off the line. Observed values, not RAM requirements.
Background
Can you get Jev-level reasoning in real time, on-device? Jev turns app state into a choice with probabilities, through an API. We tested five local model families as replacements: Imajev, Plumb, GLiNER2.5-Decide, Tev1, and Nimble. To replace the API call, a local model needs Jev’s accuracy at a similar response time.
Methodology
We picked 20 text decisions from Decision Bench before seeing any answers, two from each of ten categories. Every model got the same cases and choices on an M3 Pro with 36 GiB of memory. We sent one request at a time and report the median response time after warmup. Each model used its own interface, GLiNER with its own prompt, and no model saw the answer key.
Results
Nimble 9B was the only local model to beat Jev, 95% to 90%, but took 1.91 s, 9.1× as long. Tev1 0.8B was the fastest local model at 219 ms, scoring 80%. Imajev BF16 and 8-bit matched Jev’s 90% at about 1.24 s. Plumb scored 85% in 1.01 s, and GLiNER MLX int8 scored 70% in 227 ms.
Discussion
One extra correct answer does not make Nimble more accurate: 20 public cases cannot pick a winner, and some may be in training data. Memory readings differ by runtime, so treat that chart as a rough guide, not a RAM shopping list. First answers after unloading took about 1 s for Tev1 0.8B, 2–3 s for Nimble, and 93–101 s for Tev1 4B. The medians hide those load costs.
Conclusion
You can get Jev-level scores on-device, but not Jev’s speed. Imajev matches its score at about 1.2 s, and Nimble gets one more answer right at 1.9 s. Tev1 0.8B comes closest on speed but drops to 80%. Test that trade-off on your own decisions before replacing the API.
Data and run details
Results
| Model and runtime | Correct | Median | P95 | Observed memory | Cold start |
|---|
20 decisions each. Median and P95 are single-request wall times. Cold start is the observed first response for Tev and Nimble, or approximate model load for GLiNER. It is not a matched test across runtimes. For some quantized variants, the first measured call includes JIT or load time.
All 20 paired decisions
Each cell is the returned choice: ✓ matches the public answer, × does not. Scroll sideways on mobile. Per-request probabilities and times are in the downloads.
Method
Cases come from the Decision Bench bench-v4 public export (25 Sept 2026). The fixed sample has two cases each from engineering, agents, safety, support, commerce, finance, legal, product, data, and documents. Corpus SHA-256: 3599baea0d9c7e4e3d86c8b06af96e6850edca037b334bbc6b4e1a0033021725. Sample SHA-256: b89a790bd832223630fd2d757e38e6f24856002e757982be189534d00a5b2785. Case IDs and outputs are in the downloads; full case texts stay with the source dataset. No answers were sent with any request.
Jev 1.13, Imajev BF16 plus 4-bit and 8-bit quantizations, Plumb 8-bit, GLiNER2.5-Decide on CPU, MLX, and MLX int8, Tev1 4B and 0.8B, and Nimble 9B each answered the same 20 cases. GLiNER skips the generic LLM policy and uses its task prompt. A correct answer counts even if Imajev flagged abstention. This is a one-pass pilot, not a sealed JevBench or full Decision Bench score.
Other observations
Without Imajev’s 512 MiB MLX free-cache cap, its process grew to about 27 GiB and response times spiked. That run is saved below but left off the charts. Jev Cloud used 17232 metered input tokens for 20 calls, about $0.00072 at the recorded rate. Local runs have no API fee but use this Mac’s compute and power. Imajev can classify images; this comparison is text-only. Nimble ran in Ollama 0.35.0 as the default 9B Q8_0 variant (9.5 GB download, digest aa4a79f08ae089919b82cab5612f073868307306c4213f724588e38a4a764aca). Its two cold first responses took 1.94 s and 3.24 s.
For context, JevBench v1.4.2.2 reports composite scores of 67.37 for Imajev BF16, 65.84 for Plumb BF16, and 63.29 for Jev 1.13, on a different task mix. The full Decision Bench bench-v4 leaderboard lists Jev at 92.4% and Tev1 4B at 85.4%. None of those scores cover these local quantizations or this 20-case sample.
Downloads
All results, with 20 paired casesJev + Imajev BF16Uncapped Imajev (excluded)Imajev 4-bitImajev 8-bitPlumbGLiNER, 3 runtimesTev1 4BTev1 0.8BNimble 9B
Downloads have per-case outputs and times but not the full input text. Chart values come from the eight plotted result files; observed memory is recorded separately.