Which compaction works on long coding tasks?
Encrypted compaction uses 22% fewer tokens than text summaries.
29 Sept 2026 · 24 Terminal-Bench 2.1 tasks × 4 runs · GPT-6 Luna, high reasoning
Pass rate, tokens, and time
Background
Long coding sessions fill the context window. Textual compaction asks Pi to write a summary; encrypted compaction keeps OpenAI’s compacted state. Jev pruning scores older tool calls and results, then keeps, shortens, or drops them without rewriting the conversation. We wanted to know which saves tokens without losing the task.
Methodology
We ran 24 selected Terminal-Bench 2.1 tasks four times per method with GPT-6 Luna at high reasoning. All methods compact at 50000 tokens. Textual and encrypted runs started in pairs; Jev ran later on the same tasks. Pass rates use the 12 tasks that compacted often in the paired runs, while token, cost, and time averages use all scored runs.
Results
Encrypted compaction used 0.91 million GPT-6 tokens per run versus 1.18 million for textual, and cost 10% less. Pass rates were 53.5% encrypted, 56.9% textual, and 58.3% Jev. Jev used the most tokens and cost the most per run. The useful result is the resource saving, not a quality winner.
Discussion
The pass-rate gaps are too small to call: encrypted minus textual was −3.5 points (95% interval −15.3 to +6.9), and Jev minus textual was +1.4 (−12.5 to +13.9). Jev ran later, so time and machine load may confound its result. Errors are excluded, including one Jev run. An audit replay passed after excluding synthetic preflight records; the original flagged audit and raw logs stay available.
Conclusion
Try encrypted compaction first when token use is the problem. It saved tokens and money here, but this run does not prove equal quality. Jev pruning needs a stronger quality or cost result to justify the extra moving parts.
Data and run details
Complete · 192/192 paired runs · about $9.84 spent
Results
Jev leg complete · follow-up audit passed · 96/96 runs · 95 scored · 1 error · $4.81 spent, incl. $0.06 Jev
Jev vs textual: +1.4 pp (95% interval −12.5 to +13.9, 12 matched tasks). Jev vs encrypted: +4.9 pp (−6.3 to +17.4). Jev ran later, so this is a historical comparison, not a concurrent pair.
| Method | Pass rate, compacting tasks | GPT-6 tokens per run | Cost per run | Median time |
|---|
More measures
Results by task
Passes out of scored runs. ● pass, ○ fail, ✕ error. Groups come from the paired runs.
| Task | Group | Textual | Encrypted | Jev | Jev compactions |
|---|
Paired runs and other task groups
Mean pass rate across tasks. On tasks that never compacted, both methods run the same setup, so any gap there is run-to-run noise.
| Task group | Textual | Encrypted | Difference |
|---|
Textual vs encrypted by task
Passes out of scored runs. Compaction rate pools both methods. ⏱ marks timed-out runs, which can still pass.
| Task | Group | Compaction rate | Textual | Encrypted | Outcome |
|---|
Method
We ran 24 long Terminal-Bench 2.1 coding tasks with GPT-6 Luna at high reasoning, four times per method in fresh containers. Textual and encrypted runs started in pairs under similar machine load. Twelve tasks compacted in version 1; the other twelve are unused long tasks. We did not pick tasks by score. This selected subset is not an official leaderboard result and cannot show a general coding benefit.
Jev 1.13 scores each older tool call and result using fast-jev-compaction (pinned source). It sees the conversation with tool outputs left out, keeps the six newest messages, and never rewrites assistant or user text. For each older call, it keeps the call and result, cuts the result to 300 characters, or drops the pair. Its key stays outside the task container, and a failure never falls back to another method.
Design, scoring, and cost
Version 1 ran each task once and had only four discordant pairs, two of them on tasks that never compacted. Version 2 repeats tasks, uses the frequently compacting group for the main estimate, and checks tasks that never compacted separately. Both methods now keep Pi’s developer message after compaction, and the textual method no longer gets an extra finish-and-verify instruction.
We average each task’s pass-rate difference, then resample whole tasks for a 95% interval (20000 resamples, fixed seed), since repeats of one task are not independent. The verifier scores the container after the agent stops or times out, so a timed-out run can still pass. Infrastructure errors are excluded and listed. A compaction counts only when another agent request follows it.
Costs use billed tokens at OpenAI’s published standard rates, including summary calls. One attempt per run, with no score-dependent changes. Dataset license. The methods note covers the full design.
Jev ran later on the same 24 tasks × 4 runs, not alongside the paired methods. Task matching gives a task-cluster 95% interval, but time and machine load can still confound it. One Jev run was unscored. The original Jev audit flagged three synthetic preflight records; a separate replay without them passed and changed no trial data (audit review).
Downloads
Paired results JSONPaired protocolPaired source and auditSummary JSONJev results JSONJev protocolJev source and auditAudit reviewMethods noteVersion 1