CutCtx guides

How to Benchmark Context Compression Without Hiding Quality Loss

A reproducible protocol for comparing context strategies on held-out tasks while tracking quality, corrections, latency, and full cost.

A context strategy is useful only if it preserves the information needed for the task at an acceptable total cost and level of effort. A lower input-token count alone cannot show whether an important caveat was lost, a tool call became less effective, or a person had to repair the result.

This protocol describes a benchmark design, not a result. It includes ordinary provider behavior and baseline strategies so that a compression method is not compared only with an intentionally weak prompt. No outcome or savings figure is assumed.

Freeze tasks, inputs, and the quality rubric

Assemble representative tasks with known source material and acceptance criteria. Keep a held-out set that is not used to tune prompts or compression rules. Where tasks contain sensitive information, use a data-approved evaluation environment and follow the applicable access rules.

Define what a correct answer must include, what counts as a material omission, and which failures stop a run. Use reviewers who can judge the task, and preserve disagreement rather than forcing a false consensus.

  • Record task purpose, source set, expected evidence, and exclusions.
  • Separate development tasks from held-out confirmation tasks.
  • Define a rubric for correctness, completeness, provenance, and usability.
  • Set retry, time, and cost limits before comparing methods.
  • Record provider, model, endpoint, tool configuration, and date.

Compare meaningful baseline and treatment arms

Include the workflow's ordinary full-context baseline and relevant provider-native caching or compaction behavior. Add context selection, progressive tool discovery, or compression arms only when they are applicable to the actual task. These methods answer different questions: a selector may change which sources enter the request, while compression may transform context that was already gathered.

Keep source access and task instructions equivalent across arms. Counterbalance run order where people are timing or reviewing outputs. If a method is unavailable for a provider or task, label it unavailable instead of substituting an unrelated method.

For an incremental comparison, first combine the workflow's supported native cache or compaction settings with source selection and progressive discovery. Compare that optimized baseline with the same workflow plus compression, keeping the native controls enabled. Otherwise a gain over an unnecessarily broad prompt can be mistaken for an improvement over the process a team would actually use.

Where another direct-compression method supports the same input type, it can be an additional comparator. Separate compression-only results from end-to-end workflow results, and mark unsupported inputs or missing access explicitly.

ArmWhat it testsImportant control
Ordinary full-context workflowCurrent baseline quality and total effortUse the team's normal documented process
Provider-native cache/compaction behaviorEffect of documented native behaviorCapture provider settings and usage fields
Selection or progressive discoveryWhether narrower source/tool selection is adequateKeep the allowed source universe fixed
Combined native and selection baselineThe optimized baseline for incremental comparisonKeep normal cache/compaction and source selection enabled
Compression strategyWhether transformed context preserves task utilityRetain source pointers and inspect omissions

Measure quality, corrections, and full cost

For every run, record the rubric result, omitted or distorted facts, unsupported claims, tool failures, human correction time, retries, and latency. Capture all provider-reported input and output categories, including cache-related usage where exposed, plus any separate calls used for retrieval or transformation. Apply current provider pricing for the exact model and date; do not compare token totals as if they were prices.

Track task-level outcomes as well as aggregate summaries. A strong average can conceal that one strategy consistently fails on dates, negation, source attribution, or a high-impact exception.

  • Quality: correctness, completeness, provenance, and critical omissions.
  • Human effort: review minutes, corrections, and escalation count.
  • Runtime: latency, timeouts, tool calls, and retry count.
  • Usage/cost: uncached input, cache write/read, output, and other calls separately.
  • Reliability: failed runs, incomplete runs, and recovery actions.

Report uncertainty and decide what to test next

Publish the protocol, task counts, exclusions, and results together. Distinguish measured findings from interpretations. If an arm has too few tasks, missing usage data, or unresolved scoring disagreement, label the conclusion accordingly rather than presenting it as general performance evidence.

Use the held-out set once the method and rubric are fixed. If the confirmation result differs from the development set, preserve both results and investigate the gap. A small internal benchmark supports only the tested tasks and environment; it does not establish universal savings, safety, or superiority.

References