CutCtx guides
Prompt Caching and Context Compression: What Each One Changes
Understand the different evaluation questions for reusing repeated prompt content and reducing or selecting context for a task.
Prompt caching and context compression are different levers. Caching concerns whether repeated input can be reused under a provider's documented behavior; compression changes, filters, or selects the information sent to a task. A workflow may use either, both, or neither, depending on its provider, request pattern, quality requirements, and measurement results.
Provider behavior and pricing change. Consult the current provider documentation for the model and endpoint you use. This guide avoids fixed cache thresholds, time windows, price multipliers, or savings estimates because those details must be checked against current documentation and actual usage.
Separate reuse from information selection
A cache can reduce repeated processing of eligible prompt content according to a provider's rules. It does not automatically mean that the prompt is shorter or that irrelevant details were removed. Context compression or selection attempts to change which information is presented, often by filtering or transforming source material. That can reduce input size but can also remove a fact the task needs.
The OpenAI and Anthropic documentation describe provider-specific prompt-caching behavior. Read the current documentation for the exact model and request path rather than assuming that details transfer between providers or versions.
| Question | Caching evaluation | Compression/selection evaluation |
|---|---|---|
| What changes? | Reuse behavior for eligible repeated input | Which information is sent, and in what representation |
| Primary opportunity | Stable repeated content across requests | Redundant, irrelevant, or overly broad context |
| Main risk | Eligibility or billing behavior differs from expectation | Important evidence or nuance is omitted or distorted |
| Quality check | Compare equivalent outputs under documented cache behavior | Check task quality against the full-context baseline |
| Cost check | Use current provider billing fields and price schedule | Count all input, output, transformation, and retry charges |
Use a complete cost worksheet
Do not compare only a headline input-token count. Record uncached input, any cache-write and cache-read amounts exposed by the provider, output tokens, any separate transformation or retrieval calls, retries, and the provider's current charges for each category. Keep provider usage reporting and the applicable price schedule with the calculation.
This worksheet separates usage categories; it does not encode current prices. If a provider does not expose a category or usage field for the request, record it as unavailable rather than estimating it as zero.
Check the definitions of usage fields before adding them. A provider may report cached tokens inside a total input field or as separate input categories. Do not add a subtotal to a total that already includes it; derive uncached usage only when the provider documentation supports that calculation.
- Uncached input: ____ units × current applicable rate = ____.
- Cache-write input, if reported: ____ units × current applicable rate = ____.
- Cache-read input, if reported: ____ units × current applicable rate = ____.
- Output: ____ units × current applicable rate = ____.
- Selection/compression, retrieval, retries, or other calls: itemize separately.
- Total: add observed categories; retain the raw usage record and price date.
Design a comparison that can distinguish the effects
Keep the task, evidence, model version, output constraints, and evaluation rubric fixed. Compare a baseline with provider-native cache behavior where applicable and a context-selection or compression arm. If the workflow normally performs provider-native compaction or another built-in summarization step, include that baseline too; do not compare compression only against an unusually inefficient raw prompt.
Measure task quality, omissions, human corrections, latency, and all provider charges. Use held-out tasks for confirmation. If the provider or model is not supported in one arm, record that arm as unavailable or blocked rather than silently substituting a different system.
- Freeze task inputs and the quality rubric before comparing arms.
- Record provider, model, endpoint, cache settings, and date for each run.
- Separate quality outcomes from token counts and billing outcomes.
- Include setup, transformation, retries, failures, and human review in total effort.
- Keep a held-out set so tuning does not become the reported result.
Choose the simplest method that meets the need
If the prompt is repeated and already contains the right information, caching may be the relevant mechanism to investigate. If it contains irrelevant or redundant material, selection or compression may be worth testing. If context is retrieved progressively, a smaller initial request may be possible without trying to summarize everything in advance. These are hypotheses to verify for a particular workflow, not guarantees.
Retain the original source and a way to inspect what the task actually received. For a private-beta product or any other system, verify current supported behavior in its own documentation and route-specific tests before making product claims.