AWS’s RAG Cost Cut Comes With a 19% Latency Bill

AWS is not promising that RAG can become cheaper without a sacrifice. Its query-aware compression pattern cuts the context sent to the main answer model, but AWS’s benchmark also shows a 19% latency increase and composite answer quality at 97.5% of the baseline. The proposal is a targeted trade: spend a small-model call to avoid paying a larger model to read material the question does not need.

A second model becomes an evidence filter

The design adds one step after retrieval and before answer generation. A retriever first returns the top-k document chunks; AWS then has Claude Haiku read the user’s question and those chunks, extract only query-relevant verbatim spans, and pass that compressed evidence to Claude Sonnet for the final answer.

Both invocations run in one AWS Lambda function through the Amazon Bedrock Converse API. The key constraint is extraction rather than summarization: preserving source wording and chunk identifiers is intended to keep evidence available for downstream citations while removing irrelevant text.

The savings depend on a simple price-and-volume equation. The smaller model must process all retrieved text and produce the shortened context, but the higher-priced answer model reads only that shorter result. AWS says the pattern is most favorable when retrieved context is large, the price gap between the models is wide, and substantial material can be removed without losing needed evidence.

67% of baselineCost with compression

AWS measured compression costs at 67% of its baseline pipeline cost.

12% of baselineContext sent to primary model

Compression reduced tokens sent to the primary model to 12% of baseline in AWS’s benchmark.

+19%End-to-end latency

The additional compression step increased latency by 19% versus the baseline.

The token result is large; the evaluation is bounded

AWS tested more than 500,000 documents across nine enterprise source types and 500 questions in 10 categories. It compared an uncompressed baseline with compression and rerank-plus-compression, using an LLM judge to score correctness, completeness, citation accuracy and conciseness against reference answers; it also tracked faithfulness against the evidence given to the final model.

On that setup, compression reduced cost by 33% and cut primary-model input tokens by 8.6 times. The quality score was slightly lower than baseline, and AWS says completeness and citation accuracy declined slightly while conciseness improved. That makes the result a cost-performance measurement, not evidence that the filter improves every answer dimension.

Where the decision gets harder

• Latency-sensitive applications need their own measurement: the compression call is an added step, even if the final model has less context to process.
• Questions requiring multi-step reasoning can depend on an evidence chain staying intact, which AWS identifies as a deliberate compression-design concern.
• AWS says its benchmark represents one corpus, domain and query distribution, so performance can differ with other documents, queries and model choices.

Reranking improves the scorecard, not the deployment verdict

AWS’s combined rerank-plus-compression pipeline went further in the benchmark: cost fell to 64% of baseline and primary-model tokens to 10%. It also posted 97.6% of baseline composite quality and 12% higher latency, rather than the compression-only pipeline’s 19% increase.

The benchmark’s hallucination measure moved in the favorable direction, from 51% in the baseline to 44% with compression and 38% with reranking plus compression. But the mechanism also creates a new operational responsibility: teams must validate that an extractor does not omit the evidence a final answer needs. AWS recommends adapting the prompts to the document set and question types rather than treating the example pipeline as a drop-in default.