Gisting compresses reusable prompt context into a shorter set of learned token activations that a model can reuse instead of processing the full prompt each time. It can reduce repeated prompt work, but compression ratios and speedups depend on the model, task, context length, and serving setup—and long-context tests expose important limits.
What is gisting for LLMs?
Gisting is a learned method for compressing prompt information. During training, a model learns to route useful information from a prompt through a smaller number of special gist-token activations. At inference, those activations can stand in for the original prompt and be cached for reuse.
As an Amazon Associate I earn from qualifying purchases.
In the original method, gist tokens are inserted after a prompt during instruction tuning. A modified attention mask prevents later tokens from attending directly to the prompt tokens that come before the gist tokens. The model must therefore preserve, in the gist-token activations, the prompt information it needs to answer. This is intended for context that is reused, such as stable instructions—not a guarantee that arbitrary text can be compressed without losing task-relevant detail. Mu, Li, and Goodman’s NeurIPS 2023 paper
Recommended Free Tools
How does a gist representation get learned?
The original research trains the model so its later responses can rely on gist activations rather than direct access to the full prompt. Shopify describes a related but distinct recipe for its Sidekick GraphQL agent: it freezes model weights and trains gist embeddings using knowledge distillation. In the teacher pass, the model sees the full natural-language prompt; in the student pass, it receives gist tokens and is trained to match the teacher’s response logits. That is Shopify’s described implementation, not necessarily the exact training procedure used by every gisting system. Shopify Engineering’s account
#1 Best Overall
How much can gisting compress a prompt?
Mu, Li, and Goodman report up to 26× prompt compression and up to 40% fewer FLOPs, with minimal output-quality loss, in their evaluated LLaMA-7B and FLAN-T5-XXL configurations. These are the paper’s upper-end findings, not expected results for every model or workload. The paper also reports 4.2% wall-time speedups and storage savings; those measurements are specific to its experiments. NeurIPS 2023 paper
Shopify’s 2026 engineering article describes a production deployment on its Sidekick GraphQL agent. It says the system prompt was compressed from approximately 6,000 tokens to approximately 1,500 gist tokens, a 4:1 reduction. In Shopify’s reported load tests at 350 requests per minute, the company measured these changes:
| Metric | Before gisting | With gisting |
|---|---|---|
| Median time to first token (TTFT) | 438 ms | 354 ms |
| Median end-to-end latency | 6.8 s | 4.2 s |
| Throughput | 20.2 queries per second | 23.4 queries per second |
These are outcomes Shopify reports for its own deployment and load tests; they are not independently audited results or a forecast for another system. The paper’s FLOPs and wall-time findings and Shopify’s latency and throughput measurements describe different models, metrics, and environments, so they should not be compared as if they were the same benchmark. Shopify says it was able to reduce GPUs allocated to the GraphQL agent’s traffic. Shopify Engineering Shopify’s reported performance figures
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesDoes gisting work for long contexts?
Not reliably in every setting. A 2025 study of long-context in-context compression reports significant performance drops as context grows, including losses at minimal compression in its experiments. It identifies interrupted information flow, limited representation capacity, and difficulty restricting attention to relevant subsets of context as contributing problems. In those experiments, simple average pooling consistently outperformed the original gisting approach; the authors propose GistPool as an alternative intended to improve long-context performance. These findings qualify the original method’s promise, but do not establish one universally best compression method for every model or task. Petrov et al., “Long Context In-Context Compression by Getting to the Gist of Gisting” (2025)
How should you decide whether to use gisting?
The key question is whether the savings from reusing a compressed representation survive on your actual task and serving stack. Evaluate quality alongside efficiency: a high compression ratio is not useful if the model loses instructions or details needed for correct responses.
- Context and task: Test stable, repeated instructions separately from long documents or conversations. The method’s usefulness depends on what information must survive compression.
- Quality across compression levels: Measure task-specific output quality at several compression ratios, rather than testing only the most aggressive setting.
- Reuse frequency: Estimate how often the same context is reused. Training or distillation and caching are easier to justify when a representation serves many requests.
- Serving metrics: Benchmark latency, throughput, memory, and compute on the intended hardware and inference stack. A shorter representation does not by itself establish faster end-to-end serving.
- Long-context alternatives: For longer inputs, include average pooling and GistPool in the evaluation where appropriate; the 2025 study found them relevant alternatives to original gisting in its experiments.
What should you know before reproducing the original implementation?
The authors’ public repository provides code for inspecting and running the research implementation, but its README documents constraints that matter for reproduction. Gist compression is supported for batch size 1; larger batches have only partial implementation and less carefully checked correctness. For LLaMA-7B, larger batches also require rotary-position adjustments for gist offsets. The repository specifies a required Transformers commit and a DeepSpeed version used for reproducible training. Its released weight-diff checkpoints require the base LLaMA-7B weights. Gisting repository Repository README
The maintainers also caution that their gist-caching implementation was not heavily optimized. Additional Python logic may make wall-clock gains small or nonexistent, particularly on the CPU side. They describe that implementation as a demonstration of caching and a way to validate attention-mask behavior, not as a production serving product. Reproducing a paper result or running the repository code therefore does not establish the gains you will see with a current production stack.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
