What Makes E8 Special

The E8 lattice is a root lattice in 8 dimensions with 240 nearest neighbors. Its kissing number (240) is the highest possible in 8D. Its packing density is optimal: no other 8D arrangement of equal spheres covers more space. Mathematically, E8 achieves the theoretically maximum sphere-packing density in 8 dimensions — proven by Viazovska in 2016.

For quantization, this matters because denser packing = more codewords per unit volume = lower quantization error.

Why KV Vectors Live in E8-Friendly Space

After applying a Hadamard transform to KV cache vectors, the distribution of each coordinate becomes approximately sub-Gaussian. Specifically:

• The Hadamard spreads energy uniformly across all 8 dimensions
• Each coordinate has zero mean and bounded kurtosis
• The joint distribution approximates a spherically symmetric Gaussian cloud

A spherically symmetric Gaussian is exactly what E8 was designed to quantize. The shell structure of E8 — its concentric layers of lattice points — aligns with the probability mass shells of a Gaussian. More lattice points where the data actually lives.

The Relaxed Parity Discovery

Strict E8 imposes an even-sum parity constraint: the sum of all 8 coordinates (after scaling) must be even. This halves the set of valid codewords and enforces a rigid algebraic structure.

We found something unexpected: relaxing this constraint improves MSE by 0.3–0.4% on KV cache data.

Why? Sub-Gaussian distributions have excess probability mass near the origin compared to a pure Gaussian. Strict E8 parity creates a gap at the origin — the all-zeros vector is forbidden if it violates the even-sum rule. Relaxed parity restores codepoints near zero, which is precisely where sub-Gaussian data concentrates.

This is not a bug. Nature found a better quantizer than the textbook prescribed.

Strict E8:   valid if sum(coords mod 2) == 0   → 128 points per shell
Relaxed E8:  always valid                       → 256 points per shell
Gain at origin: more codewords where sub-Gaussian data concentrates

The Numbers

On Mistral-7B KV cache vectors (Hadamard-preprocessed, group size 64):

Relaxed E8 beats strict E8 by 22% MSE reduction. It also beats fp16 on perplexity — compression that makes the model more accurate.

Why This Works at Scale

KV cache vectors are not random. They carry structured information — token relationships, positional encodings, semantic content. After Hadamard rotation, this structure disperses into approximately sub-Gaussian noise, but the near-origin concentration persists across layers and models.

E8 with relaxed parity is not a coincidence. It is the right mathematical structure for the right data distribution. The 8-dimensional optimality of E8 matches the head-dimension granularity of modern transformers (head_dim = 64 or 128, divisible by 8).

The pipeline is three lines of math:

1. Normalize and scale (NSN)
2. Rotate to sub-Gaussian (Hadamard)
3. Quantize to nearest E8 point (relaxed parity)

That is the entire compression stack. No neural networks. No training. No calibration data.

Best regards, João Marques

The Raw Numbers

A 70B model stores KV cache at 2 bytes per element (fp16). With 96 layers, 64 heads, 128 head-dim, the KV cache per token is:

bytes_per_token = 2 * num_layers * 2 * num_heads * head_dim * bytes_per_element
                = 2 * 96 * 2 * 64 * 128 * 2
                = 6,291,456 bytes ≈ 6 MB/token

At 1M tokens: 6 TB. Two H100s hold 160 GB combined. You are 37× short.

The Compression Table

17× compression: 70B at 1M tokens fits on 2× H100 (120 GB). 33× compression: 70B at 1M tokens fits on a single H100 (80 GB).

The Python Formula

def kv_cache_gb(
    model_params_b,      # e.g. 70 for 70B
    context_length,      # e.g. 1_000_000
    compression_ratio=1, # NexusQuant preset
    bytes_per_element=2, # fp16
):
    num_layers = int(model_params_b ** 0.45 * 5.2)  # empirical fit
    num_kv_heads = 8   # GQA default for modern 70B
    head_dim = 128
    kv_bytes = (
        context_length
        * num_layers
        * 2  # K and V
        * num_kv_heads
        * head_dim
        * bytes_per_element
    )
    return kv_bytes / compression_ratio / 1e9

# Examples
print(kv_cache_gb(70, 1_000_000, compression_ratio=17))   # ~120 GB
print(kv_cache_gb(70, 1_000_000, compression_ratio=33))   # ~60 GB
print(kv_cache_gb(7,  1_000_000, compression_ratio=5))    # ~84 GB

What This Means in Practice

NexusQuant presets map directly to GPU configurations:

• Preset S (5×): 7B model, 1M context → single H100
• Preset M (10×): 13B model, 1M context → single H100
• Preset L (17×): 70B model, 1M context → 2× H100
• Preset XL (33×): 70B model, 1M context → 1× H100

The bottleneck was never the model weights. It was always the KV cache. The math is solved.

with nexusquant(model, preset="L"):  # 17x, -0.03% quality
    output = model.generate(million_token_prompt)

Best regards, João Marques

What went out the door

PyPI package — pip install nexusquant works. One line, no retraining, drop-in KV cache compression for any HuggingFace model.

HuggingFace Space — live interactive demo at huggingface.co/spaces/jagmarques/nexusquant. Upload a model, pick a compression ratio, see perplexity in real time.

Google Colab notebook — zero-setup walkthrough. Run the full pipeline in your browser without a local GPU.

13 blog posts — covering everything from E8 lattice quantization to attention-aware eviction, each with reproducible numbers and code.

9 awesome list PRs — submitted to awesome-llm, awesome-efficient-transformers, awesome-kv-cache, and six others. Four already merged.

5 GitHub issues — filed against PyTorch, vLLM, HuggingFace Transformers, LiteLLM, and llama.cpp to track upstream integration roadmap items.

NeurIPS paper draft — the research that underpins all of this: NSN + Hadamard + E8 Lattice VQ + TCC giving 7x compression with -2.26% PPL on Mistral-7B, beating TurboQuant by 32% compression with better quality.

The numbers that matter

• 7.06x compression, training-free
• -0.002% PPL at 5.3x on Llama-3-8B (essentially lossless)
• 128K context → 680K tokens in the same GPU memory at 5.3x
• 128K context → 2.6M tokens at 20x with token merging

What is next

We are looking for contributors on:

• vLLM integration (PagedAttention compatibility)
• Flash Attention 3 support
• Quantization-aware fine-tuning experiments
• Benchmarks on Gemma-3 and Qwen-3

If any of this is your area, open an issue or ping me directly.

Links

• PyPI: pypi.org/project/nexusquant
• GitHub: github.com/jagmarques/nexusquant
• HF Space: huggingface.co/spaces/jagmarques/nexusquant
• Colab: linked in the repo README

Best regards, João Marques

NexusQuant — unlimited context windows for every AI model.

The experiment

We ran KV cache eviction at two rates on Llama-3-8B, measuring perplexity degradation (lower is better) versus a full-cache baseline:

The gap grows as you evict more. At 70% eviction the importance scorer saves you 1.27 percentage points of perplexity. Push to 80% and it saves 1.52 pp. This is not a coincidence.

Why it happens

Random eviction is memoryless — it has the same probability of dropping the single token that unlocks subject-verb agreement across 400 tokens as it does of dropping a filler word. The attention-aware scorer assigns each token an importance weight based on how much accumulated attention mass it has received across all heads. Tokens that many heads consistently attend to survive; tokens that nobody looks at get evicted first.

At low eviction rates there is enough slack that random and importance-based look similar. As you push the eviction rate up, the budget gets tight and every dropped token counts. That is when the scorer earns its keep.

Run it yourself

pip install nexusquant

from nexusquant import NexusQuantConfig, apply_nexusquant
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

# Importance-based eviction at 80%
cfg = NexusQuantConfig(eviction_rate=0.80, eviction_mode="importance")
apply_nexusquant(model, cfg)

# Compare: random eviction at 80%
cfg_rand = NexusQuantConfig(eviction_rate=0.80, eviction_mode="random")
apply_nexusquant(model, cfg_rand)

The full benchmark script is in the NexusQuant repo.

Takeaway

If you are evicting KV cache tokens, use an attention-aware scorer. At 80% eviction the gap is 1.52 pp — and it only widens from here. Random eviction is a baseline, not a strategy.

Best regards, João Marques

NexusQuant — unlimited context windows for every AI model.

This is the opposite of that.

Here is a complete record of every approach I tested building NexusQuant, a KV cache compressor for LLMs. Including the ones that failed spectacularly.

The problem I was trying to solve

KV cache is what limits LLM context windows. A 70B model on a single A100 can handle roughly 128K tokens before running out of memory. Every approach I tested was measuring one thing: can I cut that memory footprint without hurting model quality?

The metric is perplexity delta (lower is better, negative means quality improved). The target was less than 1% degradation at meaningful compression.

Attempt 1: PCA rotation before quantization

Reasoning: if I rotate the KV vectors into a principal component basis, the quantization grid should align better with the actual data distribution.

Result: 3x worse perplexity than baseline quantization, same compression ratio.

Lesson: PCA rotation costs bits. You are now storing a rotation matrix. On short sequences, that overhead dominates. On long sequences, the rotation is computed over a batch that does not represent the test distribution. The math is sound. The engineering tradeoff is not.

Attempt 2: Group-32 quantization

Reasoning: quantizing in groups of 32 values gives more granular scales than group-128, which should improve quality.

Result: compression ratio penalty of ~0.4x with negligible quality gain.

Lesson: smaller groups mean more scale parameters. Each scale is 16-bit. At group-32 on a typical KV tensor, the scale overhead eats almost half your compression. I was measuring compression ratio wrong — I was ignoring the metadata. This was the most important mistake I made. After this, every experiment tracked torch.cuda.memory_allocated() directly.

Attempt 3: Adaptive bitwidth per head

Reasoning: some attention heads carry more signal than others. Allocate more bits to important heads, fewer to redundant ones.

Result: negligible improvement, high implementation complexity.

Lesson: "adaptive" anything requires a policy. A policy requires calibration data. Calibration data leaks information about your test set if you are not careful. And the marginal gain over simply using a good fixed bitwidth was consistently under 0.1% PPL. Did not ship.

Attempt 4: Per-head token eviction

Reasoning: not all tokens in a KV cache are attended to equally. Evict the low-attention tokens per head.

Result: catastrophic quality degradation. +8% perplexity on Llama, worse on Mistral.

Lesson: attention scores at layer N are a terrible proxy for importance at layer N+5. The tokens you evict at layer 3 are sometimes the ones that become critical at layer 20. Eviction requires looking at the full attention pattern across all layers simultaneously, which you cannot do in a single-pass, memory-efficient way. This approach might work with a different architecture. It did not work here.

Attempt 5: Token merging

Reasoning: merge similar adjacent tokens in the KV cache to reduce sequence length.

Result: +107% perplexity degradation. The worst single result in this log.

Lesson: token merging works in vision transformers because adjacent image patches are often semantically redundant. In language, adjacent tokens are not redundant — they are causal. Merging "the" and "cat" into a single averaged vector destroys the sequential structure that the model was trained on. This is one of those cases where a technique that makes perfect sense in one domain is completely wrong in another.

Attempt 6: Scalar quantization with learned step sizes

Reasoning: learn the quantization step sizes end-to-end on a small calibration set.

Result: good quality, but training-dependent. Quality dropped significantly on out-of-distribution prompts.

Lesson: learned quantization is not training-free. If you fit step sizes on Wikipedia and deploy on code, you will have a bad time. I wanted a system that needed zero calibration. Scratched this.

Attempt 7: Hadamard transform alone

Reasoning: rotate the KV vectors with a Hadamard matrix before quantizing. Hadamard is fast (O(n log n)), lossless, and spreads outlier values across all dimensions.

Result: meaningful improvement over vanilla quantization. The outlier suppression effect is real.

This was the first technique that survived. It became stage 2 of the current pipeline. But alone, it was not enough.

Attempt 8: E8 lattice quantization alone

Reasoning: the E8 lattice is the densest packing of spheres in 8 dimensions. If KV vectors have any spherical structure, E8 should be a better codebook than a scalar grid.

Result: strong quality at the same compression ratio as 4-bit scalar. Better than expected.

Lesson: KV vectors in the heads of large transformers do have roughly spherical structure after normalization. This is not obvious from first principles but it shows up empirically across multiple models. E8 exploits this structure. Scalar quantization does not.

Attempt 9: Hadamard + E8 together

Result: better than either alone. The combination is not additive — it is multiplicative. Hadamard removes outliers that would confuse the E8 codebook lookup. E8 then operates on a distribution that more closely matches the sphere-packing assumption.

This is now stages 2 and 3 of the pipeline.

Attempt 10: Neighbor-aware soft normalization (NSN) before Hadamard

Reasoning: the Hadamard transform is sensitive to the scale of the input. If I normalize each vector using context from its neighbors before applying Hadamard, the transform should work better.

Result: yes. Consistent improvement. Adds one stage but with measurable payoff on every model I tested.

This became stage 1.

Attempt 11: Temporal predictive coding (TCC) after E8

Reasoning: KV cache values change slowly across token positions for many heads. If I encode only the delta between adjacent KV vectors rather than the full vector, I can get additional compression for free.

Result: +35-40% compression ratio with minimal quality cost on most heads. Some heads have high delta (they do not compress well with TCC). Mixed mode — apply TCC selectively based on per-head delta statistics.

This became stage 4.

Attempt 12: More stages — cross-head redundancy removal

Reasoning: different heads sometimes learn similar representations. Compress across heads, not just within.

Result: complex to implement, marginal gain (~0.1% PPL improvement). Did not meet the bar for justifying the complexity.

This is where I stopped adding stages.

The final pipeline

NSN → Hadamard → E8 Lattice VQ → TCC

Four stages. Each one earned its place by clearing a measurable bar. Three of the twelve approaches I tested made it in. Nine did not.

Best result: 7x compression, -2.26% perplexity on Mistral-7B. Training-free, drop-in.

What the failures actually taught

The biggest insight is not about any individual technique. It is about the measurement problem.

Early in this process I was reporting compression ratios that did not include scale overhead. I thought I had a 5.3x compressor. When I measured torch.cuda.memory_allocated() honestly, it was 3.2x. That is a catastrophic difference. The lesson: every number has to trace to a specific experiment with full config. If you cannot point to the exact line of code that produced the number, the number is not real.

The second insight is about domain transfer. Techniques from vision (token merging, PCA rotation) did not transfer to language KV cache. Techniques from signal processing and sphere-packing (Hadamard, E8) did. The structure of the problem dictates which tools apply. Before importing a technique from another domain, ask whether the assumptions that make it work there hold here.

Best regards, João Marques

Most engineers know the KV cache is "large" but few have actually calculated the exact number. This post gives you the formula, a table for popular models, and a one-liner to compute it yourself.

The formula

KV cache bytes = 2 × L × H × d × T × 2

Where:

• 2 — one K tensor and one V tensor
• L — number of transformer layers
• H — number of attention heads (or KV heads for GQA models)
• d — head dimension (= hidden_size / num_heads)
• T — sequence length in tokens
• 2 — bytes per value in FP16

That is it. No approximation. This is the exact allocation.

Memory table: popular models

Llama-3-8B (L=32, H=8 KV heads, d=128)

Mistral-7B (L=32, H=8 KV heads, d=128)

Llama-3-70B (L=80, H=8 KV heads, d=128)

Mixtral-8x7B (L=32, H=8 KV heads, d=128)

Note: Mixtral uses the same attention architecture as Mistral-7B per expert; MoE only affects the FFN layers, so KV cache size is identical.

After NexusQuant compression

Llama-3-8B at 128K context (baseline: 16 GB)

Llama-3-70B at 128K context (baseline: 160 GB)

At high preset, a 70B model 128K context KV cache drops from requiring a multi-GPU setup to fitting on a single 80 GB A100.

Python one-liner

kv_gb = lambda L, H, d, T: 2 * L * H * d * T * 2 / 1e9

Examples:

# Llama-3-8B at 128K
print(kv_gb(32, 8, 128, 128_000))   # 16.78 GB

# Llama-3-70B at 128K
print(kv_gb(80, 8, 128, 128_000))   # 167.77 GB

# Mistral-7B at 32K
print(kv_gb(32, 8, 128, 32_000))    # 4.19 GB

How to read this from a HuggingFace config

from transformers import AutoConfig

cfg = AutoConfig.from_pretrained("mistralai/Mistral-7B-v0.1")
L = cfg.num_hidden_layers
H = cfg.num_key_value_heads
d = cfg.hidden_size // cfg.num_attention_heads

T = 128_000
kv_gb = 2 * L * H * d * T * 2 / 1e9
print(f"KV cache at {T//1000}K tokens: {kv_gb:.1f} GB")

The practical takeaway

For a 7B-class model, KV cache at 128K is already 16 GB — as large as a mid-range GPU total memory. At 1M tokens, it is 128 GB. The model weights themselves (in FP16) are only ~14 GB.

For 70B+ models, KV cache dominates entirely. Long-context inference on large models without compression is a multi-GPU problem by default.

NexusQuant makes it a single-GPU problem.

pip install nexusquant-kv

GitHub: https://github.com/jagmarques/nexusquant

Best regards, João Marques

Prerequisites

pip install nexusquant-kv transformers torch datasets

You need a HuggingFace causal LM (any model using split-half RoPE — every Llama, Mistral, Qwen, and Phi variant since 2023).

Step 1: Load your model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_NAME = "mistralai/Mistral-7B-v0.1"  # replace with yours

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()

Step 2: Compute baseline perplexity

import math
from datasets import load_dataset

def compute_ppl(model, tokenizer, text: str, stride: int = 512) -> float:
    encodings = tokenizer(text, return_tensors="pt")
    input_ids = encodings.input_ids.to(model.device)
    seq_len = input_ids.size(1)
    max_len = model.config.max_position_embeddings
    nlls = []
    prev_end = 0
    for begin in range(0, seq_len, stride):
        end = min(begin + max_len, seq_len)
        target_len = end - prev_end
        input_chunk = input_ids[:, begin:end]
        target_chunk = input_chunk.clone()
        target_chunk[:, :-target_len] = -100
        with torch.no_grad():
            loss = model(input_chunk, labels=target_chunk).loss
        nlls.append(loss.item() * target_len)
        prev_end = end
        if end == seq_len:
            break
    return math.exp(sum(nlls) / seq_len)

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
corpus = " ".join(dataset["text"][:200])

baseline_ppl = compute_ppl(model, tokenizer, corpus)
print(f"Baseline PPL: {baseline_ppl:.4f}")

Step 3: Apply NexusQuant at each preset

from nexusquant import nexusquant_evict

for preset in ["high", "balanced", "max"]:
    with nexusquant_evict(model, quality=preset):
        nq_ppl = compute_ppl(model, tokenizer, corpus)
    compression = {"high": "10x", "balanced": "17x", "max": "33x"}[preset]
    delta = ((nq_ppl - baseline_ppl) / baseline_ppl) * 100
    print(f"[{preset:8s}] compression={compression}  PPL={nq_ppl:.4f}  delta={delta:+.2f}%")

Expected output on Mistral-7B:

[high    ] compression=10x  PPL=...  delta=+0.4%
[balanced] compression=17x  PPL=...  delta=+1.3%
[max     ] compression=33x  PPL=...  delta=+2.6%

Step 4: Test on your own text

with open("my_domain_corpus.txt") as f:
    my_text = f.read()

for preset in ["high", "balanced", "max"]:
    with nexusquant_evict(model, quality=preset):
        ppl = compute_ppl(model, tokenizer, my_text)
    print(f"{preset}: {ppl:.4f}")

Domain-specific text (legal, medical, code) may show different degradation curves than wikitext. Always benchmark on text representative of your actual use case.

What to look for

• delta < +1% at high preset → your model is a good fit, safe for production
• delta > +5% at any preset → open an issue with your model name
• balanced is usually the sweet spot — 17x compression for ~1% quality cost

pip install nexusquant-kv

GitHub: https://github.com/jagmarques/nexusquant

Best regards, João Marques

The most useful research I've done this year isn't in the NexusQuant paper. It's the experiments that failed, the ideas that sounded smart in theory and didn't survive contact with real KV cache data.

Negative results build trust. They also save time — if you're working on KV cache compression, this list might save you weeks of effort.

Each entry: what we tried, what we expected, what happened, what we learned.

1. PCA Rotation (3x Worse Distortion)

The idea: Apply PCA to KV vectors to align the quantization axes with the data's principal components. This is optimal for Gaussian data — principal components diagonalize the covariance matrix, and uniform quantization along them minimizes MSE.

What happened: 3x worse distortion than Hadamard rotation. PPL degradation jumped ~0.9 percentage points at the same compression ratio.

Why: PCA is computed per-layer from calibration data. KV distributions shift by layer depth, head index, token position, and input domain. The PCA axes fit the calibration statistics, not the inference statistics. Hadamard rotation, being data-free and orthogonal, spreads energy uniformly without overfitting. It's less optimal in theory but more robust in practice.

Lesson: Data-free rotations outperform data-fitted rotations when distribution shift is unavoidable.

2. Group Size 32 (Compression Ratio Penalty)

The idea: Use group size 32 instead of 8 for per-group scaling. Larger groups reduce scale overhead — from 2 bits/dim (group-8) to 0.5 bits/dim (group-32).

What happened: The quality drop was larger than expected (+0.4% PPL at the same nominal bit rate). The compression improvement was real but smaller than the accounting suggested, because zstd compresses the scale stream efficiently anyway.

Why: Group-32 means one scale covers 4x more variance. After Hadamard, variance is equalized globally but not perfectly at the group level. Group-32 is more sensitive to the residual non-uniformity. The E8 lattice's quantization error grows with scale mismatch.

Lesson: The "free" bits from larger groups aren't free — they trade against quantization accuracy nonlinearly.

3. Adaptive Bitwidth Allocation (Negligible Gain)

The idea: Assign 4-bit quantization to "important" KV vectors (high attention weight) and 2-bit to unimportant ones. Budget the average to 3 bits. Better attention quality where it matters.

What happened: +0.05% PPL improvement over flat 3-bit. Not worth the complexity — requires attention weights at quantization time, adds a routing pass, complicates the bit-packing scheme.

Why: After token eviction, the surviving tokens are already the important ones. The remaining low-importance tokens were mostly evicted. So adaptive bitwidth is redundant — eviction already did the importance filtering.

Lesson: Token eviction and quantization solve related problems. Don't do both; pick one per axis (count vs precision).

4. Per-Head Token Eviction (Catastrophic)

The idea: Each attention head attends to different tokens. Instead of using a consensus importance score across heads, evict different tokens per head based on that head's individual attention pattern.

What happened: +47% PPL degradation. Completely broken.

Why: The KV cache serves all heads simultaneously. If head A evicts token 42 and head B keeps it, you need two separate caches — or you corrupt head B's keys/values. We implemented the wrong version first (corrupting B's cache). The correct implementation requires separate storage per head, which multiplies memory overhead, defeating the compression goal.

Lesson: KV cache is shared infrastructure. Eviction must operate on the shared sequence, not per-head slices.

5. Token Merging (Temporal Predictive Coding) (+107% PPL)

The idea: Instead of evicting tokens, merge similar adjacent tokens into one representative vector. Saves memory while theoretically preserving all information.

What happened: +107% PPL degradation on Mistral-7B at 60% merge rate. Completely unusable.

Why: Token positions carry semantic meaning. Merging token i and i+1 into their mean destroys the positional distinction. When the model generates the next token, it uses Q·K attention over the merged KV, which now corresponds to no specific position. RoPE embeddings make this worse — each position has a distinct rotation, and the mean of two rotated vectors is not a valid rotated vector.

We tried removing RoPE before merging — still +31% PPL. The positional information is baked into the content, not just the rotation.

Lesson: Token position is semantic, not just a coordinate. Merging destroys it.

6. Entropy-Coded Indices Without Delta Coding (2x Compression Loss)

The idea: Apply Huffman or arithmetic coding directly to E8 lattice indices. The index distribution is non-uniform, so entropy coding should compress well.

What happened: ~1.4x compression from the entropy coder alone vs. the expected ~2.5x.

Why: Raw lattice indices have high entropy because each token's KV vector is quantized independently. Adjacent tokens produce different indices even for similar semantic content. The distribution is wide and moderately uniform.

Delta coding first — subtracting each token's index from the previous token's index — produces a distribution tightly peaked at 0 (most tokens change little). Then entropy coding achieves 2.3x. The pipeline order matters enormously.

Lesson: Statistics of compressed representations are not the statistics of the raw data. Profile what you're actually compressing.

7. Outlier-Aware Quantization (Wasted on KV)

The idea: Popular in weight quantization (SmoothQuant, LLM.int8()) — identify outlier channels, scale them separately, quantize the rest normally. Should handle the heavy tails in KV data.

What happened: +0.02% improvement over standard Hadamard + E8. Negligible.

Why: KV cache outliers exist, but Hadamard rotation already spreads them across all dimensions. By the time E8 sees the data, there are no outlier channels — just mildly non-uniform overall magnitude. Outlier-specific treatment addresses a problem that Hadamard already solved.

Lesson: Don't apply weight-quantization techniques to KV quantization without checking whether the problem definition is the same.

8. Low-Rank Key Approximation (Quality Cliff)

The idea: Compute SVD of the key cache, keep the top-r singular values/vectors. Aggressive rank reduction = high compression. No quantization needed.

What happened: Worked well at r ≥ 64 (head_dim=128, so 50% rank). At r=32, quality fell off a cliff: +8% PPL. At r=16: +40% PPL.

Why: KV caches do not have a clean low-rank structure across the time dimension. They can have low-rank structure within a head's channel dimension, but the rank-r approximation confuses these two. Also, SVD is expensive — O(n²r) per layer per generation step.

Lesson: Dimensionality reduction across the token axis and the channel axis are different problems. SVD on the full cache conflates both.

9. FP8 Quantization (Too Naive)

The idea: Just cast KV to FP8 (e5m2 or e4m3). Modern GPUs support FP8 natively. Fast, hardware-accelerated, 2x from FP16.

What happened: +0.8% PPL for only 2x compression. E8 lattice at comparable bit rate is 0.4% PPL with 7-8x compression.

Why: FP8 is 8 bits per element. E8 achieves ~2-3 effective bits per element after entropy coding. FP8 is not a compression technique — it's a lower-precision storage format. The 2x ratio only matches FP16 → FP8, while E8 is targeting FP16 → 2.5 effective bits.

Lesson: Comparing "FP8" to "2-bit E8" conflates precision reduction with compression. They're solving different parts of the same problem.

10. Importance Scoring by L2 Norm (Wrong Signal)

The idea: Evict tokens with small L2 norm — small keys probably don't attract attention.

What happened: +12% PPL at 40% eviction. Much worse than cross-head attention scoring.

Why: L2 norm doesn't predict attention weight. A small-norm key can still attract high attention if the corresponding query also points in that direction. Attention weight is Q·K / √d, not |K|. Using |K| alone ignores Q entirely.

Lesson: Importance is relational (Q, K together), not intrinsic to K alone.

11. Dynamic Eviction During Generation (Latency Catastrophe)

The idea: Re-score and re-evict tokens every N generation steps. Adaptive to changing attention patterns as generation progresses.

What happened: 8x inference latency increase. Throughput collapsed.

Why: Re-scoring requires running the importance computation over the full current KV cache (O(seq²) per step). At 100K tokens, this dominates the generation step. We had it right the first time: evict once after prefill, never again during generation.

Lesson: Prefill and decode phases have fundamentally different compute budgets. Prefill is O(n²). Decode is O(n). Don't move O(n²) work into the decode loop.

12. Calibration Data for Quantization Scales (Overfitting)

The idea: Instead of computing per-group scales from the current KV values, pre-compute scales from a calibration corpus and reuse them. Saves compute at inference.

What happened: +1.9% PPL on out-of-domain data. Scales tuned on Wikipedia performed poorly on code or dialogue.

Why: KV magnitude profiles are domain-specific. Code tokens, dialogue, and factual text have different KV distributions at different layers. Per-inference scaling, though slightly more expensive, adapts to the actual data being compressed.

Lesson: KV cache is input-dependent. Calibration data solves the wrong version of the problem. TurboQuant's approach (calibration for weight quantization, not KV) is fine because weights don't change with input.

What Actually Works

After 12 failures, the surviving pipeline is:

1. Cross-head attention importance scoring → token eviction (2.5x from fewer tokens)
2. RoPE removal on keys
3. Hadamard rotation
4. E8 lattice VQ, group-8, per-group FP16 scales (3-8x from fewer bits)
5. Delta coding → zstd (2-3x from entropy coding)

Five stages, each justified by a measurable improvement that survived ablation. Everything else, we tried and measured and deleted.

The cleaner version with just eviction + E8 (no entropy coding) still achieves 10x at +0.4% PPL. That's the "high quality" preset. The full pipeline gets to 33x.

Repo: github.com/jagmarques/nexusquant

pip install nexusquant-kv

Best regards, João Marques

Standard scalar quantization — what every LLM quantizer from GPTQ to AWQ does — rounds each number independently to the nearest representable value. E8 lattice quantization rounds groups of 8 numbers jointly to the nearest point on a mathematical lattice. The difference sounds subtle. It isn't.

This post is a complete walkthrough of how E8 quantization works, why it beats scalar quantization by ~30% in distortion, and exactly what the algorithm does line by line.

Why Lattices?

The core problem in quantization is sphere packing. You want to cover n-dimensional space with the fewest representable points, such that any real vector is "close" to at least one codebook entry.

For 1D scalar quantization, you're placing points on a number line. Easy — evenly space them.

For 8D vector quantization, you want to pack 8D balls as densely as possible. The densest known packing in 8 dimensions is the E8 root lattice, proven optimal by Maryna Viazovska in 2016 (she won the Fields Medal for it in 2022). Its packing density is 2x better than any irregular arrangement.

What this means for quantization: for the same number of bits, E8 introduces 30% less distortion than optimal scalar quantization, and about 1.4 dB better signal-to-noise ratio.

The E8 Lattice, Briefly

The E8 lattice is the set of all 8D vectors that satisfy either of these two conditions:

1. All coordinates are integers, and their sum is even
2. All coordinates are half-integers (x.5 values), and their sum is an integer (even or odd)

That's the whole definition. A lattice, not a codebook. No learned vectors, no storage.

The packing radius is 1/√2. Every point in ℝ⁸ is within distance 1/√2 of some E8 lattice point.

The Conway-Sloane Nearest-Point Algorithm

Finding the nearest E8 point to an arbitrary input x is what you need for quantization. The algorithm, from Conway and Sloane (1982), is:

1. Find the nearest point in each of the two cosets separately
2. Return whichever is closer to x

Coset 1: Integer lattice with even coordinate sum (D8)

Round each coordinate to the nearest integer. If the sum is already even, done. If odd, flip the coordinate where x[i] was farthest from its rounded value (i.e., we pay the smallest extra cost to make the sum even).

Coset 2: Half-integer lattice (D8 + (0.5,...,0.5))

Subtract 0.5 from x, round each coordinate, add 0.5 back. Apply the same even-sum correction.

Pick the winner:

Compute squared distance to each coset's candidate. Return the closer one.

Python Implementation

Here's the full nearest_point function from NexusQuant's e8_lattice.py:

import torch

def nearest_point(x: torch.Tensor) -> torch.Tensor:
    """Find nearest E8 lattice point to each 8D vector.

    Args:
        x: (..., 8) tensor
    Returns:
        Nearest E8 lattice point, same shape
    """
    # --- Coset 1: integer lattice with even coordinate sum ---
    r_int = x.round()
    sums = r_int.sum(dim=-1)
    odd = (sums % 2 != 0)

    if odd.any():
        # Find the coordinate we can flip at minimum cost
        gaps = (x - r_int).abs()
        idx = gaps[odd].argmin(dim=-1)          # cheapest coordinate to adjust
        fix = torch.zeros_like(r_int[odd])
        fix.scatter_(-1, idx.unsqueeze(-1), 1.0)
        # Which direction to flip? Toward x, not away
        sign = ((x[odd] - r_int[odd]).gather(-1, idx.unsqueeze(-1)) >= 0).float() * 2 - 1
        r_int[odd] = r_int[odd] + fix * sign

    # --- Coset 2: half-integer lattice ---
    r_half = (x - 0.5).round() + 0.5
    sums_h = r_half.sum(dim=-1)

    # NOTE: strict E8 requires even integer sum for the half-integer coset.
    # We intentionally relax this — see the "Relaxed Parity" section below.
    odd_h = ((sums_h * 2).round() % 2 != 0)

    if odd_h.any():
        gaps_h = (x - r_half).abs()
        idx_h = gaps_h[odd_h].argmin(dim=-1)
        fix_h = torch.zeros_like(r_half[odd_h])
        fix_h.scatter_(-1, idx_h.unsqueeze(-1), 1.0)
        sign_h = ((x[odd_h] - r_half[odd_h]).gather(-1, idx_h.unsqueeze(-1)) >= 0).float() * 2 - 1
        r_half[odd_h] = r_half[odd_h] + fix_h * sign_h

    # --- Pick closer coset ---
    d_int  = ((x - r_int)  ** 2).sum(dim=-1)
    d_half = ((x - r_half) ** 2).sum(dim=-1)

    res = r_int.clone()
    res[d_half < d_int] = r_half[d_half < d_int]
    return res

The full quantizer wraps this with per-group scaling:

def quantize(x: torch.Tensor, levels: int = 8) -> torch.Tensor:
    """Quantize tensor with E8 lattice VQ and per-group scaling.

    levels: 4 = 2-bit, 8 = 3-bit, 16 = 4-bit
    """
    shape = x.shape
    pad = (8 - shape[-1] % 8) % 8
    if pad > 0:
        x = torch.nn.functional.pad(x, (0, pad))

    flat = x.reshape(-1, 8)
    amax = flat.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8)
    sc = amax / (levels / 2)          # scale each 8D group to [-levels/2, levels/2]

    lp = nearest_point(flat / sc)     # find nearest E8 point in normalized space
    lp = lp.clamp(-levels / 2, levels / 2)
    result = (lp * sc).reshape(x.shape)

    if pad > 0:
        result = result[..., :shape[-1]]
    return result.reshape(shape)

Why Per-Group Scaling?

Raw KV values span a wide dynamic range. A single global scale would waste most of the representable range on the quiet regions and saturate the loud ones.

Per-group scaling (one FP16 scale per 8 values) costs 2 bytes / 8 values = 2 bits per value of overhead — the same as the lattice encoding itself. This doubles the effective bits. But the Hadamard rotation applied upstream equalizes variance across dimensions, so within each 8D group the dynamic range is small. The overhead is worth it.

Alternatively, quantize_perhead uses one scale per full 128D head vector. This cuts scale overhead to 0.125 bits/dim at a cost of ~0.07% more PPL degradation. The headroom depends on how well Hadamard equalizes your specific data.

The Relaxed Parity Discovery

This is the part that surprised us.

Strict E8 theory says the half-integer coset must also have an even sum. When we first implemented this and tested on actual KV cache data, the strict version performed worse than the relaxed version — by 0.3-0.4% PPL.

Why? KV cache distributions are sub-Gaussian (lighter tails than Gaussian, more mass near zero). The strict parity constraint over-regularizes the half-integer coset for this regime. Relaxing it effectively adds beneficial dithering: the quantizer can reach slightly off-lattice half-integer points that happen to sit closer to the true KV values.

The original E8 construction targets uniform or Gaussian distributions. Real transformer attention data is neither. The relaxed constraint is a free 0.3% quality improvement, just by removing three lines of enforcement code.

This is the kind of result that only shows up when you measure empirically rather than trusting the textbook.

Bit Rate Accounting

At 3 bits per dimension with per-group scales (group size 8):

• Lattice codes: 3 bits × 8 dims = 24 bits per group
• Scale (FP16): 16 bits per group
• Total: 40 bits / 8 dims = 5 bits per dimension raw storage

After delta coding + zstd compression on the lattice indices (consecutive tokens produce similar codes; deltas compress 2-3x):

• Effective rate: ~2 bits per dimension

That's 16 bits FP16 → 2 bits effective = 8x compression from quantization alone. Combined with 2.5x token eviction: ~17x total (the "balanced" preset).

Running It

pip install nexusquant-kv

from nexusquant.core.e8_lattice import E8Lattice
import torch

# Quantize any tensor — works for arbitrary shape
x = torch.randn(100, 128)  # 100 KV vectors, 128-dim heads
q = E8Lattice.quantize(x, levels=8)  # 3-bit E8

print(f"MSE: {((x - q)**2).mean():.6f}")
print(f"Relative error: {((x - q)**2).sum() / (x**2).sum() * 100:.2f}%")

# Or per-head scaling (less overhead, slightly more error)
q2 = E8Lattice.quantize_perhead(x, levels=8)

What Makes This Hard in Practice

Three things the textbook doesn't tell you:

1. The Hadamard rotation is load-bearing. Without it, KV vectors have heavy outliers in specific dimensions (the same phenomenon that makes LLM quantization hard). E8 performs poorly on unrotated data. Hadamard spreads the energy uniformly so the per-group scale assumption holds.

2. RoPE removal matters. Rotary position embeddings make adjacent tokens' keys look "rotated" relative to each other. Removing RoPE before quantization reduces inter-token correlation and cuts quantization error ~0.7% PPL.

3. GPU kernels are unfinished. The current implementation dequantizes immediately for compatibility. Real memory savings require keeping the compressed representation on-device, which needs a Triton kernel to pack/unpack E8 codes on the fly. That's the main engineering gap.

Repo: github.com/jagmarques/nexusquant

Best regards, João Marques

KV cache compression numbers like "10x" sound impressive in a paper. But what does that mean in practice, for a real GPU, serving real users? Let me give you a concrete memory calculator so you can answer this for your own setup.

The KV Cache Formula

For any transformer, the KV cache size is:

KV_bytes = 2 × num_layers × num_heads × head_dim × seq_len × bytes_per_element

The 2 is for keys AND values. bytes_per_element is 2 for FP16, 4 for FP32.

For Mistral-7B (32 layers, 8 KV heads, head_dim=128, FP16):

KV_bytes = 2 × 32 × 8 × 128 × seq_len × 2
         = 131,072 × seq_len bytes
         ≈ 128 KB per token

At 128K tokens: 128 KB × 131,072 = 16.7 GB just for the KV cache.

GPU Memory Table

Here's what that means on real hardware:

Numbers leave 8 GB headroom for model weights and activations on 7B models.

Practical Scenarios

Scenario A: A10G serving 128K context

Without NexusQuant: 128K × 128 KB = 16.7 GB. With only 24 GB total and ~14 GB for the 7B model, you're OOM before the first token generates.

With NexusQuant 17x preset: 16.7 / 17 = 0.98 GB for KV. Fits comfortably. You can now serve 128K context on a single A10G.

Scenario B: A100 80GB, maximize throughput

A single request at 128K takes 16.7 GB KV. With 80 GB total and ~14 GB for weights, you have ~66 GB for KV — enough for about 4 concurrent sessions at 128K.

At 17x compression, each session uses ~1 GB of KV. You can serve ~66 concurrent sessions instead of 4. That's 16x more users per GPU.

Scenario C: Long context research

You want 1M token context on Mistral-7B. KV cache = 1M × 128 KB = 128 GB. Impossible on a single GPU.

At 17x: 7.5 GB. Fits on a single A100 40GB alongside the model weights.

Python Calculator

def kv_cache_gb(
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    seq_len: int,
    dtype_bytes: int = 2,  # FP16
) -> float:
    """Returns KV cache size in GB."""
    total_bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes
    return total_bytes / (1024 ** 3)


def savings_report(model_name: str, num_layers: int, num_kv_heads: int,
                   head_dim: int, seq_len: int, gpu_vram_gb: float,
                   model_weights_gb: float) -> None:
    base_gb = kv_cache_gb(num_layers, num_kv_heads, head_dim, seq_len)
    available = gpu_vram_gb - model_weights_gb

    print(f"\n=== {model_name} @ {seq_len:,} tokens ===")
    print(f"Base KV cache: {base_gb:.1f} GB")
    print(f"GPU available for KV (after weights): {available:.0f} GB")

    for ratio, label in [(10, "high quality"), (17, "balanced"), (33, "max")]:
        compressed_gb = base_gb / ratio
        concurrent = int(available / compressed_gb)
        fits = "YES" if base_gb / ratio < available else "NO"
        print(f"  {ratio}x ({label}): {compressed_gb:.2f} GB | fits={fits} | concurrent sessions={concurrent}")


# Mistral-7B examples
savings_report("Mistral-7B", 32, 8, 128, 128_000, 80, 14)
savings_report("Mistral-7B", 32, 8, 128, 1_000_000, 80, 14)

# Llama-3-8B
savings_report("Llama-3-8B", 32, 8, 128, 128_000, 40, 16)

# Llama-3-70B (multi-GPU: assume 160 GB total)
savings_report("Llama-3-70B", 80, 8, 128, 128_000, 160, 140)

Sample output:

=== Mistral-7B @ 128,000 tokens ===
Base KV cache: 16.7 GB
GPU available for KV (after weights): 66 GB
  10x (high quality): 1.67 GB | fits=YES | concurrent sessions=39
  17x (balanced): 0.98 GB | fits=YES | concurrent sessions=67
  33x (max): 0.51 GB | fits=YES | concurrent sessions=129

=== Mistral-7B @ 1,000,000 tokens ===
Base KV cache: 130.4 GB
GPU available for KV (after weights): 66 GB
  10x (high quality): 13.04 GB | fits=YES | concurrent sessions=5
  17x (balanced): 7.67 GB | fits=YES | concurrent sessions=8
  33x (max): 3.95 GB | fits=YES | concurrent sessions=16

=== Llama-3-8B @ 128,000 tokens ===
Base KV cache: 16.7 GB
GPU available for KV (after weights): 24 GB
  10x (high quality): 1.67 GB | fits=YES | concurrent sessions=14
  17x (balanced): 0.98 GB | fits=YES | concurrent sessions=24
  33x (max): 0.51 GB | fits=YES | concurrent sessions=47

The Model Weight Overhead

One thing papers often gloss over: the model itself takes memory. Here are real-world numbers:

At 70B+, KV cache compression becomes less about fitting the model and more about fitting the context. A 70B model with 128K context uses only ~2x its own weight in KV cache. NexusQuant is still valuable here, but the math changes — you're extending context headroom, not primarily enabling the model to load.

What We Measured

All numbers in NexusQuant are validated with torch.cuda.memory_allocated() before and after cache materialization. No estimate-only ratios. The paper's compression column includes all overhead: scales (FP16), lattice indices (packed uint8), and metadata. If it's not in memory, it's not in the ratio.

Repo: github.com/jagmarques/nexusquant

pip install nexusquant-kv

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced"):  # 17x
    output = model.generate(input_ids, max_new_tokens=512)

Best regards, João Marques

Install

pip install nexusquant

Requires Python 3.9+, PyTorch 2.1+, and Transformers 4.40+. No CUDA-specific wheels — it runs on CPU for small models and on CUDA for production workloads.

The one-liner

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced"):
    output = model.generate(input_ids, max_new_tokens=500)

That is it. The context manager hooks into the model's forward pass, intercepts the KV cache after prefill, compresses it, and restores the original hooks on exit.

Quality presets

Three presets, pick based on your use case:

# Conservative: 35% eviction, 10x compression
# Use for: general-purpose, long prompts (>1K tokens), production default
with nexusquant_evict(model, quality="conservative"):
    ...

# Balanced: 60% eviction, 16x compression
# Use for: RAG over structured documents, short-to-medium prompts (<1.5K tokens)
with nexusquant_evict(model, quality="balanced"):
    ...

# Aggressive: 80% eviction, 32x compression
# Use for: memory-constrained environments, factual recall only
# Do NOT use for: prompts >1K tokens, creative tasks, multi-detail reasoning
with nexusquant_evict(model, quality="aggressive"):
    ...

# Lossless: no eviction, quantization only
# Use for: when you need maximum quality, 6-7x compression
with nexusquant_evict(model, quality="lossless"):
    ...

How to choose eviction rate for your use case

Do not guess. Test on your actual data. Here is a script:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from nexusquant import nexusquant_evict

def measure_ppl(model, tokenizer, texts, quality):
    total_loss = 0
    count = 0
    for text in texts:
        ids = tokenizer(text, return_tensors="pt").input_ids.cuda()
        with nexusquant_evict(model, quality=quality):
            with torch.no_grad():
                out = model(ids, labels=ids)
        total_loss += out.loss.item()
        count += 1
    return (total_loss / count)

# Load your model
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1").cuda()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

# Sample 20 texts from YOUR domain
your_texts = [...]  # replace with real samples

# Baseline
baseline_loss = measure_ppl(model, tokenizer, your_texts, quality="lossless")

# Test presets
for quality in ["conservative", "balanced", "aggressive"]:
    loss = measure_ppl(model, tokenizer, your_texts, quality=quality)
    delta_pct = (loss - baseline_loss) / baseline_loss * 100
    print(f"{quality}: PPL delta = {delta_pct:+.2f}%")

Run this with 20-50 samples from your actual domain. If "balanced" gives > 2% PPL delta on your data, drop to "conservative".

Domain sensitivity: what to watch for

From our experiments, here is what correlates with compression difficulty:

Compresses well (use balanced or aggressive):

• Dense factual prose (academic papers, encyclopedia entries)
• Structured technical documentation
• Formal legal or financial text

Compresses poorly (use conservative or lossless):

• Creative/narrative text (fiction, stories)
• Casual conversational text
• Code with unusual identifier names
• Mixed-language text

The attention patterns on creative text are more diffuse — fewer tokens dominate — so eviction hurts more. The quantization also hurts more because KV values for creative text have less structured distribution.

Here are our measured numbers at 500-token prefix on Mistral-7B:

Note that creative text at 70% eviction is only slightly worse than 35% eviction (4.62% vs 2.48%). The first eviction step hurts creative text proportionally more than subsequent steps.

Memory measurement

Do not trust our reported ratios. Measure yourself:

import torch

def measure_kv_memory(model, input_ids, quality=None):
    torch.cuda.reset_peak_memory_stats()
    baseline_mem = torch.cuda.memory_allocated()

    if quality:
        with nexusquant_evict(model, quality=quality):
            with torch.no_grad():
                out = model(input_ids, use_cache=True)
    else:
        with torch.no_grad():
            out = model(input_ids, use_cache=True)

    compressed_mem = torch.cuda.memory_allocated()
    kv_mem = compressed_mem - baseline_mem
    return kv_mem

baseline = measure_kv_memory(model, input_ids, quality=None)
compressed = measure_kv_memory(model, input_ids, quality="balanced")
print(f"Compression ratio: {baseline / compressed:.1f}x")

If the ratio you measure differs substantially from what we report, please open a GitHub issue with your model and config.

The latency caveat

NexusQuant is currently CPU-bound on the compression step.

The pipeline compresses the KV cache after prefill. The bottleneck is the E8 VQ nearest-neighbor lookup and the zstd entropy coding, both of which run on CPU. On a Mistral-7B prefill of 1664 tokens:

• Prefill: ~180ms (GPU)
• Compression step: ~340ms (CPU)
• Total: ~520ms vs ~180ms baseline

This means NexusQuant currently makes your time-to-first-token slower, not faster. The compression saves memory (which enables larger batches or longer contexts), but it adds latency.

The fix is Triton kernels for the VQ and entropy coding steps. We have not written them yet. This is on the roadmap and we will post an update when it is done.

If your use case is memory-bound (fitting more users in GPU memory, extending context length beyond what fits otherwise), NexusQuant solves that today. If your use case is latency-bound (fastest possible TTFT), do not use NexusQuant until the Triton kernels ship.

Context length limits

Validated context ranges:

We have not validated eviction on prefixes longer than ~3K tokens beyond the catastrophic failure at 60%. For long-context applications (>3K token prefixes), use quality="lossless" to get quantization-only compression without eviction.

Model compatibility

Validated:

• Mistral-7B-v0.1 (MHA)
• Llama-3-8B (GQA)

Known issues:

• GPT-NeoX-style models: our RoPE removal assumes split-half rotation (Llama/Mistral style). GPT-NeoX uses interleaved RoPE. It will produce wrong results. Do not use on GPT-NeoX.
• Llama-3.1 with extended context: rope_scaling config is not fully handled for context lengths beyond the standard window.
• Batch size > 1: there is a bug in NexusQuantSimple where only the first batch element is processed for keys. The HuggingFace context manager (nexusquant_evict) handles this correctly. Use that, not the low-level API.

What is not there yet

Being direct about gaps:

1. Triton kernels — compression is CPU-bound, adds ~340ms latency. Critical for production.
2. 16K+ context — not validated above 3K token prefixes.
3. Eviction for batch > 1 — the low-level API has a bug here; context manager is fine.
4. LongBench — proper long-context benchmark not yet run.
5. Multi-model presets — the quality presets are tuned for Mistral/Llama. Other architectures may need different defaults.

The core use case that works well today: Mistral or Llama family model, prefill up to ~1.7K tokens, memory-bound deployment (fitting more requests per GPU), with quality=conservative or quality=balanced.

Best regards, João Marques

This post is an honest head-to-head. For each competitor: what they do, their reported numbers from their papers, our numbers, where we win, and where they win.

The comparison table

None of these numbers are from our experiments. KVTC, TurboQuant, CommVQ, and Palu numbers are from their papers. We have not run their code on our data. That is a limitation we are transparent about.

TurboQuant (Google)

What it does: Scalar quantization with learned per-channel scale factors. 193 lines of code, training-free at the core. The simplest competitive approach.

Their reported numbers: Near-zero quality degradation at 5.3x compression on Llama-class models.

Our numbers at the same point: NexusQuant at 6.71x on Llama-3-8B shows -1.20% PPL (a quality improvement). At 5.3x it would be better still.

Where TurboQuant wins: Quality at low compression (below 6x). If you need minimal quality loss and 4-6x compression, TurboQuant is extremely competitive and simpler than our pipeline.

Where we win: Compression beyond 6x. TurboQuant does not have a token eviction component, so it cannot reach 10-16x ratios. We extend that range significantly while staying training-free.

The honest caveat: We have not run TurboQuant on our exact dataset with our exact evaluation setup. The 193-line simplicity is genuinely elegant. We have a lot of respect for this work.

KVTC (NVIDIA)

What it does: KVTC combines scalar quantization with a temporal coherence coding approach, compressing differences between adjacent KV states. Requires ~10 minutes of calibration on sample data.

Their reported numbers: Up to 20x compression with less than 1 perplexity point degradation.

Our numbers at 20x: We have validated 16.6x at +0.82% PPL. At 20x (80% eviction on 1664-token prefix), we get +2.13% PPL. KVTC appears to achieve better quality at the extreme end.

Where KVTC wins: Quality at 20x+ compression. Their temporal coding approach is genuinely effective. And 20x > 16.6x — they reach higher ratios with better quality in that regime.

Where we win: No calibration required. The 10-minute calibration step sounds trivial but matters in practice — it requires representative data, it adds a deployment step, and it means the system is not truly drop-in. Our with nexusquant_evict(model): requires no data, no calibration, nothing except the model.

The honest caveat: KVTC's 20x result is the competition's strongest. We are not the best at maximum compression.

CommVQ (Apple)

What it does: Trains a vector quantization codebook on KV cache data with an EM-style training loop. Uses a commutative design that works naturally with RoPE embeddings.

Their reported numbers: ~8x compression with near-zero quality degradation. The commutative approach is elegant — they avoid the RoPE removal issue entirely by designing the quantizer to commute with the rotation.

Our numbers at 8x: We reach 10.1x at +0.90% (500-tok prefix) or 10.4x at +0.14% (1664-tok prefix). Quality at pure 8x would be even better.

Where CommVQ wins: Quality. If you can afford training time and have representative KV cache data, a trained codebook will beat a training-free approach. Also their RoPE handling is mathematically cleaner than our blanket RoPE-removal strategy.

Where we win: Training-free. CommVQ requires a training loop. Our pipeline works on any model from the moment you install the package. No training data, no training time.

The honest caveat: We solve RoPE by removing it before quantization and reapplying after. CommVQ solves it properly at the algebra level. If we revisit this design, their approach is the right one.

Palu

What it does: Low-rank projection of the KV cache heads, combining PCA-based dimensionality reduction with quantization. Needs calibration data to compute the PCA rotation.

Their reported numbers: 11.4x compression at ~+1.19% PPL. The low-rank projection is a different approach — instead of quantizing all dimensions, they reduce the number of dimensions.

Our numbers at comparable compression: 10.4x at +0.14% PPL (1664-tok prefix). Our quality is substantially better at similar compression.

Where Palu wins: If you have calibration data and want a single-architecture low-rank approach, Palu integrates cleanly with fine-tuning workflows.

Where we win: Quality at 10-11x. +0.14% vs +1.19% is a significant gap. Also training-free for the base pipeline.

The main competitive claim

We are the best training-free option above 6x compression. Below 6x, TurboQuant matches or beats us with a simpler system. Above 6x and up to ~16x, we are the only training-free approach with validated sub-1% quality degradation.

If you need to go to 20x+, KVTC is currently ahead (with calibration). If you need near-zero quality loss at 8x and can train, CommVQ is the right choice.

None of this is cherry-picked. The numbers above are the numbers.

What we cannot claim

We have not run any competitor's code on our exact dataset. All competitor numbers come from their papers. It is possible that on wikitext-2 with our specific passage selection, TurboQuant or KVTC would show different numbers than reported. We are aware of this limitation and plan to run a proper head-to-head in our NeurIPS submission.

We also cannot claim "beats TurboQuant" in any general sense. We beat their reported numbers at high compression ratios. At low compression, their simplicity is competitive.

Best regards, João Marques

We chose all of them. This post is every benchmark result we have, including the ones that did not work.

The pipeline

Quick context. NexusQuant compresses the KV cache of transformer models at inference time, training-free:

Prefill → Key-Key Attention Score → Evict → RoPE-remove → Hadamard → 2-bit E8 VQ → Temporal Delta → zstd

The context manager API:

with nexusquant_evict(model, quality="balanced"):
    output = model.generate(input_ids, max_new_tokens=200)

All numbers below are from an A10G GPU (24 GB). Perplexity delta is measured against the uncompressed baseline on the same passages.

Mistral-7B: the full picture

These are our numbers at different prefix lengths and eviction rates. Every row is real.

The 3K catastrophe is real and we are not hiding it. At 2924-token prefixes, evicting 60% or more of KV tokens causes perplexity to blow up to +42% and beyond. This is not a scorer bug. We tried four different attention-score variants; all fail at this eviction rate for this prefix length. It is a fundamental capacity loss: the model simply cannot attend to the right tokens when that much context is gone.

If you are deploying NexusQuant on prompts longer than ~2K tokens, use quality="conservative" (35% eviction maximum).

Llama-3-8B: the surprise

We validated on a second architecture to make sure Mistral results were not flukes. What we found was unexpected.

All four configurations have negative perplexity delta. Compression is making the model perform better than the uncompressed baseline on the wikitext-2 distribution.

Our hypothesis: Llama-3-8B uses grouped-query attention (GQA). The compression and eviction act as a regularizer that suppresses noise in the attention pattern. The GQA architecture appears to have redundancy in its KV cache that quantization removes beneficially. This is a standalone publishable finding that we did not anticipate.

Domain sensitivity

Not all text compresses equally. We tested three text domains at 500-token prefixes on Mistral-7B:

Academic text (dense factual prose, repetitive structure) compresses best. Creative/narrative text is the hardest — at 35% eviction you already lose 2.48%, compared to 0.39% on academic text.

If you are building a RAG system over structured documents, the 35% eviction preset works well. If you are doing creative writing assistance or summarising fiction, use quality="lossless" (no eviction) or test your specific domain before deploying.

Downstream task evaluation (Mistral-7B, 5 tasks)

Perplexity is a proxy. We ran five QA-style downstream tasks with and without compression.

"MATCH" means the compressed model gave the same answer as the uncompressed model. "PARTIAL" means it got the main point but missed a specific detail.

Factual recall is fully preserved at 10x. Nuanced multi-detail questions lose some specificity at both compression levels. Single-hop reasoning holds at 10x but starts slipping at 16x.

What failed (12 approaches we killed)

We tried things that did not work. Here they are:

Variable-rate E8 (per-group water-filling): Allocate more bits to high-energy groups. Result: +3.67% PPL at matched rate — 3.3x worse than fixed-rate. Energy does not equal importance.

Cross-layer KV prediction: Predict layer L+1 from layer L. Result: 0.0 bits saved. Layers project to orthogonal subspaces.

L2 norm as token importance proxy: Use L2 norm to select which tokens to keep. Result: +207% PPL. L2 norm and attention importance are completely different quantities.

Strict E8 parity enforcement: Fix our half-integer parity to be mathematically correct E8. Result: 0.3-0.4% worse quality. The relaxed parity acts as dithering on sub-Gaussian KV data.

Token zeroing for eviction: Set evicted KV to zero. Result: catastrophic attention mass stealing. exp(Q @ 0) = exp(0) = 1 in softmax; evicted tokens steal attention weight. Fix: -inf masking.

Eviction on small models (TinyLlama 1.1B): 80-90% eviction rate. Result: +322% PPL at 1K tokens, +1195% at 2K. TinyLlama's top 10% of tokens capture only 28% of attention mass vs. 65-88% on Mistral-7B. Eviction only works when attention is sharp enough.

Honest compression numbers

We made an early mistake: reporting compression ratios without scale overhead. Here is the honest table.

The 16-bit scale per 8-element group adds 2 bits/element overhead. At 3 bits/element stored, the real cost is 5 bits/element, not 3. We got burned by ignoring this early on. Now every ratio is measured with torch.cuda.memory_allocated() before and after.

The number that does not exist

One number appeared in an early draft of our paper that cannot be traced to any experiment log: +0.339% at 8x for Mistral-7B. Our validated 50-passage result for that config is +0.47%. We do not know where +0.339% came from. It is not in any research file. It is gone from the paper.

This kind of thing happens. You run an experiment, write a number down, run more experiments, and the earlier number stops being reproducible. If you cannot trace a number to a specific run with a specific config, it does not belong in a paper.

What is still missing

For full honesty:

• No LongBench results yet (needs multi-hour GPU run on proper dataset)
• No latency benchmarks (CPU-bound compression loop, needs Triton kernels)
• No 16K+ context validation
• Downstream evals above are small-N; full MMLU/GSM8K runs in progress

These are gaps we are filling. The numbers above are what we have validated.

Best regards, João Marques

The data

Same model (Mistral-7B), same method, same eviction rate. Only prefix length changes:

At 1,600 tokens, 60% eviction gives +0.82%. At 500 tokens, same rate gives +4.5%. A 5-6x quality improvement from more context.

Why

The importance scorer ranks tokens by attention weight. At 500 tokens, ~30 query positions — attention is flat. At 1,600 tokens, ~100 query positions — attention concentrates on genuinely important tokens.

More data → sharper importance estimates → safer eviction.

What it means

Production inference runs at thousands of tokens. Short-context benchmarks understate eviction-based compression.

NexusQuant achieves:

• 10x at 500-token context (<1% PPL)
• 17x at 1,600-token context (<1% PPL)
• 33x at any length (~2.5% PPL)

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced"):
    output = model.generate(long_input_ids, max_new_tokens=512)

GitHub · Paper

Best regards, João Marques

The E8 lattice quantizes 8 numbers at once. Result: 3x better compression under entropy coding at the same distortion.

Scalar quantization

Scalar INT2 rounds each of 128 values independently to 4 levels. Indices are near-uniform, so entropy coding barely helps — maybe 1.2x.

E8: quantize 8 at a time

The E8 root lattice is the densest sphere packing in 8 dimensions. Split each 128-dim vector into 16 groups of 8, snap each to the nearest E8 point.

from nexusquant.core.e8_lattice import E8Lattice

groups = vector.reshape(-1, 8)
lattice_points = E8Lattice.nearest_point(groups)

The entropy advantage

3x advantage from lattice structure alone.

Handling outliers

Raw KV vectors are heavy-tailed. Fix: Hadamard rotation first.

from nexusquant.core.hadamard import hadamard_matrix

H = hadamard_matrix(128)
rotated = vector @ H.T

Orthogonal, no information loss. After rotation, E8 at 2 bits/dim causes less than 0.1% PPL degradation.

NexusQuant on GitHub · Paper

Best regards, João Marques

I built NexusQuant, a library that compresses the KV cache 10-33x at inference time. No training, no calibration data, no model changes.

Before

# OOM at 32K tokens on a 24GB GPU
output = model.generate(input_ids, max_new_tokens=512)

After

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced"):
    output = model.generate(input_ids, max_new_tokens=512)

128K context now fits in the memory that used to hold 7.5K.

How it works

1. Rank tokens by attention importance
2. Drop lowest-scoring tokens
3. Undo rotary position embeddings on keys
4. Hadamard rotation to spread energy uniformly
5. E8 lattice quantization (2 bits/dim)
6. Delta coding + zstd compression

Numbers

Mistral-7B, A100, FP16, all overhead included:

Comparison

GitHub · Paper

Best regards, João Marques