LookAtMyQuant 2 is a quantization compiler that squeezes Gemma 4 E2B down to the size of a typical 4-bit file while preserving much more of the original model’s behavior. Measured by KL divergence from the uncompressed model, it comes out 12 times closer than llama.cpp’s Q4_K_M at the same file size. On GSM8K math problems it scores 59%, where Q4_K_M manages 50% and the full model reaches 61%.
The improvement comes from treating a model as a collection of parts that tolerate compression very differently.
Why a single recipe wastes bytes
Standard quantized files apply one format and one bit width almost everywhere, with a few hand-picked exceptions. Take Q4_K_M in the GGUF family: most matrices sit near 4.5 bits per weight, while a set list of tensors gets the higher-precision Q6_K. Nothing in that rule changes with model size, vocabulary or target hardware.
Real models don’t behave that uniformly. In Gemma 4 E2B, a single small matrix with 0.4 million weights hurts quality at 3 bits about as much as twenty complete layers would. Half of all weights live in a lookup table that copes fine with 4 bits. And the most common 256 tokens account for one third of ordinary text, whereas the least common half of the vocabulary barely registers.
How the quantization compiler decides
The quantization compiler tests what happens to the whole model when one stored piece alone is compressed. Pieces include each weight matrix, groups of embedding rows sorted by how often their tokens occur, and the key-value cache of every layer. A few candidate codes get measured per piece and the rest are predicted. Then it picks, for each piece, the code that adds the least total damage while the model still meets the size or speed target of the chosen device.
Codes are written as short programs in a grammar of tensor-network rewrites. The authors say nine previously published quantization and pruning methods can be expressed in that grammar and are reproduced exactly.
The lookup table that pays for everything
Gemma 4 E2B hands every one of its 35 layers a vector, 256 values wide, that is fetched by token. Stacked together, those vectors make a table with 262,144 rows and 8,960 columns. It accounts for 2.35 billion of the 4.63 billion weights in the text model. Nothing multiplies this table; the model just reads one row per token.
That makes it inexpensive to compress. Q4_K_M keeps it at Q6_K, or 6.56 bits per weight, which takes 1.93 GB and 56% of the whole file. The compiler, limited to the same llama.cpp formats, picks IQ4_XS at 4.25 bits. The 0.68 GB freed this way goes to the matrices, and most of them end up at Q8_0 or BF16.
Results against public 4-bit files
The test set included Google’s quantization-aware-trained q4_0 file, unsloth’s UD-Q4_K_XL, the Q4_K_M from bartowski and the 4-bit MLX build from LM Studio, along with llama.cpp’s own recipes from Q2_K up to Q8_0. Each file was turned back into full-precision weights so one script could score them all on identical text. That text mixed WikiText-2 with held-out samples drawn from 13 sources, covering English, ten other languages, code and chat.
The advantage showed up on every source. On WikiText-2, which wasn’t in the calibration data, a compiled file of 3.39 GB posts a KL of 0.008. For Q4_K_M it is 0.143.
Multiple-choice tests barely separate the files. Excluding Q2_K and Google’s QAT file, all of them stay about two points from the original on MMLU, HellaSwag and ARC-Challenge. GSM8K is where the gap opens: public 4-bit files drop as many as 27 points, while compiled files drop no more than 6. With only 250 problems, each GSM8K number could be off by around 6 points either way.
Trellis codes for Apple Silicon
The compiler’s own menu also includes trellis and Lloyd-Max codes, which run on Apple Silicon through the team’s MLX runtime. Using both, a 2.00 GB plan reaches KL 0.074, better than any public file smaller than 3.5 GB, and answers 60.4% of GSM8K problems correctly. A 3.41 GB plan brings KL down to 0.0057.
Different models, different devices
Qwen3.5-4B reuses one embedding table, with 248,320 rows, for reading input tokens and for producing output. Kept at 16 bits, that table weighs roughly the same as every matrix stored at 3 bits. Matching the size of a model that pairs 3-bit matrices with a 16-bit embedding, the compiler averages 3.1 bits on the embedding and spends 5.35 on the matrices. Perplexity climbs 0.13% rather than 4.9%, while KL drops to one twelfth.
Inside that table the top 256 rows keep 16 bits, the following 8,000 get 8, and the least used 117,000 get 2. Another 183,000 rows belong to tokens absent from WikiText’s training split, and squeezing all of them to 2 bits costs just 0.0007 KL together. Measured on Qwen3.5-4B and SmolLM2-360M against 3-bit baselines that leave the embedding or cache at 16 bits, plans score between 6 and 81 times lower KL for the same cost on each device.
Hardware changes the best answer. On disk only stored bits matter. A laptop producing one sequence reads weights and cache on every token. A server handling 32 sequences at once shares each weight read among them, so the cache becomes the main expense. For SmolLM2-360M the compiler writes a separate plan for each case. The server version spends 7.4 bits on matrices, and the plan for compute-bound devices skips trellis codes because decoding them takes extra work.
What the numbers don’t cover
Any quantization compiler is only as good as its measurements, and these are the authors’ own, on a small set of models. The GSM8K sample is 250 problems. Anyone planning to use LookAtMyQuant 2 should test it on their own prompts, since KL and benchmark scores miss some kinds of degradation.
Source: lookatmyquant.pages.dev