By “work out” you mean no accuracy degradation? That’s a big ask - currently we can barely quantize to dynamic fp4 with small block size - still not completely lossless on all benchmarks.
PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
Only a presence bitmap? If we're contemplating packing schemes I'm tempted to write a paper that uses arithmetic coding to squeeze out a few more centi-bits.
If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
Who knew that if you actually look at information entropy you can pack stuff better!