Model quantisation: how to shrink AI so it fits on a phone

Quantisation is a way to reduce the size of a neural network by storing its parameters as less precise numbers (for example, 4 bits instead of 16). The model gets smaller and faster while losing only a little quality.

Quantisation shrinks a model so it fits on a phone

In plain words

Imagine a 50-megapixel photo you need to send in a messenger. You compress it - the file is dozens of times smaller, yet on a phone screen you can barely see a difference. Quantisation does the same with a model's parameters: it rounds the numbers, the model slims down several times, and it answers almost the same.

How it works

Model parameters are usually stored as 16- or 32-bit numbers.
Quantisation rounds them down to 8, 4 or even fewer bits.
The model takes several times less memory and runs faster on an NPU or graphics card.
Clever methods round smartly so the most important numbers keep their precision.
The harder the compression, the more noticeably answer quality can drop.

Less precise numbers mean a smaller model

Where you'll see it

Quantised models are the backbone of on-device AI: smartphones and laptops run compressed versions. Enthusiasts who run models on their own computers see labels like Q4, Q8, INT4 and GGUF - these refer to the quantisation level and file format.

Does it matter when buying

For a regular user, it is invisible work done by the manufacturer. But if you run local models, it matters: a 4-bit version gives a sensible balance of quality and size. It also explains why INT4 and INT8 support in a neural processor is a useful spec.

Myths

Myth: A quantised model is a completely different, worse model. Reality: It is the same model, just with less precise numbers. For most tasks, the difference is small.
Myth: You can compress down to 1 bit with no loss. Reality: The harder the compression, the more quality is lost - there are no miracles.

FAQ

What does Q4 in a model file name mean?

That the parameters are stored in roughly four bits - one of the most popular options.

Can I quantise a model myself?

Yes, there are ready-made tools for open models.

Why do phones like INT8 and INT4?

Because the NPU processes such short numbers faster and with less energy.