Hugging Face's logo

TheBloke
/
Llama-2-70B-Chat-GPTQ

Text Generation
Transformers
Safetensors
PyTorch
English
llama
facebook
meta
llama-2
text-generation-inference
4-bit precision
gptq
Model card Files Community
52

Performance Drop due to quantization?

#34
by Teja-Gollapudi - opened
Discussion
Teja-Gollapudi

Hi,
Are there any benchmark comparisions for the Quantized model vs the full model?
I want to gauge the performance drop introduced by quantization.

Thank you!

Teja-Gollapudi changed discussion title from Benchmark comparison to Performance Drop due to quantization?
yravindranath
•
edited Sep 11, 2023

Did you manage to find any comparison?

Teja-Gollapudi

Never got around to doing it 😕.

YaTharThShaRma999

4 bits are roughly 95 percent as accurate as full precision model

ehussein

I found a couple of subreddits discussing that topic:

https://www.reddit.com/r/LocalLLaMA/comments/15zz81s/llama2_quantized_model_vs_regular_one_whats_the/

https://www.reddit.com/r/LocalLLaMA/comments/15rh3op/effects_of_quantization_of_arc_benchmark/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=1

Sign up or log in to comment