TLDR Tech / AI
AutoRound (GitHub Repo)
AutoRound is an advanced quantization toolkit for large language and vision-language models, offering high accuracy at extremely low bit widths with minimal tuning. It is compatible with Transformers, vLLM, SGLang, and can quantize 7B models in ten minutes using a single GPU.
Read the full story