Convert AutoAWQ checkpoints to compressed-tensors

[mutichung]

Summary

This PR introduces a new script to convert AutoAWQ checkpoints into compressed-tensors-compatible format under modifiers/awq. Resolves #2087.

Usage

• Via CLI:

  python -m llmcompressor.modifiers.awq.convert_autoawq \
    --model-name-or-path /path/to/model \
    --output-dir /path/to/compressed/model \
    --quantization-format naive-quantized

• Via Python:

  from llmcompressor.modifiers.awq.convert_autoawq import load_and_convert_from_autoawq
  
  awq_model_path = "/path/to/model"  # can also be model_id on huggingface hub
  model = load_and_convert_from_autoawq(awq_model_path)

Known Issue

Asymmetric Support in llm-compressor & compressed-tensors

• AutoAWQ with version GEMM only supports asymmetric quantization ¹.
  • AssertionError will be raised despite setting zero_point=False.
• Support for zero-point decompression in PackedQuantizationCompressor is a WIP ².
• 2025/12/15 Update: zero-point decompression has been merged in ³ but reverted shortly after ⁴.

Test Plan

• Create tests to compare output logits between AutoAWQ-dequantized floating-point model and llmcompressor-compressed model with CompressedLinear.
  • The logits do not satisfy torch.testing.assert_close, potentially due to GEMM kernel's internal precision?
• Run and compare benchmark results between AutoAWQForCausalLM and vLLM.
  • Using compressed-tensors based on ³.
• Created tests to compare benchmark results between AutoAWQ and llmcompressor checkpoints.

ruikangliu/DeepSeek-R1-Distill-Qwen-1.5B-quantized.awq-autoawq-w4g128

Format	Inference Backend	ARC-Easy	ARC-Challenge
AutoAWQ	hf	0.6435	0.3584
naive-quantized	hf	0.6431	0.3584
packed-quantized	hf	0.6431	0.3584
packed-quantized	vllm	0.6427	0.3592

AMead10/Llama-3.2-3B-Instruct-AWQ

Format	Inference Backend	ARC-Easy	ARC-Challenge
AutoAWQ	hf	0.7976	0.5017
naive-quantized	hf	0.7971	0.5026
packed-quantized	hf	0.7971	0.5026
packed-quantized	vllm	0.7976	0.5043

fbaldassarri/mistralai_Mistral-7B-Instruct-v0.3-autoawq-int4-gs128-asym

Format	Inference Backend	ARC-Easy	ARC-Challenge
AutoAWQ	hf	0.8641	0.6280
naive-quantized	hf	0.8645	0.6280
packed-quantized	hf	0.8645	0.6280
packed-quantized	vllm	0.8649	0.6280

Future Work

• Support other AutoAWQ versions, e.g., GEMV.
• Set default quantization format to packed-quantized once asymmetric decompression is finalized.
• Replace AutoModelForCausalLM with a more generalized autoclass.

Footnotes

1. awq/modules/linear/gemm.py#L187 ↩

2. [Feature] Support Zero-point Decompression #1704 ↩

3. compressed-tensors@f9e7426 ↩ ↩²

4. compressed-tensors@cf5980d ↩

[gemini-code-assist]

Summary of Changes

Hello @mutichung, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a dedicated utility to bridge the gap between AutoAWQ-quantized models and the llmcompressor framework's compressed-tensors format. It provides a robust conversion pipeline, allowing users to take existing AutoAWQ checkpoints, dequantize them, and then re-compress them into a format that llmcompressor can natively understand and utilize, thereby expanding the interoperability of quantized models.

Highlights

• AutoAWQ to compressed-tensors conversion: A new script is added to convert AutoAWQ checkpoints into the compressed-tensors-compatible format, enabling broader interoperability.
• CLI and Python API: The conversion process can be initiated either through a command-line interface or programmatically using a dedicated Python function.
• GEMM version support: The script specifically supports the GEMM version of AutoAWQ quantization, handling its unique dequantization and re-packing requirements.
• Zero-point adjustment: The conversion correctly adjusts the zero-point representation to align AutoAWQ's [0, 2^bits - 1] range with compressed-tensors' [-2^(bits - 1), 2^(bits - 1) - 1] range for accurate quantization.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new script to convert AutoAWQ checkpoints into a compressed-tensors-compatible format. The implementation covers loading model weights, dequantizing them according to the AutoAWQ GEMM version, and then re-packing them using ModelCompressor. The script also includes CLI and Python interfaces for conversion. Overall, the changes are well-structured and address the stated objective. However, there are a few areas related to security, correctness, and consistency that could be improved.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[mutichung]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a valuable script for converting AutoAWQ checkpoints to the compressed-tensors format. The implementation is well-structured, with clear separation of concerns and good use of existing libraries. However, I've identified a couple of potential issues in the dequantization logic that could lead to incorrect behavior, particularly concerning tensor shapes and the handling of quantization parameters. My review includes suggestions to address these points to ensure the conversion is robust and correct for a wider range of models. The accompanying tests are a great start for validation.

[brian-dellabetta]

Thank you for the contribution! This looks great, but I have a couple questions about where this should be placed and if the test can be made a little shorter. I will add some other maintainers to review and discuss, and we can decide from there. Thanks again!

[brian-dellabetta]

running lm_eval can be expensive, when comparing models we just want to ensure the logits are the same for a given set of input_ids. So one way to make this cheaper would be

input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids
orig_logits = orig_model.forward(input_ids=input_ids).logits
new_logits = new_model.forward(input_ids=input_ids).logits

# possible things to compare
print(f"Norm Diff {(orig_logits-new_logits).norm()}")
print(f"Norm MSE {torch.nn.MSELoss()(orig_logits,new_logits).norm()}")
print(f"Norm {orig_logits.norm()}, {new_logits.norm()}")

[mutichung]

Got it. Will work on this and see what I can do.

[brian-dellabetta]

this may be more suitable for a higher scope, for example placing it into examples/ or tools/, or even directly into compressed-tensors as it does not involve any src code in llmcompressor.

This is the first time we've added a feature like this, just posting here to see what the rest of the team thinks, and we can decide after that.

[HDCharles]

agree, this is not a modifier and should probably be in tools or maybe utils?

[mutichung]

Here's my take from a user experience perspective. None of them are strong opinions btw 😆:

• Users with their AutoAWQ checkpoint would naturally visit the document site of llm-compressor and look for things related to AWQ. Putting the conversion script under the AWQ modifier module improves discoverability.
• Putting it into some other places (examples/, tools/) + mentioning it in the AWQ modifier's description should also work but adds an extra layer of maintenance effort if things are subjected to changes.
• compressed-tensors does not have any documentation at the moment. Users would have to be redirected from some warning messages, AWQ modifier documentation, or vLLM documentation to be aware of the existence of this tool.

Also, just curious, here's a question regarding the purpose of this tool:

• vLLM seems to support serving AutoAWQ checkpoints ¹. Why do we need to convert the format? Is vLLM planning on dropping the support and removing the AutoAWQ kernels?
• If so, then mentioning this conversion tool in the vLLM doc and the warning/error message makes the most sense.
• Otherwise, what are reasons and the potential entry points do you expect users to have the need to convert AutoAWQ checkpoints?

Footnotes

1. vLLM AutoAWQ page ↩

[HDCharles]

AutoAWQ is no longer maintained. It's be good to simplify formats and code while being able to support legacy usage through conversion

[HDCharles]

The document site for AWQ and the place where the core AWQ components are located aren't necessarily the same. I doubt people are going file by file in core components to find what they want. There's also no readme in there and the general modifiers readme one level up doesn't mention AWQ (bad). Searching AWQ usually leads users to examples/AWQ where there is a dedicated README. it feels like this is the expected landing spot. What path do you expect users would use to find modifiers/awq as their first landing point?

[brian-dellabetta]

Hi @mutichung , we discussed this internally, and would like to move this to compressed-tensors instead, into a new module and file like src/compressed_tensors/converters/autoawq.py. We landed on this because it only has to do with moving something out of the W4A16 serialization format introduced by AutoAWQ, and doesn't really have anything to do with the AWQ algorithm, and because it doesn't rely on any src code in llmcompressor/AWQModifier. We can still include it in the examples/awq README though, and add it to our docs and the vllm docs. Anyone with llm-compressor installed will still be able to use it from the dependency.

Please let us know what you think. Happy to help set up that PR in compressed-tensors.

[mutichung]

@brian-dellabetta Great, sounds reasonable to me! I'm happy to move this PR to compressed-tensors. Thank you all for the review and precious feedback.

[mutichung]

Closing in favor of compressed-tensors#531.

[brian-dellabetta]

Thanks @mutichung , we can follow up on the compressed-tensors PR. Appreciate it!