Support VLM calibration with image-text data #755

Edwardf0t1 · 2026-01-09T09:26:47Z

What does this PR do?

Type of change: New feature

Overview:

The primary goal of this PR is to allow the model optimizer to use image-text pair data during the calibration phase of quantization, which is likely help improve accuracy of quantized VLMs like Nemotron VL on visual understanding tasks particularly, compared to text-only calibration data.

New Feature: Adds support for VLM calibration specifically using image-text data.
Dataset Integration: Introduces support for sampling from the Nemotron-VLM-Dataset-v2.
Refactoring: Created a separate utility for VLM datasets to keep the main Hugging Face PTQ script (hf_ptq.py) clean.
Simplified logic for handling multimodal inputs.
Addressed specific issues encountered when calibrating the Nemotron-Nano-VL-12B-V2 model with image data.
Documentation: Updated the README to include instructions and examples for VLM calibration.

This PR complements #347 and we will consolidate llm_ptq and vlm_ptq examples in follow-up PRs.

Usage

python3 hf_ptq.py   --pyt_ckpt_path /home/scratch.omniml_data_2/models/Nemotron-Nano-VL-12B-V2   --qformat nvfp4   --export_path /home/omniml_data_3/zhiyuc/checkpoints/Nemotron-Nano-VL-12B-V2-NVFP4-doccalib   --trust_remote_code   --kv_cache_qformat none --calib_with_images   --vlm_dataset nemotron_vlm_dataset_v2   --vlm_subsets sparsetables,plotqa_cot   --calib_size 512

Testing

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed.
Is this change backward compatible?: Yes
Did you write any new necessary tests?: Yes/No
Did you add or update any necessary documentation?: Yes
Did you update Changelog?: Not yet

Additional Information

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

copy-pr-bot · 2026-01-09T09:26:51Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

codecov · 2026-01-09T09:37:57Z

Codecov Report

❌ Patch coverage is 9.84615% with 293 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.17%. Comparing base (307fe71) to head (2611b0e).
⚠️ Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
modelopt/torch/utils/vlm_dataset_utils.py	8.37%	175 Missing ⚠️
modelopt/torch/utils/nemotron_vlm_dataset_utils.py	11.94%	118 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #755      +/-   ##
==========================================
- Coverage   74.66%   73.17%   -1.50%     
==========================================
  Files         192      193       +1     
  Lines       18975    19352     +377     
==========================================
- Hits        14167    14160       -7     
- Misses       4808     5192     +384

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

…for Nemotron-VLM-Dataset-v2 Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

shengliangxu · 2026-01-14T01:34:44Z

So, we only support image quantization for just nemotron-vl? If yes, why?

examples/llm_ptq/hf_ptq.py

cjluo-nv · 2026-01-14T17:00:12Z

modelopt/torch/utils/vlm_dataset_utils.py

 # limitations under the License.

-"""Utility functions for getting samples and forward loop function for different vlm datasets."""
+"""Utility functions for getting samples and dataloader for different VLM calibration datasets.


@ajrasane could you review this change?

examples/llm_ptq/README.md

cjluo-nv · 2026-01-14T17:02:44Z

@Edwardf0t1 do you have experiments evaluating the accuracy impact of using the new dataset?

examples/llm_ptq/README.md

examples/llm_ptq/hf_ptq.py

Edwardf0t1 · 2026-01-16T06:34:38Z

So, we only support image quantization for just nemotron-vl? If yes, why?

At this time, only Nemotron VL has been tested. We can extend the logic to support other VLMs later. Note that different VLMs may have different forward functions—e.g., the way the vision encoder interacts with the language decoder can vary across models.

Do you have a preferred VL model you’d like us to support next? For instance, Qwen3-VL?

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

Edwardf0t1 · 2026-01-20T17:55:11Z

@Edwardf0t1 do you have experiments evaluating the accuracy impact of using the new dataset?

Tested on two benchmarks DocVQA and InfoVQA for Nemotron Nano VL v2 with vLLM backend:

BF16 Baseline: 94.2184, 79.1404
NVFP4 text-only calibration: 93.9472, 77.7221
NVFP4 image-text calibration: 94.0854, 77.9598

Image-text calibration is only marginally better in these cases, but the calibration flow in this PR should be ready. The follow-up experiments can be

Choose different subsets in Nemotron-VLM-Dataset-v2 or another image-text dataset for calibration
Check more evaluation metrics.
Run benchmarks on other VLMs such as Nemotron Parse, Qwen3-VL.

cjluo-nv · 2026-01-20T18:32:28Z

examples/llm_ptq/README.md


 [PTQ for DeepSeek](../deepseek/README.md) shows how to quantize the DeepSeek model with FP4 and export to TensorRT-LLM.

+#### VLM calibration with image-text pairs (e.g., Nemotron VL)


Do you feel this can fall into: https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/vlm_ptq?

cjluo-nv · 2026-01-20T18:33:19Z

examples/llm_ptq/README.md

+  --qformat nvfp4 \
+  --export_path <quantized_ckpt_path> \
+  --trust_remote_code \
+  --calib_with_images \


qq: Can user choose which vlm dataset to use or we just provide one option

cjluo-nv · 2026-01-20T18:33:40Z

examples/llm_ptq/hf_ptq.py

    calib_dataloader = None
    first_text_speech_dataset = None
-    if model_type == "mllama":
+    if getattr(args, "calib_with_images", False):


why use getattr here?

cjluo-nv · 2026-01-20T18:33:54Z

examples/llm_ptq/hf_ptq.py

 ):
    """Auto search quantization of multiple formats."""

+    if getattr(args, "calib_with_images", False):


same here. And why not just use assert?

cjluo-nv · 2026-01-20T18:34:42Z

examples/llm_ptq/hf_ptq.py

        )
+    elif is_nemotron_vl_model and getattr(args, "calib_with_images", False):
+        # For Nemotron VL image calibration, we need an AutoProcessor to build multimodal inputs.
+        try:


do we need this try except?

cjluo-nv · 2026-01-20T18:37:06Z

examples/llm_ptq/hf_ptq.py

+        tokenizer.padding_side = "left"
+
+        # Quantize only the language model, but keep the full_model for calibration forward.
+        language_model_lineage = get_language_model_from_vl(full_model)


please avoid duplicated codes with below

cjluo-nv · 2026-01-20T18:37:55Z

examples/llm_ptq/hf_ptq.py

+            # Those kwargs must be consumed by the *full* VLM model, not the extracted language_model.
+            if getattr(args, "calib_with_images", False) and is_nemotron_vl_model:
+
+                def calibrate_full_model(_model):


can we make these helper functions and move it output hf_ptq

cjluo-nv · 2026-01-20T18:40:11Z

modelopt/torch/utils/vlm_dataset_utils.py

+    #   prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    #   inputs = processor(text=[prompt], images=[pil_image], ...)
+
+    def _collate_fn(examples: list[dict[str, Any]]) -> dict[str, torch.Tensor] | dict[str, Any]:


why do we need to introduce these while the original one does not?

jingyu-ml

LGTM. I only reviewed the dataset processing part, which behaves as expected, loading the dataset on demand rather than downloading the entire dataset.

Edwardf0t1 added 2 commits January 9, 2026 01:21

Add support for VLM calibration with image-text pair data

b9acc43

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

Add support for VLM calibration with image-text pair data

528b51d

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

Edwardf0t1 added 13 commits January 9, 2026 18:09

add support for sampling from Nemotron-VLM-Dataset-v2

3ef4b9d

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

update readme

2d60f98

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

fix issues when calibrate with image data for Nemotron Nano VL

42a8406

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

fix issues when calibrate with image data for Nemotron Nano VL

7489a36

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

fix issues when calibrate with image data for Nemotron Nano VL

bd87154

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

fix issues when calibrate with image data for Nemotron Nano VL

3200a63

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

simplify

8964aa5

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

refactor to make hf_ptq cleaner, create a separate vlm dataset utils …

3b7373d

…for Nemotron-VLM-Dataset-v2 Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

refactor to make hf_ptq cleaner, create a separate vlm dataset utils …

5c774f9

…for Nemotron-VLM-Dataset-v2 Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

update readme

f2774fc

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

update readme

59d97a6

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

update readme

e2e59f6

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

minor refactor

2a3868a

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

Edwardf0t1 self-assigned this Jan 14, 2026

Edwardf0t1 marked this pull request as ready for review January 14, 2026 01:16

Edwardf0t1 requested review from a team as code owners January 14, 2026 01:16

Edwardf0t1 requested review from ChenhanYu, ajrasane, cjluo-nv, jingyu-ml, kaix-nv, meenchen, mxinO, realAsma and shengliangxu January 14, 2026 01:16

cjluo-nv reviewed Jan 14, 2026

View reviewed changes

examples/llm_ptq/hf_ptq.py Show resolved Hide resolved

cjluo-nv reviewed Jan 14, 2026

View reviewed changes

examples/llm_ptq/README.md Outdated Show resolved Hide resolved

cjluo-nv reviewed Jan 14, 2026

View reviewed changes

examples/llm_ptq/README.md Show resolved Hide resolved

meenchen reviewed Jan 14, 2026

View reviewed changes

examples/llm_ptq/README.md Show resolved Hide resolved

examples/llm_ptq/hf_ptq.py Show resolved Hide resolved

address reviews

2611b0e

Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>

cjluo-nv reviewed Jan 20, 2026

View reviewed changes

jingyu-ml approved these changes Jan 20, 2026

View reviewed changes


		[PTQ for DeepSeek](../deepseek/README.md) shows how to quantize the DeepSeek model with FP4 and export to TensorRT-LLM.

		#### VLM calibration with image-text pairs (e.g., Nemotron VL)

Support VLM calibration with image-text data #755

Are you sure you want to change the base?

Support VLM calibration with image-text data #755

Conversation

Edwardf0t1 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Usage

Testing

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot bot commented Jan 9, 2026

Uh oh!

codecov bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shengliangxu commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cjluo-nv commented Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Edwardf0t1 commented Jan 16, 2026

Uh oh!

Edwardf0t1 commented Jan 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjluo-nv Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jingyu-ml left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Edwardf0t1 commented Jan 9, 2026 •

edited

Loading

codecov bot commented Jan 9, 2026 •

edited

Loading

shengliangxu commented Jan 14, 2026 •

edited

Loading

cjluo-nv Jan 20, 2026 •

edited

Loading