Migrate Importer to Advisory v2 & Collect Existing Fix Commits for Project KB by ziadhany · Pull Request #1987 · aboutcode-org/vulnerablecode

ziadhany · 2025-08-25T01:13:19Z

Issues:

keshav-space

Thanks @ziadhany, see some suggestions.

vulnerabilities/pipelines/v2_importers/project_kb_importer.py

vulnerabilities/pipelines/v2_improvers/collect_commits_project_kb.py

ziadhany · 2025-12-19T00:50:14Z

INFO 2026-01-06 23:37:14.559524 UTC Pipeline [ProjectKBStatementsPipeline] starting
INFO 2026-01-06 23:37:14.559665 UTC Step [clone] starting
INFO 2026-01-06 23:37:14.559726 UTC Cloning ProjectKB Statements advisory data...
Importing data using project-kb-statements_v2
INFO 2026-01-06 23:47:35.150849 UTC Step [clone] completed in 621 seconds (10.3 minutes)
INFO 2026-01-06 23:47:35.150972 UTC Step [collect_and_store_advisories] starting
INFO 2026-01-06 23:47:35.198054 UTC Estimated advisories to process: 1297
INFO 2026-01-06 23:47:35.198164 UTC Collecting 1,297 advisories
INFO 2026-01-06 23:47:35.198220 UTC Collecting fix commits from YAML statements under /statements....
INFO 2026-01-06 23:47:39.656827 UTC Progress: 10% (130/1297) ETA: 40 seconds
INFO 2026-01-06 23:47:44.858846 UTC Progress: 20% (260/1297) ETA: 39 seconds
INFO 2026-01-06 23:47:50.056013 UTC Progress: 30% (390/1297) ETA: 35 seconds
INFO 2026-01-06 23:47:56.038865 UTC Progress: 40% (519/1297) ETA: 31 seconds
INFO 2026-01-06 23:48:00.162088 UTC Progress: 50% (649/1297) ETA: 25 seconds
INFO 2026-01-06 23:48:05.250671 UTC Progress: 60% (779/1297) ETA: 20 seconds
INFO 2026-01-06 23:48:14.116650 UTC Progress: 70% (908/1297) ETA: 17 seconds
INFO 2026-01-06 23:48:19.894134 UTC Progress: 80% (1038/1297) ETA: 11 seconds
INFO 2026-01-06 23:48:27.031290 UTC Progress: 90% (1168/1297) ETA: 6 seconds
INFO 2026-01-06 23:48:37.578051 UTC Progress: 100% (1297/1297)
INFO 2026-01-06 23:48:37.595528 UTC Successfully collected 1,297 advisories
INFO 2026-01-06 23:48:37.595719 UTC Step [collect_and_store_advisories] completed in 62 seconds (1.0 minutes)
INFO 2026-01-06 23:48:37.595819 UTC Step [clean_downloads] starting
INFO 2026-01-06 23:48:37.595899 UTC Removing cloned repository...
INFO 2026-01-06 23:48:37.752148 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-01-06 23:48:37.752347 UTC Pipeline completed in 683 seconds (11.4 minutes)

Process finished with exit code 0

Importing data using project-kb-msr-2019_v2
INFO 2026-01-06 23:53:13.387213 UTC Pipeline [ProjectKBMSR2019Pipeline] starting
INFO 2026-01-06 23:53:13.387372 UTC Step [clone] starting
INFO 2026-01-06 23:53:13.387425 UTC Cloning ProjectKB advisory data...
INFO 2026-01-06 23:53:59.877546 UTC Step [clone] completed in 46 seconds
INFO 2026-01-06 23:53:59.877668 UTC Step [collect_and_store_advisories] starting
INFO 2026-01-06 23:53:59.878934 UTC Estimated advisories to process: 1281
INFO 2026-01-06 23:53:59.879016 UTC Collecting 1,281 advisories
INFO 2026-01-06 23:53:59.879068 UTC Collecting fix commits from ProjectKB ( vulas_db_msr2019_release )...
INFO 2026-01-06 23:54:00.909065 UTC Progress: 10% (129/1281) ETA: 9 seconds
INFO 2026-01-06 23:54:01.961816 UTC Progress: 20% (257/1281) ETA: 8 seconds
INFO 2026-01-06 23:54:02.941568 UTC Progress: 30% (385/1281) ETA: 7 seconds
INFO 2026-01-06 23:54:03.933323 UTC Progress: 40% (513/1281) ETA: 6 seconds
INFO 2026-01-06 23:54:04.936665 UTC Progress: 50% (641/1281) ETA: 5 seconds
INFO 2026-01-06 23:54:05.861854 UTC Progress: 60% (769/1281) ETA: 4 seconds
INFO 2026-01-06 23:54:06.701578 UTC Progress: 70% (897/1281) ETA: 3 seconds
INFO 2026-01-06 23:54:07.683254 UTC Progress: 80% (1025/1281) ETA: 2 seconds
INFO 2026-01-06 23:54:08.680071 UTC Progress: 90% (1153/1281) ETA: 1 seconds
INFO 2026-01-06 23:54:09.650591 UTC Progress: 100% (1281/1281)
INFO 2026-01-06 23:54:09.658872 UTC Successfully collected 1,281 advisories
INFO 2026-01-06 23:54:09.658991 UTC Step [collect_and_store_advisories] completed in 10 seconds
INFO 2026-01-06 23:54:09.659057 UTC Step [clean_downloads] starting
INFO 2026-01-06 23:54:09.659103 UTC Removing cloned repository...
INFO 2026-01-06 23:54:09.668470 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-01-06 23:54:09.668553 UTC Pipeline completed in 56 seconds

Process finished with exit code 0

TG1999

LGTM!

vulnerabilities/importers/project_kb_msr2019.py

keshav-space

Thanks @ziadhany, see comments below.

keshav-space · 2025-12-23T17:57:45Z

vulnerabilities/pipelines/v2_importers/project_kb_statements_importer.py

+
+            note_texts = []
+            for note_entry in yaml_data.get("notes", []):
+                text_content = note_entry.get("text")


Notes section may also contain list of links which should be collected and stored as references see example https://github.com/SAP/project-kb/blob/730d2192bfa9b909246e02bc051e4fad2958a6d9/statements/CVE-2018-16406/statement.yaml

vulnerabilities/pipelines/v2_importers/project_kb_importer.py

ziadhany · 2025-12-25T03:31:57Z

@keshav-space I updated the logs for this Project KB and fixed the logic for handling the affected packages. Please have a look when you have time.

keshav-space

Thanks @ziadhany, LGTM! Minor nits for your consideration.

vulnerabilities/pipelines/v2_importers/project_kb_msr2019_importer.py

vulnerabilities/pipelines/v2_importers/project_kb_statements_importer.py

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add a test for the ProjectKB importer and collect fix commits pipeline. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix CI falling test Resolve merge conflict and Update migration file Remove duplication and create append_patch_classifications function Update the project-kb test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Update Project-KB importer Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix step code style format Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany requested review from TG1999 and keshav-space and removed request for keshav-space October 15, 2025 14:50

keshav-space requested changes Oct 17, 2025

View reviewed changes

ziadhany mentioned this pull request Dec 17, 2025

VCIO-API-V2: Project KB MSR importer to advisory V2 #2036

Closed

ziadhany changed the title ~~Collect existing fix commits for project-kb~~ Migrate Importer to Advisory v2 & Collect Existing Fix Commits for Project KB Dec 17, 2025

ziadhany force-pushed the kb-commits branch 2 times, most recently from 1b9472e to 666e774 Compare December 19, 2025 00:26

TG1999 approved these changes Dec 19, 2025

View reviewed changes

ziadhany commented Dec 21, 2025

View reviewed changes

vulnerabilities/importers/project_kb_msr2019.py Outdated Show resolved Hide resolved

keshav-space requested changes Dec 23, 2025

View reviewed changes

ziadhany requested a review from keshav-space December 25, 2025 03:32

keshav-space approved these changes Jan 6, 2026

View reviewed changes

ziadhany force-pushed the kb-commits branch from 390789c to 98cbdfb Compare January 7, 2026 00:00

ziadhany added 7 commits January 7, 2026 02:11

Add initial support for collecting ProjectKBP old fix commits.

050995e

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add ProjectKBv2 importer

2387232

Add a test for the ProjectKB importer and collect fix commits pipeline. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Drop project_kb_msr2019 V1 importer

de2028a

Fix CI falling test Resolve merge conflict and Update migration file Remove duplication and create append_patch_classifications function Update the project-kb test Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Split the project-kb into two separate pipelines

fe1d9ea

Update Project-KB importer Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Refactor CSV processing to avoid duplicate iteration over items

0b9f837

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix an indentation bug and update the test case

38a72d9

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Rename the pipeline_id for ProjectKBMSR2019Pipeline

f70b2a4

Fix step code style format Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany force-pushed the kb-commits branch from 98cbdfb to f70b2a4 Compare January 7, 2026 00:11

ziadhany merged commit 05b26dd into aboutcode-org:main Jan 7, 2026
5 of 6 checks passed

ziadhany deleted the kb-commits branch January 7, 2026 00:31

Uh oh!

Conversation

ziadhany commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

keshav-space Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ziadhany commented Aug 25, 2025 •

edited

Loading

ziadhany commented Dec 19, 2025 •

edited

Loading

ziadhany commented Dec 25, 2025 •

edited

Loading