Merge branch 'aboutcode-org:main' into Page-Updated

Author: Rishi-source

CHANGELOG.rst

@@ -1,8 +1,89 @@
 Release notes
 =============
 
-Version (next)
------------------------
+
+Version v35.0.0
+---------------------
+
+- Add scores in bulk search V1 API #1675
+- Add improver pipeline to flag ghost packages #644 #917 #1395 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1533
+- Add base pipeline for importers and migrate PyPa importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1559
+- Remove dupe Package.get_non_vulnerable_versions by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1570
+- Import data from GSD #706 by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/787
+- Add curl advisories importer by @ambuj-1211 in https://github.com/aboutcode-org/vulnerablecode/pull/1439
+- Update dependencies by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1590
+- Bump django from 4.2.0 to 4.2.15 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1591
+- Bump cryptography from 42.0.4 to 43.0.1 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1582
+- Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1581
+- Improve export command by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1571
+- Fix typo in Kev requests import by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1594
+- Prepare for release v34.0.1 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1595
+- Bump upload-artifact to v4 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1596
+- Migrate Npm importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1574
+- Use correct regex for CVE by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1599
+- Migrate Nginx importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1575
+- Migrate GitLab importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1580
+- Migrate GitHub importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1584
+- Migrate NVD importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1587
+- Match affected and fixed-by Packages by @johnmhoran in https://github.com/aboutcode-org/vulnerablecode/pull/1528
+- Add management command to commit exported data by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1600
+- Add support to Exploits model by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1562
+- Fix 500 Server Error with DRF browsable API and resolve blank Swagger API documentation by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1603
+- Release v34.0.2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1604
+- Bump VCIO version by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1605
+- Bump django from 4.2.15 to 4.2.16 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1608
+- Bump fetchcode from v0.3.0 to v0.6.0 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1607
+- Use 4-tier system for storing package metadata by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1609
+- Fix vers range crash by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1598
+- Add GitHub action to publish aboutcode.hashid PyPI by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1615
+- Segregate PackageRelatedVulnerability model to new models by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1612
+- Add documentation for new pipeline design by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1621
+- Fix 500 error in /api/cpes endpoint by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1629
+- Migrate pysec importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1628
+- Avoid memory exhaustion during data migration by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1630
+- Add support for Calculating Risk in VulnerableCode by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1593
+- Bulk create in migrations by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1640
+- Update README.rst by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1641
+- Prepare for release v34.1.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1642
+- Add V2 API endpoints by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1631
+- Prepare for release v34.2.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1647
+- Refactor severity score model and fix incorrect suse scores by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1636
+- Add bulk search in v2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1649
+- Prepare release v34.3.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1652
+- Add `on_failure` to handle cleanup during pipeline failure by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1651
+- Fix API bug by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1654
+- Add reference score to package endpoint  by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1655
+- Prepare for release v34.3.2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1656
+- Add support for storing  exploitability and weighted severity by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1646
+- Avoid migrations on version bumps by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1660
+- Prepare v35.0.0rc1 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1664
+
+
+
+Version v35.0.0rc1
+---------------------
+
+- Add support for storing exploitability and weighted severity #1646
+- Avoid migrations on version bumps #1660
+
+
+Version v34.3.2
+----------------
+
+- HOTFIX: Add reference score to package endpoint #1655
+
+
+Version v34.3.1
+----------------
+
+- HOTFIX: Fix API bug #1654
+
+
+Version v34.3.0
+-----------------
+
+- Add bulk search in v2 #1649 
+- Refactor severity score model and fix incorrect suse scores #1636
 
 
 Version v34.2.0

docs/source/contributing.rst

@@ -89,7 +89,7 @@ Helpful Resources
 
 - Review our `comprehensive guide <https://scancode-toolkit.readthedocs.io/en/latest/contribute/index.html>`_
   for more details on how to add quality contributions to our codebase and documentation
-- Check this free resource on `how to contribute to an open source project on github <https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github>`_
+- Check this free resource on `How to contribute to an open source project on github <https://egghead.io/lessons/javascript-identifying-how-to-contribute-to-an-open-source-project-on-github>`_
 - Follow `this wiki page <https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_
   on how to write good commit messages
 - `Pro Git book <https://git-scm.com/book/en/v2>`_

docs/source/tutorial_add_importer_pipeline.rst

@@ -298,10 +298,14 @@ version management from `univers <https://github.com/aboutcode-org/univers>`_.
     **advisories_count** should never be directly added in steps.
 
 
+.. attention::
+
+   Implement ``on_failure`` to handle cleanup in case of pipeline failure.
+   Cleanup of downloaded archives or cloned repos is necessary to avoid potential resource leakage.
 
 .. note::
 
-   | Use ``make valid`` to format your code using black and isort automatically.
+   | Use ``make valid`` to format your new code using black and isort automatically.
    | Use ``make check`` to check for formatting errors.
 
 Register the Importer Pipeline

docs/source/tutorial_add_improver_pipeline.rst

@@ -187,6 +187,11 @@ methods.
             self.log(f"Successfully flagged {ghost_package_count:,d} ghost Packages")
 
 
+.. attention::
+
+   Implement ``on_failure`` to handle cleanup in case of pipeline failure.
+   Cleanup of downloaded archives or cloned repos is necessary to avoid potential resource leakage.
+
 .. note::
 
    | Use ``make valid`` to format your new code using black and isort automatically.

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = vulnerablecode
-version = 34.2.0
+version = 35.0.0
 license = Apache-2.0 AND CC-BY-SA-4.0
 
 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390

vulnerabilities/api.py

@@ -22,9 +22,7 @@
 from rest_framework import viewsets
 from rest_framework.decorators import action
 from rest_framework.response import Response
-from rest_framework.reverse import reverse
 from rest_framework.throttling import AnonRateThrottle
-from rest_framework.throttling import UserRateThrottle
 
 from vulnerabilities.models import Alias
 from vulnerabilities.models import Exploit
@@ -54,13 +52,25 @@ def to_representation(self, instance):
 
 
 class VulnerabilityReferenceSerializer(serializers.ModelSerializer):
-    scores = VulnerabilitySeveritySerializer(many=True, source="vulnerabilityseverity_set")
+    scores = serializers.SerializerMethodField()
     reference_url = serializers.CharField(source="url")
 
     class Meta:
         model = VulnerabilityReference
         fields = ["reference_url", "reference_id", "reference_type", "scores", "url"]
 
+    def get_scores(self, instance):
+        severities_related_to_reference = [
+            severity
+            for severity in self.context.get("severities", [])
+            if severity.url == instance.url
+        ]
+
+        return VulnerabilitySeveritySerializer(
+            severities_related_to_reference,
+            many=True,
+        ).data
+
 
 class BaseResourceSerializer(serializers.HyperlinkedModelSerializer):
     """
@@ -143,17 +153,39 @@ class VulnSerializerRefsAndSummary(BaseResourceSerializer):
         many=True, source="filtered_fixed_packages", read_only=True
     )
 
-    references = VulnerabilityReferenceSerializer(many=True, source="vulnerabilityreference_set")
+    references = serializers.SerializerMethodField()
 
     aliases = serializers.SerializerMethodField()
 
     def get_aliases(self, obj):
         # Assuming `obj.aliases` is a queryset of `Alias` objects
         return [alias.alias for alias in obj.aliases.all()]
 
+    def get_references(self, vulnerability):
+        references = vulnerability.vulnerabilityreference_set.all()
+        severities = vulnerability.severities.all()
+
+        serialized_references = VulnerabilityReferenceSerializer(
+            references,
+            context={"severities": severities},
+            many=True,
+        ).data
+
+        return serialized_references
+
     class Meta:
         model = Vulnerability
-        fields = ["url", "vulnerability_id", "summary", "references", "fixed_packages", "aliases"]
+        fields = [
+            "url",
+            "vulnerability_id",
+            "summary",
+            "references",
+            "fixed_packages",
+            "aliases",
+            "risk_score",
+            "exploitability",
+            "weighted_severity",
+        ]
 
 
 class WeaknessSerializer(serializers.HyperlinkedModelSerializer):
@@ -199,8 +231,7 @@ class VulnerabilitySerializer(BaseResourceSerializer):
         many=True, source="filtered_fixed_packages", read_only=True
     )
     affected_packages = MinimalPackageSerializer(many=True, read_only=True)
-
-    references = VulnerabilityReferenceSerializer(many=True, source="vulnerabilityreference_set")
+    references = serializers.SerializerMethodField()
     aliases = AliasSerializer(many=True, source="alias")
     exploits = ExploitSerializer(many=True, read_only=True)
     weaknesses = WeaknessSerializer(many=True)
@@ -214,10 +245,22 @@ def to_representation(self, instance):
 
         return data
 
+    def get_references(self, vulnerability):
+        references = vulnerability.vulnerabilityreference_set.all()
+        severities = vulnerability.severities.all()
+
+        serialized_references = VulnerabilityReferenceSerializer(
+            references,
+            context={"severities": severities},
+            many=True,
+        ).data
+
+        return serialized_references
+
     def get_severity_range_score(self, instance):
         severity_vectors = []
         severity_values = set()
-        for s in instance.severities:
+        for s in instance.severities.all():
             if s.scoring_system == EPSS.identifier:
                 continue
 
@@ -251,6 +294,9 @@ class Meta:
             "weaknesses",
             "exploits",
             "severity_range_score",
+            "exploitability",
+            "weighted_severity",
+            "risk_score",
         ]
 
 
@@ -300,7 +346,7 @@ def get_vulnerabilities_for_a_package(self, package, fix) -> dict:
         otherwise return vulnerabilities fixed by the `package`.
         """
         fixed_packages = self.get_fixed_packages(package=package)
-        if fix:
+        if not fix:
             qs = package.affected_by_vulnerabilities.all()
         else:
             qs = package.fixing_vulnerabilities.all()
@@ -321,6 +367,10 @@ def get_fixing_vulnerabilities(self, package) -> dict:
         """
         Return a mapping of vulnerabilities fixed in the given `package`.
         """
+        # Ghost package should not fix any vulnerability.
+        if package.is_ghost:
+            return []
+
         return self.get_vulnerabilities_for_a_package(package=package, fix=True)
 
     def get_affected_vulnerabilities(self, package) -> dict:
@@ -595,7 +645,10 @@ def get_fixed_packages_qs(self):
         """
         return (
             self.get_packages_qs()
-            .filter(fixingpackagerelatedvulnerability__isnull=False)
+            .filter(
+                fixingpackagerelatedvulnerability__isnull=False,
+                is_ghost=False,
+            )
             .with_is_vulnerable()
         )
 
@@ -621,6 +674,9 @@ def get_queryset(self):
             .get_queryset()
             .prefetch_related(
                 "weaknesses",
+                "references",
+                "exploits",
+                "severities",
                 Prefetch(
                     "fixed_by_packages",
                     queryset=self.get_fixed_packages_qs(),

vulnerabilities/api_extension.py

@@ -84,11 +84,10 @@ class Meta:
 
 class V2VulnerabilitySeveritySerializer(ModelSerializer):
     score = CharField(source="value")
-    reference = V2VulnerabilityReferenceSerializer()
 
     class Meta:
         model = VulnerabilitySeverity
-        fields = ("score", "scoring_system", "scoring_elements", "published_at", "reference")
+        fields = ("url", "score", "scoring_system", "scoring_elements", "published_at")
 
 
 class V2WeaknessSerializer(ModelSerializer):
@@ -127,9 +126,9 @@ class V2VulnerabilitySerializer(ModelSerializer):
 
     aliases = SerializerMethodField("get_aliases")
     weaknesses = V2WeaknessSerializer(many=True, source="weaknesses_set")
-    scores = V2VulnerabilitySeveritySerializer(many=True, source="vulnerabilityseverity_set")
     references = V2VulnerabilityReferenceSerializer(many=True, source="vulnerabilityreference_set")
     exploits = V2ExploitSerializer(many=True, source="weaknesses")
+    severities = V2VulnerabilitySeveritySerializer(many=True)
 
     def get_aliases(self, vulnerability):
         return vulnerability.aliases.only("alias").values_list("alias", flat=True)
@@ -145,11 +144,11 @@ class Meta:
             "vulnerability_id",
             "aliases",
             "status",
-            "scores",
             "weaknesses",
             "summary",
             "exploits",
             "references",
+            "severities",
         )
 
 
@@ -358,7 +357,7 @@ def get_queryset(self):
             .get_queryset()
             .prefetch_related(
                 "weaknesses",
-                # "severities",
+                "severities",
                 # "exploits",
             )
         )