Merge branch 'rc/1.26' into 126-mergeback

Author: james

.github/workflows/generate-query-help-docs.yml

@@ -0,0 +1,57 @@
+name: Generate CodeQL query help documentation using Sphinx
+
+on:
+  push:
+    branches:
+     - main
+     - 'rc/**'
+     - 'lgtm.com'
+  pull_request:
+    paths:
+      - '.github/workflows/generate-query-help-docs.yml'
+      - 'docs/codeql/query-help/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Clone github/codeql
+      uses: actions/checkout@v2
+      with:
+        path: codeql 
+    - name: Clone github/codeql-go
+      uses: actions/checkout@v2
+      with:
+        repository: 'github/codeql-go'
+        path: codeql-go
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+    - name: Download CodeQL CLI
+      uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
+      with:
+        repo: "github/codeql-cli-binaries"
+        version: "latest"
+        file: "codeql-linux64.zip"
+        token: ${{ secrets.GITHUB_TOKEN }}
+    - name: Unzip CodeQL CLI
+      run: unzip -d codeql-cli codeql-linux64.zip
+    - name: Set up query help docs folder  
+      run: |
+        cp -r codeql/docs/codeql/** .
+    - name: Query help to markdown
+      run: |
+        PATH="$PATH:codeql-cli/codeql" python codeql/docs/codeql/query-help-markdown.py
+    - name: Run Sphinx for query help
+      uses: ammaraskar/sphinx-action@8b4f60114d7fd1faeba1a712269168508d4750d2 # v0.4
+      with:
+        docs-folder: "query-help/"
+        pre-build-command: "python -m pip install --upgrade recommonmark"
+        build-command: "sphinx-build -b dirhtml . _build" 
+    - name: Upload HTML artifacts
+      uses: actions/upload-artifact@v2
+      with:
+        name: query-help-html
+        path: query-help/_build
+    

change-notes/1.26/analysis-python.md

@@ -4,19 +4,34 @@ The following changes in version 1.26 affect Python analysis in all applications
 
 ## General improvements
 
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
 ## Changes to existing queries
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
-
-
+|`py/unsafe-deserialization` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/path-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/command-line-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/reflective-xss` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/sql-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/code-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
 ## Changes to libraries
-
+* Some of the security queries now use the shared data flow library for data flow and taint tracking. This has resulted in an overall more robust and accurate analysis. The libraries mentioned below have been modelled in this new framework. Other libraries (e.g. the web framework `CherryPy`) have not been modelled yet, and this may lead to a temporary loss of results for these frameworks.
+* Improved modelling of the following serialization libraries:
+  - `PyYAML`
+  - `dill`
+  - `pickle`
+  - `marshal`
+* Improved modelling of the following web frameworks:
+  - `Django` (Note that modelling of class-based response handlers is currently incomplete.)
+  - `Flask`
+* Support for Werkzeug `MultiDict`.
+* Support for the [Python Database API Specification v2.0 (PEP-249)](https://www.python.org/dev/peps/pep-0249/), including the following libraries:
+  - `MySQLdb`
+  - `mysql-connector-python`
+  - `django.db`
+* Improved modelling of the following command execution libraries:
+  - `Fabric`
+  - `Invoke`
+* Improved modelling of security-related standard library modules, such as `os`, `popen2`, `platform`, and `base64`.
+* The original versions of the updated queries have been preserved [here](https://github.com/github/codeql/tree/main/python/ql/src/experimental/Security-old-dataflow).
 * Added taint tracking support for string formatting through f-strings.

cpp/ql/src/semmle/code/cpp/dataflow/internal/FlowVar.qll

@@ -620,7 +620,8 @@ module FlowVar_internal {
   private predicate largeVariable(Variable v, int liveBlocks, int defs) {
     liveBlocks = strictcount(SubBasicBlock sbb | variableLiveInSBB(sbb, v)) and
     defs = strictcount(SubBasicBlock sbb | exists(TBlockVar(sbb, v))) and
-    liveBlocks * defs > 1000000
+    // Convert to float to avoid int overflow (32-bit two's complement)
+    liveBlocks.(float) * defs.(float) > 100000.0
   }
 
   /**

docs/codeql/_templates/layout.html

@@ -38,27 +38,19 @@
 {%- block content %}
 <header class="Header">
     <div class="Header-item--full">
-        <a href="{{ pathto(master_doc) }}" class="Header-link f2 d-flex flex-items-center">
+        <a href="https://codeql.github.com/docs" class="Header-link f2 d-flex flex-items-center">
             <!-- <%= octicon "mark-github", class: "mr-2", height: 32 %> -->
             <svg height="32" class="octicon octicon-mark-github mr-2" viewBox="0 0 16 16" version="1.1" width="32"
                 aria-hidden="true">
                 <path fill-rule="evenodd"
                     d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z">
                 </path>
             </svg>
-            <span class="hide-sm">{{ project }}</span>
+            <span class="hide-sm">CodeQL documentation</span>
         </a>
     </div>
     <div class="Header-item hide-sm hide-md">
-        <form class="search" action="{{ pathto('search') }}" method="get">
-            <input class="form-control input-dark" type="text" name="q" placeholder="Search" />
-            <input class="btn" type="submit" value="Search" />
-            <input type="hidden" name="check_keywords" value="yes" />
-            <input type="hidden" name="area" value="default" />
-        </form>
-        <script type="text/javascript">$('#searchbox').show(0);</script>
-
-        <div class="clearer"></div>
+        <script src="https://addsearch.com/js/?key=93b4d287e2fc079a4089412b669785d5&categories=!0xhelp.semmle.com,0xcodeql.github.com,1xdocs,1xcodeql-standard-libraries,1xcodeql-query-help"></script>
     </div>
     <div class="Header-item">
 
@@ -69,20 +61,30 @@
             </summary>
 
             <ul class="dropdown-menu dropdown-menu-se dropdown-menu-dark">
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/codeql-overview">CodeQL overview</a></li>
+                <li class="dropdown-divider" role="separator"></li>
                 <div class="dropdown-header">
-                    Help docs
+                    CodeQL tools
                 </div>
-                <li><a class="dropdown-item" href="https://help.semmle.com/QL/learn-ql/">Learn CodeQL</a></li>
-                <li><a class="dropdown-item" href="https://help.semmle.com/codeql/codeql-tools.html">CodeQL tools</a>
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/codeql-for-visual-studio-code">CodeQL for VS Code</a>
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/codeql-cli">CodeQL CLI</a>
                 </li>
                 <li class="dropdown-divider" role="separator"></li>
+                <div class="dropdown-header">
+                    CodeQL guides
+                </div>
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/writing-codeql-queries">Writing CodeQL queries</a></li>
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/codeql-language-guides">CodeQL language guides</a>
+                <li class="dropdown-divider" role="separator"></li>
                 <div class="dropdown-header">
                     Reference docs
                 </div>
-                <li><a class="dropdown-item" href="https://help.semmle.com/QL/ql-handbook/">QL language reference</a>
-                <li><a class="dropdown-item" href="https://help.semmle.com/QL/ql-libraries.html">CodeQL libraries</a>
-                <li><a class="dropdown-item" href="https://help.semmle.com/QL/ql-built-in-queries.html">CodeQL
-                        queries</a>
+                <li><a class="dropdown-item" href="https://codeql.github.com/docs/ql-language-reference/">QL language
+                        reference</a>
+                <li><a class="dropdown-item" href="https://codeql.github.com/codeql-standard-libraries">CodeQL
+                        standard-libraries</a>
+                <li><a class="dropdown-item" href="https://codeql.github.com/codeql-query-help">CodeQL
+                        query help</a>
                 <li class="dropdown-divider" role="separator"></li>
                 <div class="dropdown-header">
                     Source files

…/codeql-cli-reference/about-ql-packs.rst docs/codeql/codeql-cli/about-ql-packs.rstdocs/codeql/codeql-cli/codeql-cli-reference/about-ql-packs.rst renamed to docs/codeql/codeql-cli/about-ql-packs.rst docs/codeql/codeql-cli/codeql-cli-reference/about-ql-packs.rst renamed to docs/codeql/codeql-cli/about-ql-packs.rst

@@ -85,11 +85,11 @@ The following properties are supported in ``qlpack.yml`` files.
    * - ``suites``
      - ``suites``
      - Optional
-     - The path to a directory that contains the "well-known" query suites in the pack, defined relative to the pack directory. You can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. To use query suites stored in other directories in the pack, you must provide their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites <../using-the-codeql-cli/creating-codeql-query-suites>`."
+     - The path to a directory that contains the "well-known" query suites in the pack, defined relative to the pack directory. You can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. To use query suites stored in other directories in the pack, you must provide their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites <creating-codeql-query-suites>`."
    * - ``extractor``
      - ``javascript``
      - All test packs
-     - The CodeQL language extractor to use when the CLI creates a database from test files in the pack. For more information about testing queries, see ":doc:`Testing custom queries <../using-the-codeql-cli/testing-custom-queries>`."
+     - The CodeQL language extractor to use when the CLI creates a database from test files in the pack. For more information about testing queries, see ":doc:`Testing custom queries <testing-custom-queries>`."
    * - ``tests``
      - ``.``
      - Optional for test packs
@@ -124,7 +124,7 @@ and ``libraryPathDependencies`` properties. If the pack contains query suites, y
 use the ``suites`` property to define their location. Query suites defined 
 here are called "well-known" suites, and can be used on the command line by referring to 
 their name only, rather than their full path.
-For more information about query suites, see ":doc:`Creating CodeQL query suites <../using-the-codeql-cli/creating-codeql-query-suites>`."
+For more information about query suites, see ":doc:`Creating CodeQL query suites <creating-codeql-query-suites>`."
 
 For example, a ``qlpack.yml`` file for a QL pack featuring custom C++ queries
 and libraries may contain:
@@ -154,10 +154,10 @@ For custom QL packs containing test files, you also need to include an
 ``extractor`` property so that the ``test run`` command knows how to create test
 databases. You may also wish to specify the ``tests`` property.
 
-.. include:: ../../reusables/test-qlpack.rst
+.. include:: ../reusables/test-qlpack.rst
 
 For more information about running tests, see ":doc:`Testing custom queries
-<../using-the-codeql-cli/testing-custom-queries>`."
+<testing-custom-queries>`."
 
 .. _standard-ql-packs:
 

…-the-codeql-cli/about-the-codeql-cli.rst …deql/codeql-cli/about-the-codeql-cli.rstdocs/codeql/codeql-cli/using-the-codeql-cli/about-the-codeql-cli.rst renamed to docs/codeql/codeql-cli/about-the-codeql-cli.rst docs/codeql/codeql-cli/using-the-codeql-cli/about-the-codeql-cli.rst renamed to docs/codeql/codeql-cli/about-the-codeql-cli.rst

@@ -26,4 +26,4 @@ command line. To run a command, use::
    codeql [command] [subcommand]
 
 To view the reference documentation for a command, add the ``--help`` flag, or visit the 
-"`CodeQL CLI manual <../../codeql-cli-manual>`__."
+"`CodeQL CLI manual <../manual>`__."

…lyzing-databases-with-the-codeql-cli.rst …lyzing-databases-with-the-codeql-cli.rstdocs/codeql/codeql-cli/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli.rst renamed to docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst docs/codeql/codeql-cli/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli.rst renamed to docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst

@@ -11,7 +11,7 @@ CodeQL analyses produce :ref:`interpreted results
 For information about writing queries to run with ``database analyze``, see
 ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
 
-.. include:: ../../reusables/advanced-query-execution.rst
+.. include:: ../reusables/advanced-query-execution.rst
 
 Before starting an analysis you must:
 
@@ -49,13 +49,13 @@ You must specify:
   <sarif-file>`, and graph formats. For more information about CSV and SARIF,
   see `Results <#results>`__. To find out which other results formats are
   supported, see the `database analyze reference
-  <../codeql-cli-manual/database-analyze.html>`__.
+  <../manual/database-analyze>`__.
 
 - ``--output``: the output path of the results file generated during analysis.
 
 You can also specify:
 
-- .. include:: ../../reusables/threads-query-execution.rst
+- .. include:: ../reusables/threads-query-execution.rst
 
 
 .. pull-quote:: 
@@ -70,7 +70,7 @@ You can also specify:
    <upgrading-codeql-databases>`."
 
 For full details of all the options you can use when analyzing databases, see
-the `database analyze reference documentation <../codeql-cli-manual/database-analyze.html>`__.
+the `database analyze reference documentation <../manual/database-analyze>`__.
 
 .. _database-analyze-examples:
 
@@ -149,7 +149,7 @@ recursively, so any queries contained in subfolders will also be executed.
    Important
 
    You shouldn't specify the root of a :doc:`QL pack
-   <../codeql-cli-reference/about-ql-packs>` when executing ``database analyze``
+   <about-ql-packs>` when executing ``database analyze``
    as it contains some special queries that aren't designed to be used with
    the command. Rather, to run a wide range of useful queries, run one of the
    LGTM.com query suites. 
@@ -171,7 +171,7 @@ You can save analysis results in a number of different formats, including SARIF
 and CSV.
 
 The SARIF format is designed to represent the output of a broad range of static
-analysis tools. For more information, see :doc:`SARIF output <../codeql-cli-reference/sarif-output>`.
+analysis tools. For more information, see :doc:`SARIF output <sarif-output>`.
 
 If you choose to generate results in CSV format, then each line in the output file
 corresponds to an alert. Each line is a comma-separated list with the following information:

…odeql-cli/codeql-cli-reference/index.rst …deql/codeql-cli/codeql-cli-reference.rstdocs/codeql/codeql-cli/codeql-cli-reference/index.rst renamed to docs/codeql/codeql-cli/codeql-cli-reference.rst docs/codeql/codeql-cli/codeql-cli-reference/index.rst renamed to docs/codeql/codeql-cli/codeql-cli-reference.rst

@@ -28,4 +28,4 @@ CodeQL CLI manual
 -----------------
 
 To view detailed information about each CodeQL CLI command,
-including its usage and options, add the ``--help`` flag or visit the "`CodeQL CLI manual <../codeql-cli-manual>`__."
+including its usage and options, add the ``--help`` flag or visit the "`CodeQL CLI manual <../manual>`__."

…codeql-cli/creating-codeql-databases.rst …codeql-cli/creating-codeql-databases.rstdocs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-databases.rst renamed to docs/codeql/codeql-cli/creating-codeql-databases.rst docs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-databases.rst renamed to docs/codeql/codeql-cli/creating-codeql-databases.rst

@@ -35,7 +35,7 @@ You must specify:
 - ``--language``: the identifier for the language to create a database for.
   CodeQL supports creating databases for the following languages:
 
-  .. include:: ../../reusables/extractors.rst
+  .. include:: ../reusables/extractors.rst
 
 Other options may be specified depending on the location of your source file and
 the language you want to analyze:
@@ -50,7 +50,7 @@ the language you want to analyze:
   detect the build system automatically, using a built-in autobuilder. 
 
 For full details of all the options you can use when creating databases,
-see the `database create reference documentation <../codeql-cli-manual/database-create.html>`__.  
+see the `database create reference documentation <../manual/database-create>`__.  
 
 Progress and results
 --------------------
@@ -75,7 +75,7 @@ CodeQL. For each project on LGTM.com, you can download an archived CodeQL
 database corresponding to the most recently analyzed revision of the code. These
 databases can also be analyzed using the CodeQL CLI. 
 
-.. include:: ../../reusables/download-lgtm-database.rst
+.. include:: ../reusables/download-lgtm-database.rst
 
 Before running an analysis, unzip the databases and try :doc:`upgrading <upgrading-codeql-databases>` the
 unzipped databases to ensure they are compatible with your local copy of the
@@ -85,7 +85,7 @@ CodeQL queries and libraries.
 
    Note
 
-   .. include:: ../../reusables/index-files-note.rst
+   .. include:: ../reusables/index-files-note.rst
 
 Creating databases for non-compiled languages
 ---------------------------------------------

…eql-cli/creating-codeql-query-suites.rst …eql-cli/creating-codeql-query-suites.rstdocs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-query-suites.rst renamed to docs/codeql/codeql-cli/creating-codeql-query-suites.rst docs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-query-suites.rst renamed to docs/codeql/codeql-cli/creating-codeql-query-suites.rst

@@ -19,7 +19,7 @@ suite definition have been executed, the result is a set of selected queries.
 .. note::
 
    Any custom queries that you want to add to a query suite must be in a :doc:`QL
-   pack <../codeql-cli-reference/about-ql-packs>` and contain the correct query metadata. 
+   pack <about-ql-packs>` and contain the correct query metadata. 
    For more information, see
    ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
 
@@ -234,7 +234,7 @@ instruction::
    - description: <name-of-query-suite>
 
 This value is displayed when you run `codeql resolve queries
-<../codeql-cli-manual/resolve-queries.html>`__, if the suite is added to a "well-known"
+<../manual/resolve-queries>`__, if the suite is added to a "well-known"
 directory. For more information, see "`Specifying well-known query suites
 <#specifying-well-known-query-suites>`__." 
 
@@ -254,7 +254,7 @@ without providing their full path. This gives you a simple way of specifying a
 set of queries, without needing to search inside QL packs and distributions.
 To declare a directory that contains "well-known" query suites, add the directory
 to the ``suites`` property in the ``qlpack.yml`` file at the root of your QL pack.
-For more information, see "`About QL packs <../codeql-cli-reference/qlpack-overview.html#qlpack-yml-properties>`__."
+For more information, see "`About QL packs <qlpack-overview.html#qlpack-yml-properties>`__."
 
 Using query suites with CodeQL
 ------------------------------