Merge branch 'main' of github.com:github/codeql into python-dataflow-modernize-tests

Author: yoff

.github/workflows/generate-query-help-docs.yml

@@ -0,0 +1,60 @@
+name: Generate CodeQL query help documentation using Sphinx
+
+on:
+  workflow_dispatch:
+      inputs:
+        description:
+          description: A description of the purpose of this job. For human consumption.
+          required: false
+  push:
+    branches:
+     - 'lgtm.com'
+  pull_request:
+    paths:
+      - '.github/workflows/generate-query-help-docs.yml'
+      - 'docs/codeql/query-help/**'
+      
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Clone github/codeql
+      uses: actions/checkout@v2
+      with:
+        path: codeql
+    - name: Clone github/codeql-go
+      uses: actions/checkout@v2
+      with:
+        repository: 'github/codeql-go'
+        path: codeql-go
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+    - name: Download CodeQL CLI
+      uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
+      with:
+        repo: "github/codeql-cli-binaries"
+        version: "latest"
+        file: "codeql-linux64.zip"
+        token: ${{ secrets.GITHUB_TOKEN }}
+    - name: Unzip CodeQL CLI
+      run: unzip -d codeql-cli codeql-linux64.zip
+    - name: Set up query help docs folder  
+      run: |
+        cp -r codeql/docs/codeql/** .
+    - name: Query help to markdown
+      run: |
+        PATH="$PATH:codeql-cli/codeql" python codeql/docs/codeql/query-help-markdown.py
+    - name: Run Sphinx for query help
+      uses: ammaraskar/sphinx-action@8b4f60114d7fd1faeba1a712269168508d4750d2 # v0.4
+      with:
+        docs-folder: "query-help/"
+        pre-build-command: "python -m pip install --upgrade recommonmark"
+        build-command: "sphinx-build -b dirhtml . _build" 
+    - name: Upload HTML artifacts
+      uses: actions/upload-artifact@v2
+      with:
+        name: query-help-html
+        path: query-help/_build
+    

CONTRIBUTING.md

@@ -38,6 +38,8 @@ If you have an idea for a query that you would like to share with other CodeQL u
 
     - The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://help.semmle.com/codeql/codeql-for-vscode/procedures/about-codeql-for-vscode.html).
 
+    If you prefer, you can use this [pre-commit hook](misc/scripts/pre-commit) that automatically checks whether your files are correctly formatted. See the [pre-commit hook installation guide](docs/install-pre-commit-hook.md) for instructions on how to install the hook.
+
 4. **Compilation**
 
     - Compilation of the query and any associated libraries and tests must be resilient to future development of the [supported](docs/supported-queries.md) libraries. This means that the functionality cannot use internal libraries, cannot depend on the output of `getAQlClass`, and cannot make use of regexp matching on `toString`.

change-notes/1.26/analysis-python.md

@@ -4,19 +4,34 @@ The following changes in version 1.26 affect Python analysis in all applications
 
 ## General improvements
 
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
 ## Changes to existing queries
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
-
-
+|`py/unsafe-deserialization` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/path-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/command-line-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/reflective-xss` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/sql-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/code-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
 ## Changes to libraries
-
+* Some of the security queries now use the shared data flow library for data flow and taint tracking. This has resulted in an overall more robust and accurate analysis. The libraries mentioned below have been modelled in this new framework. Other libraries (e.g. the web framework `CherryPy`) have not been modelled yet, and this may lead to a temporary loss of results for these frameworks.
+* Improved modelling of the following serialization libraries:
+  - `PyYAML`
+  - `dill`
+  - `pickle`
+  - `marshal`
+* Improved modelling of the following web frameworks:
+  - `Django` (Note that modelling of class-based response handlers is currently incomplete.)
+  - `Flask`
+* Support for Werkzeug `MultiDict`.
+* Support for the [Python Database API Specification v2.0 (PEP-249)](https://www.python.org/dev/peps/pep-0249/), including the following libraries:
+  - `MySQLdb`
+  - `mysql-connector-python`
+  - `django.db`
+* Improved modelling of the following command execution libraries:
+  - `Fabric`
+  - `Invoke`
+* Improved modelling of security-related standard library modules, such as `os`, `popen2`, `platform`, and `base64`.
+* The original versions of the updated queries have been preserved [here](https://github.com/github/codeql/tree/main/python/ql/src/experimental/Security-old-dataflow).
 * Added taint tracking support for string formatting through f-strings.

cpp/change-notes/2020-11-05-formatting-function.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* `FormattingFunction.getOutputParameterIndex` now has a parameter identifying whether the output at that index is a buffer or a stream.
+* `FormattingFunction` now has a predicate `isOutputGlobal` indicating when the output is to a global stream.
+* The `primitiveVariadicFormatter` and `variadicFormatter` predicates have more parameters exposing information about the function.

cpp/change-notes/2020-11-05-private-models.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Various classes in `semmle.code.cpp.models.implementations` have been made private. Users should not depend on library implementation details.
+* The `OperatorNewAllocationFunction`, `OperatorDeleteDeallocationFunction`, `Iterator` and `Snprintf` classes now have interfaces in `semmle.code.cpp.models.interfaces`.

cpp/change-notes/2020-11-12-unsafe-use-of-this.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* A new query (`cpp/unsafe-use-of-this`) has been added. The query finds pure virtual function calls whose qualifier is an object under construction.

cpp/change-notes/2020-11-27-downgrade-to-recommendation.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The queries `cpp/local-variable-hides-global-variable` and `cpp/missing-header-guard` now have severity `recommendation` instead of `warning`.

cpp/config/suites/cpp/correctness

@@ -9,6 +9,7 @@
 + semmlecode-cpp-queries/Likely Bugs/Conversion/CastArrayPointerArithmetic.ql: /Correctness/Dangerous Conversions
 + semmlecode-cpp-queries/Likely Bugs/Underspecified Functions/MistypedFunctionArguments.ql: /Correctness/Dangerous Conversions
 + semmlecode-cpp-queries/Security/CWE/CWE-253/HResultBooleanConversion.ql: /Correctness/Dangerous Conversions
++ semmlecode-cpp-queries/Likely Bugs/OO/UnsafeUseOfThis.ql: /Correctness/Dangerous Conversions
   # Consistent Use
 + semmlecode-cpp-queries/Critical/ReturnValueIgnored.ql: /Correctness/Consistent Use
 + semmlecode-cpp-queries/Likely Bugs/InconsistentCheckReturnNull.ql: /Correctness/Consistent Use

cpp/ql/src/Architecture/General Class-Level Information/ClassHierarchies.ql

@@ -4,9 +4,6 @@
  * @kind graph
  * @id cpp/architecture/class-hierarchies
  * @graph.layout organic
- * @workingset jhotdraw
- * @result succeed 48
- * @result_ondemand succeed 48
  * @tags maintainability
  */
 

cpp/ql/src/Architecture/General Class-Level Information/InheritanceDepthDistribution.ql

@@ -4,9 +4,6 @@
  * @kind chart
  * @id cpp/architecture/inheritance-depth-distribution
  * @chart.type line
- * @workingset jhotdraw
- * @result succeed 48
- * @result_ondemand succeed 48
  * @tags maintainability
  */