Playframework test cases & review fixes

Author: torque59

.github/workflows/generate-query-help-docs.yml

@@ -0,0 +1,60 @@
+name: Generate CodeQL query help documentation using Sphinx
+
+on:
+  workflow_dispatch:
+      inputs:
+        description:
+          description: A description of the purpose of this job. For human consumption.
+          required: false
+  push:
+    branches:
+     - 'lgtm.com'
+  pull_request:
+    paths:
+      - '.github/workflows/generate-query-help-docs.yml'
+      - 'docs/codeql/query-help/**'
+      
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Clone github/codeql
+      uses: actions/checkout@v2
+      with:
+        path: codeql
+    - name: Clone github/codeql-go
+      uses: actions/checkout@v2
+      with:
+        repository: 'github/codeql-go'
+        path: codeql-go
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+    - name: Download CodeQL CLI
+      uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
+      with:
+        repo: "github/codeql-cli-binaries"
+        version: "latest"
+        file: "codeql-linux64.zip"
+        token: ${{ secrets.GITHUB_TOKEN }}
+    - name: Unzip CodeQL CLI
+      run: unzip -d codeql-cli codeql-linux64.zip
+    - name: Set up query help docs folder  
+      run: |
+        cp -r codeql/docs/codeql/** .
+    - name: Query help to markdown
+      run: |
+        PATH="$PATH:codeql-cli/codeql" python codeql/docs/codeql/query-help-markdown.py
+    - name: Run Sphinx for query help
+      uses: ammaraskar/sphinx-action@8b4f60114d7fd1faeba1a712269168508d4750d2 # v0.4
+      with:
+        docs-folder: "query-help/"
+        pre-build-command: "python -m pip install --upgrade recommonmark"
+        build-command: "sphinx-build -b dirhtml . _build" 
+    - name: Upload HTML artifacts
+      uses: actions/upload-artifact@v2
+      with:
+        name: query-help-html
+        path: query-help/_build
+    

CONTRIBUTING.md

@@ -38,6 +38,8 @@ If you have an idea for a query that you would like to share with other CodeQL u
 
     - The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://help.semmle.com/codeql/codeql-for-vscode/procedures/about-codeql-for-vscode.html).
 
+    If you prefer, you can use this [pre-commit hook](misc/scripts/pre-commit) that automatically checks whether your files are correctly formatted. See the [pre-commit hook installation guide](docs/install-pre-commit-hook.md) for instructions on how to install the hook.
+
 4. **Compilation**
 
     - Compilation of the query and any associated libraries and tests must be resilient to future development of the [supported](docs/supported-queries.md) libraries. This means that the functionality cannot use internal libraries, cannot depend on the output of `getAQlClass`, and cannot make use of regexp matching on `toString`.

change-notes/1.26/analysis-csharp.md

@@ -18,4 +18,3 @@ The following changes in version 1.26 affect Java analysis in all applications.
 
 ## Changes to libraries
 
-* The QL class `Block`, denoting the `{ ... }` statement, is renamed to `BlockStmt`.

change-notes/1.26/analysis-java.md

@@ -4,13 +4,19 @@
 
 * Angular-specific taint sources and sinks are now recognized by the security queries.
 
+* Support for React has improved, with better handling of react hooks, react-router path parameters, lazy-loaded components, and components transformed using `react-redux` and/or `styled-components`.
+
+* Dynamic imports are now analyzed more precisely.
+
 * Support for the following frameworks and libraries has been improved:
   - [@angular/*](https://www.npmjs.com/package/@angular/core)
   - [AWS Serverless](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html)
   - [Alibaba Serverless](https://www.alibabacloud.com/help/doc-detail/156876.htm)
   - [debounce](https://www.npmjs.com/package/debounce)
   - [bluebird](https://www.npmjs.com/package/bluebird)
   - [call-limit](https://www.npmjs.com/package/call-limit)
+  - [classnames](https://www.npmjs.com/package/classnames)
+  - [clsx](https://www.npmjs.com/package/clsx)
   - [express](https://www.npmjs.com/package/express)
   - [fast-json-stable-stringify](https://www.npmjs.com/package/fast-json-stable-stringify)
   - [fast-safe-stringify](https://www.npmjs.com/package/fast-safe-stringify)
@@ -27,7 +33,13 @@
   - [needle](https://www.npmjs.com/package/needle)
   - [object-inspect](https://www.npmjs.com/package/object-inspect)
   - [pretty-format](https://www.npmjs.com/package/pretty-format)
+  - [react](https://www.npmjs.com/package/react)
+  - [react-router-dom](https://www.npmjs.com/package/react-router-dom)
+  - [react-redux](https://www.npmjs.com/package/react-redux)
+  - [redis](https://www.npmjs.com/package/redis)
+  - [redux](https://www.npmjs.com/package/redux)
   - [stringify-object](https://www.npmjs.com/package/stringify-object)
+  - [styled-components](https://www.npmjs.com/package/styled-components)
   - [throttle-debounce](https://www.npmjs.com/package/throttle-debounce)
   - [underscore](https://www.npmjs.com/package/underscore)
 

change-notes/1.26/analysis-javascript.md

@@ -4,19 +4,34 @@ The following changes in version 1.26 affect Python analysis in all applications
 
 ## General improvements
 
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
 ## Changes to existing queries
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
-
-
+|`py/unsafe-deserialization` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/path-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/command-line-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/reflective-xss` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/sql-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
+|`py/code-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
 ## Changes to libraries
-
+* Some of the security queries now use the shared data flow library for data flow and taint tracking. This has resulted in an overall more robust and accurate analysis. The libraries mentioned below have been modelled in this new framework. Other libraries (e.g. the web framework `CherryPy`) have not been modelled yet, and this may lead to a temporary loss of results for these frameworks.
+* Improved modelling of the following serialization libraries:
+  - `PyYAML`
+  - `dill`
+  - `pickle`
+  - `marshal`
+* Improved modelling of the following web frameworks:
+  - `Django` (Note that modelling of class-based response handlers is currently incomplete.)
+  - `Flask`
+* Support for Werkzeug `MultiDict`.
+* Support for the [Python Database API Specification v2.0 (PEP-249)](https://www.python.org/dev/peps/pep-0249/), including the following libraries:
+  - `MySQLdb`
+  - `mysql-connector-python`
+  - `django.db`
+* Improved modelling of the following command execution libraries:
+  - `Fabric`
+  - `Invoke`
+* Improved modelling of security-related standard library modules, such as `os`, `popen2`, `platform`, and `base64`.
+* The original versions of the updated queries have been preserved [here](https://github.com/github/codeql/tree/main/python/ql/src/experimental/Security-old-dataflow).
 * Added taint tracking support for string formatting through f-strings.

change-notes/1.26/analysis-python.md

@@ -19,15 +19,17 @@
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl3.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl4.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl5.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImpl.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImpl2.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl2.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl3.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl4.qll"
   ],
   "DataFlow Java/C++/C#/Python Common": [
     "java/ql/src/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll",
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowImplCommon.qll",
     "cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImplCommon.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImplCommon.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll"
   ],
   "TaintTracking::Configuration Java/C++/C#/Python": [
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
@@ -41,14 +43,17 @@
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/tainttracking5/TaintTrackingImpl.qll",
     "java/ql/src/semmle/code/java/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
     "java/ql/src/semmle/code/java/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
-    "python/ql/src/experimental/dataflow/internal/tainttracking1/TaintTrackingImpl.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking1/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking2/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking3/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking4/TaintTrackingImpl.qll"
   ],
   "DataFlow Java/C++/C#/Python Consistency checks": [
     "java/ql/src/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll",
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowImplConsistency.qll",
     "cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImplConsistency.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll"
   ],
   "SsaReadPosition Java/C#": [
     "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll",
@@ -345,10 +350,22 @@
     "csharp/ql/src/experimental/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
     "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll"
   ],
+  "C# ControlFlowReachability": [
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/ControlFlowReachability.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/ControlFlowReachability.qll"
+  ],
   "Inline Test Expectations": [
     "cpp/ql/test/TestUtilities/InlineExpectationsTest.qll",
     "python/ql/test/TestUtilities/InlineExpectationsTest.qll"
   ],
+  "C++ ExternalAPIs": [
+    "cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll",
+    "cpp/ql/src/Security/CWE/CWE-020/ir/ExternalAPIs.qll"
+  ],
+  "C++ SafeExternalAPIFunction": [
+    "cpp/ql/src/Security/CWE/CWE-020/SafeExternalAPIFunction.qll",
+    "cpp/ql/src/Security/CWE/CWE-020/ir/SafeExternalAPIFunction.qll"
+  ],
   "XML": [
     "cpp/ql/src/semmle/code/cpp/XML.qll",
     "csharp/ql/src/semmle/code/csharp/XML.qll",
@@ -400,5 +417,12 @@
     "java/ql/src/Metrics/Files/CommentedOutCodeReferences.qhelp",
     "javascript/ql/src/Comments/CommentedOutCodeReferences.qhelp",
     "python/ql/src/Lexical/CommentedOutCodeReferences.qhelp"
+  ],
+  "IDE Contextual Queries": [
+    "cpp/ql/src/IDEContextual.qll",
+    "csharp/ql/src/IDEContextual.qll",
+    "java/ql/src/IDEContextual.qll",
+    "javascript/ql/src/IDEContextual.qll",
+    "python/ql/src/analysis/IDEContextual.qll"
   ]
-}
+}

config/identical-files.json

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The `cpp/wrong-type-format-argument` and `cpp/non-portable-printf` queries have been hardened so that they do not produce nonsensical results on databases that contain errors (specifically the `ErroneousType`).

cpp/change-notes/2020-10-21-erroneous-types.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Two issues causing the 'Unused local variable' query (`cpp/unused-local-variable`) to produce false positive results have been fixed.

cpp/change-notes/2020-11-02-unused-local-variable.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* `FormattingFunction.getOutputParameterIndex` now has a parameter identifying whether the output at that index is a buffer or a stream.
+* `FormattingFunction` now has a predicate `isOutputGlobal` indicating when the output is to a global stream.
+* The `primitiveVariadicFormatter` and `variadicFormatter` predicates have more parameters exposing information about the function.