Skip to content

Commit 748f534

Browse files
MathiasVPMathiasVP
committed
Merge branch 'main' into rdmarsh/cpp/use-taint-configuration-dtt
2 parents 7772991 + cee9677 commit 748f534

File tree

2,421 files changed

+264172
-76086
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,421 files changed

+264172
-76086
lines changed

‎.github/workflows/check-change-note.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
on:
2+
pull_request_target:
3+
types: [labeled, unlabeled, opened, synchronize, reopened, ready_for_review]
4+
paths:
5+
- "*/ql/src/**/*.ql"
6+
- "*/ql/src/**/*.qll"
7+
- "!**/experimental/**"
8+
9+
jobs:
10+
check-change-note:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Fail if no change note found. To fix, either add one, or add the `no-change-note-required` label.
14+
if: |
15+
github.event.pull_request.draft == false &&
16+
!contains(github.event.pull_request.labels.*.name, 'no-change-note-required')
17+
env:
18+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
19+
run: |
20+
gh api 'repos/${{github.repository}}/pulls/${{github.event.number}}/files' --paginate |
21+
jq 'any(.[].filename ; test("/change-notes/.*[.]md$"))' --exit-status

‎.github/workflows/codeql-analysis.yml‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,15 @@ name: "Code scanning - action"
22

33
on:
44
push:
5+
branches:
6+
- main
7+
- 'rc/*'
58
pull_request:
9+
branches:
10+
- main
11+
- 'rc/*'
12+
paths:
13+
- 'csharp/**'
614
schedule:
715
- cron: '0 9 * * 1'
816

@@ -14,16 +22,7 @@ jobs:
1422
steps:
1523
- name: Checkout repository
1624
uses: actions/checkout@v2
17-
with:
18-
# We must fetch at least the immediate parents so that if this is
19-
# a pull request then we can checkout the head.
20-
fetch-depth: 2
21-
22-
# If this run was triggered by a pull request event, then checkout
23-
# the head of the pull request instead of the merge commit.
24-
- run: git checkout HEAD^2
25-
if: ${{ github.event_name == 'pull_request' }}
26-
25+
2726
# Initializes the CodeQL tools for scanning.
2827
- name: Initialize CodeQL
2928
uses: github/codeql-action/init@v1

‎.github/workflows/generate-query-help-docs.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: Generate CodeQL query help documentation using Sphinx
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
description:
7+
description: A description of the purpose of this job. For human consumption.
8+
required: false
9+
push:
10+
branches:
11+
- 'lgtm.com'
12+
pull_request:
13+
paths:
14+
- '.github/workflows/generate-query-help-docs.yml'
15+
- 'docs/codeql/query-help/**'
16+
17+
jobs:
18+
build:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Clone github/codeql
22+
uses: actions/checkout@v2
23+
with:
24+
path: codeql
25+
- name: Clone github/codeql-go
26+
uses: actions/checkout@v2
27+
with:
28+
repository: 'github/codeql-go'
29+
path: codeql-go
30+
- name: Set up Python 3.8
31+
uses: actions/setup-python@v2
32+
with:
33+
python-version: 3.8
34+
- name: Download CodeQL CLI
35+
uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
36+
with:
37+
repo: "github/codeql-cli-binaries"
38+
version: "latest"
39+
file: "codeql-linux64.zip"
40+
token: ${{ secrets.GITHUB_TOKEN }}
41+
- name: Unzip CodeQL CLI
42+
run: unzip -d codeql-cli codeql-linux64.zip
43+
- name: Set up query help docs folder
44+
run: |
45+
cp -r codeql/docs/codeql/** .
46+
- name: Query help to markdown
47+
run: |
48+
PATH="$PATH:codeql-cli/codeql" python codeql/docs/codeql/query-help-markdown.py
49+
- name: Run Sphinx for query help
50+
uses: ammaraskar/sphinx-action@8b4f60114d7fd1faeba1a712269168508d4750d2 # v0.4
51+
with:
52+
docs-folder: "query-help/"
53+
pre-build-command: "python -m pip install --upgrade recommonmark && python -m pip install --upgrade sphinx-markdown-tables"
54+
build-command: "sphinx-build -b dirhtml . _build"
55+
- name: Upload HTML artifacts
56+
uses: actions/upload-artifact@v2
57+
with:
58+
name: query-help-html
59+
path: query-help/_build
60+

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@
1717
# Byte-compiled python files
1818
*.pyc
1919

20+
# python virtual environment folder
21+
.venv/
22+
2023
# It's useful (though not required) to be able to unpack codeql in the ql checkout itself
2124
/codeql/
2225

‎CODEOWNERS‎

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,9 @@
44
/javascript/ @github/codeql-javascript
55
/python/ @github/codeql-python
66

7-
# Assign query help for docs review
8-
/cpp/**/*.qhelp @hubwriter
9-
/csharp/**/*.qhelp @jf205
10-
/java/**/*.qhelp @felicitymay
11-
/javascript/**/*.qhelp @mchammer01
12-
/python/**/*.qhelp @felicitymay
13-
/docs/language/ @shati-patel @jf205
14-
15-
# Exclude help for experimental queries from docs review
16-
/cpp/**/experimental/**/*.qhelp @github/codeql-c-analysis
17-
/csharp/**/experimental/**/*.qhelp @github/codeql-csharp
18-
/java/**/experimental/**/*.qhelp @github/codeql-java
19-
/javascript/**/experimental/**/*.qhelp @github/codeql-javascript
20-
/python/**/experimental/**/*.qhelp @github/codeql-python
7+
# Make @xcorail (GitHub Security Lab) a code owner for experimental queries so he gets pinged when we promote a query out of experimental
8+
/cpp/**/experimental/**/* @github/codeql-c-analysis @xcorail
9+
/csharp/**/experimental/**/* @github/codeql-csharp @xcorail
10+
/java/**/experimental/**/* @github/codeql-java @xcorail
11+
/javascript/**/experimental/**/* @github/codeql-javascript @xcorail
12+
/python/**/experimental/**/* @github/codeql-python @xcorail

‎CONTRIBUTING.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ If you have an idea for a query that you would like to share with other CodeQL u
3838

3939
- The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://help.semmle.com/codeql/codeql-for-vscode/procedures/about-codeql-for-vscode.html).
4040

41+
If you prefer, you can use this [pre-commit hook](misc/scripts/pre-commit) that automatically checks whether your files are correctly formatted. See the [pre-commit hook installation guide](docs/install-pre-commit-hook.md) for instructions on how to install the hook.
42+
4143
4. **Compilation**
4244

4345
- Compilation of the query and any associated libraries and tests must be resilient to future development of the [supported](docs/supported-queries.md) libraries. This means that the functionality cannot use internal libraries, cannot depend on the output of `getAQlClass`, and cannot make use of regexp matching on `toString`.
@@ -47,7 +49,11 @@ If you have an idea for a query that you would like to share with other CodeQL u
4749

4850
- The query must have at least one true positive result on some revision of a real project.
4951

50-
Experimental queries and libraries may not be actively maintained as the [supported](docs/supported-queries.md) libraries evolve. They may also be changed in backwards-incompatible ways or may be removed entirely in the future without deprecation warnings.
52+
6. **Query help files and unit tests**
53+
54+
- Query help (`.qhelp`) files and unit tests are optional (but strongly encouraged!) for queries in the `experimental` directories. For more information about contributing query help files and unit tests, see [Supported CodeQL queries and libraries](docs/supported-queries.md).
55+
56+
Experimental queries and libraries may not be actively maintained as the supported libraries evolve. They may also be changed in backwards-incompatible ways or may be removed entirely in the future without deprecation warnings.
5157

5258
After the experimental query is merged, we welcome pull requests to improve it. Before a query can be moved out of the `experimental` subdirectory, it must satisfy [the requirements for being a supported query](docs/supported-queries.md).
5359

‎change-notes/1.26/analysis-python.md‎

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,34 @@ The following changes in version 1.26 affect Python analysis in all applications
44

55
## General improvements
66

7-
8-
## New queries
9-
10-
| **Query** | **Tags** | **Purpose** |
11-
|-----------------------------|-----------|--------------------------------------------------------------------|
12-
13-
147
## Changes to existing queries
158

169
| **Query** | **Expected impact** | **Change** |
1710
|----------------------------|------------------------|------------------------------------------------------------------|
18-
19-
11+
|`py/unsafe-deserialization` | Different results. | The underlying data flow library has been changed. See below for more details. |
12+
|`py/path-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
13+
|`py/command-line-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
14+
|`py/reflective-xss` | Different results. | The underlying data flow library has been changed. See below for more details. |
15+
|`py/sql-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
16+
|`py/code-injection` | Different results. | The underlying data flow library has been changed. See below for more details. |
2017
## Changes to libraries
21-
18+
* Some of the security queries now use the shared data flow library for data flow and taint tracking. This has resulted in an overall more robust and accurate analysis. The libraries mentioned below have been modelled in this new framework. Other libraries (e.g. the web framework `CherryPy`) have not been modelled yet, and this may lead to a temporary loss of results for these frameworks.
19+
* Improved modelling of the following serialization libraries:
20+
- `PyYAML`
21+
- `dill`
22+
- `pickle`
23+
- `marshal`
24+
* Improved modelling of the following web frameworks:
25+
- `Django` (Note that modelling of class-based response handlers is currently incomplete.)
26+
- `Flask`
27+
* Support for Werkzeug `MultiDict`.
28+
* Support for the [Python Database API Specification v2.0 (PEP-249)](https://www.python.org/dev/peps/pep-0249/), including the following libraries:
29+
- `MySQLdb`
30+
- `mysql-connector-python`
31+
- `django.db`
32+
* Improved modelling of the following command execution libraries:
33+
- `Fabric`
34+
- `Invoke`
35+
* Improved modelling of security-related standard library modules, such as `os`, `popen2`, `platform`, and `base64`.
36+
* The original versions of the updated queries have been preserved [here](https://github.com/github/codeql/tree/main/python/ql/src/experimental/Security-old-dataflow).
2237
* Added taint tracking support for string formatting through f-strings.

‎config/identical-files.json‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,8 +356,17 @@
356356
],
357357
"Inline Test Expectations": [
358358
"cpp/ql/test/TestUtilities/InlineExpectationsTest.qll",
359+
"java/ql/test/TestUtilities/InlineExpectationsTest.qll",
359360
"python/ql/test/TestUtilities/InlineExpectationsTest.qll"
360361
],
362+
"C++ ExternalAPIs": [
363+
"cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll",
364+
"cpp/ql/src/Security/CWE/CWE-020/ir/ExternalAPIs.qll"
365+
],
366+
"C++ SafeExternalAPIFunction": [
367+
"cpp/ql/src/Security/CWE/CWE-020/SafeExternalAPIFunction.qll",
368+
"cpp/ql/src/Security/CWE/CWE-020/ir/SafeExternalAPIFunction.qll"
369+
],
361370
"XML": [
362371
"cpp/ql/src/semmle/code/cpp/XML.qll",
363372
"csharp/ql/src/semmle/code/csharp/XML.qll",
@@ -416,5 +425,10 @@
416425
"java/ql/src/IDEContextual.qll",
417426
"javascript/ql/src/IDEContextual.qll",
418427
"python/ql/src/analysis/IDEContextual.qll"
428+
],
429+
"SSA C#": [
430+
"csharp/ql/src/semmle/code/csharp/dataflow/internal/SsaImplCommon.qll",
431+
"csharp/ql/src/semmle/code/csharp/controlflow/internal/pressa/SsaImplCommon.qll",
432+
"csharp/ql/src/semmle/code/csharp/dataflow/internal/basessa/SsaImplCommon.qll"
419433
]
420-
}
434+
}

‎cpp/change-notes/2020-11-05-formatting-function.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
lgtm,codescanning
2+
* `FormattingFunction.getOutputParameterIndex` now has a parameter identifying whether the output at that index is a buffer or a stream.
3+
* `FormattingFunction` now has a predicate `isOutputGlobal` indicating when the output is to a global stream.
4+
* The `primitiveVariadicFormatter` and `variadicFormatter` predicates have more parameters exposing information about the function.

‎cpp/change-notes/2020-11-05-private-models.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
lgtm,codescanning
2+
* Various classes in `semmle.code.cpp.models.implementations` have been made private. Users should not depend on library implementation details.
3+
* The `OperatorNewAllocationFunction`, `OperatorDeleteDeallocationFunction`, `Iterator` and `Snprintf` classes now have interfaces in `semmle.code.cpp.models.interfaces`.

0 commit comments

Comments
 (0)