Skip to content

Commit 989d7ae

Browse files
markshannonmarkshannon
committed
Merge branch 'master' into python-cwe-312
2 parents 8909c3d + 1215da2 commit 989d7ae

File tree

359 files changed

+39854
-17161
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

359 files changed

+39854
-17161
lines changed

‎change-notes/1.22/analysis-cpp.md‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,42 @@
11
# Improvements to C/C++ analysis
22

3-
## General improvements
4-
5-
## New queries
6-
7-
| **Query** | **Tags** | **Purpose** |
8-
|-----------------------------|-----------|--------------------------------------------------------------------|
3+
The following changes in version 1.22 affect C/C++ analysis in all applications.
94

105
## Changes to existing queries
116

127
| **Query** | **Expected impact** | **Change** |
138
|----------------------------|------------------------|------------------------------------------------------------------|
14-
| Call to alloca in a loop (`cpp/alloca-in-loop`) | Fewer false positive results | Fixed false positives where the stack allocation could not be reached multiple times in the loop, typically due to a `break` or `return` statement. |
9+
| Call to alloca in a loop (`cpp/alloca-in-loop`) | Fewer false positive results | The query no longer highlights code where the stack allocation could not be reached multiple times in the loop, typically due to a `break` or `return` statement. |
1510
| Continue statement that does not continue (`cpp/continue-in-false-loop`) | Fewer false positive results | Analysis is now restricted to `do`-`while` loops. This query is now run and displayed by default on LGTM. |
16-
| Expression has no effect (`cpp/useless-expression`) | Fewer false positive results | Calls to functions with the `weak` attribute are no longer considered to be side effect free, because they could be overridden with a different implementation at link time. |
17-
| No space for zero terminator (`cpp/no-space-for-terminator`) | Fewer false positive results | False positives involving strings that are not null-terminated have been excluded. |
18-
| Sign check of bitwise operation (`cpp/bitwise-sign-check`) | Fewer false positive results and more true positive results | The query now understands the direction of each comparison, making it more accurate. |
11+
| Expression has no effect (`cpp/useless-expression`) | Fewer false positive results | Calls to functions with the `weak` attribute are no longer considered to be side-effect free, because they could be overridden with a different implementation at link time. |
12+
| No space for zero terminator (`cpp/no-space-for-terminator`) | Fewer false positive results | False positive results for strings that are not null-terminated have been excluded. |
13+
| Non-constant format string (`cpp/non-constant-format`) | Fewer false positive results | The query was rewritten using the taint-tracking library. |
14+
| Sign check of bitwise operation (`cpp/bitwise-sign-check`) | Fewer false positive and more true positive results | The query now understands the direction of each comparison, making it more accurate. |
1915
| Suspicious pointer scaling (`cpp/suspicious-pointer-scaling`) | Lower precision | The precision of this query has been reduced to "medium". This coding pattern is used intentionally and safely in a number of real-world projects. Results are no longer displayed on LGTM unless you choose to display them. |
20-
| Non-constant format string (`cpp/non-constant-format`) | Fewer false positive results | Rewritten using the taint-tracking library. |
21-
| Variable used in its own initializer (`cpp/use-in-own-initializer`) | Fewer false positive results | False positives for constant variables with the same name in different namespaces have been removed. |
16+
| Variable used in its own initializer (`cpp/use-in-own-initializer`) | Fewer false positive results | False positive results for constant variables with the same name in different namespaces have been removed. |
2217

2318
## Changes to QL libraries
2419

20+
- The data flow library (`semmle.code.cpp.dataflow.DataFlow`) has had the
21+
following improvements, all of which benefit the taint tracking library
22+
(`semmle.code.cpp.dataflow.TaintTracking`) as well.
23+
- This release includes preliminary support for interprocedural flow through
24+
fields (non-static data members). In some cases, data stored in a field in
25+
one function can now flow to a read of the same field in a different
26+
function.
27+
- The possibility of specifying barrier edges using
28+
`isBarrierEdge`/`isSanitizerEdge` in data-flow and taint-tracking
29+
configurations has been replaced with the option of specifying in- and
30+
out-barriers on nodes by overriding `isBarrierIn`/`isSanitizerIn` and
31+
`isBarrierOut`/`isSanitizerOut`. This should be simpler to use effectively,
32+
as it does not require knowledge about the actual edges used internally by
33+
the library.
34+
- The library now models data flow through `std::swap`.
35+
- Recursion through the `DataFlow` library is now always a compile error. Such recursion has been deprecated since release 1.16 in March 2018. If one `DataFlow::Configuration` needs to depend on the results of another, switch one of them to use one of the `DataFlow2` through `DataFlow4` libraries.
36+
- In the `semmle.code.cpp.dataflow.TaintTracking` library, the second copy of `Configuration` has been renamed from `TaintTracking::Configuration2` to `TaintTracking2::Configuration`, and the old name is now deprecated. Import `semmle.code.cpp.dataflow.TaintTracking2` to access the new name.
37+
- The `semmle.code.cpp.security.TaintTracking` library now considers a pointer difference calculation as blocking taint flow.
2538
- The predicate `Variable.getAnAssignedValue()` now reports assignments to fields resulting from aggregate initialization (` = {...}`).
2639
- The predicate `TypeMention.toString()` has been simplified to always return the string "`type mention`". This may improve performance when using `Element.toString()` or its descendants.
27-
- The `semmle.code.cpp.security.TaintTracking` library now considers a pointer difference calculation as blocking taint flow.
28-
- The second copy of the interprocedural `TaintTracking` library has been renamed from `TaintTracking::Configuration2` to `TaintTracking2::Configuration`, and the old name is now deprecated. Import `semmle.code.cpp.dataflow.TaintTracking2` to access the new name.
29-
- Fixed the `LocalScopeVariableReachability.qll` library's handling of loops with an entry condition is both always true upon first entry, and where there is more than one control flow path through the loop condition. This change increases the accuracy of the `LocalScopeVariableReachability.qll` library and queries which depend on it.
30-
- The `semmle.code.cpp.models` library now models data flow through `std::swap`.
40+
- Fixed the `LocalScopeVariableReachability.qll` library's handling of loops where the entry condition is always true on first entry, and where there is more than one control flow path through the loop condition. This change increases the accuracy of the `LocalScopeVariableReachability.qll` library and queries that depend on it.
3141
- There is a new `Variable.isThreadLocal()` predicate. It can be used to tell whether a variable is `thread_local`.
32-
- Recursion through the `DataFlow` library is now always a compile error. Such recursion has been deprecated since release 1.16. If one `DataFlow::Configuration` needs to depend on the results of another, switch one of them to use one of the `DataFlow2` through `DataFlow4` libraries.
42+
- C/C++ code examples have been added to QLDoc comments on many more classes in the QL libraries.

‎change-notes/1.22/analysis-csharp.md‎

Lines changed: 40 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,38 @@
11
# Improvements to C# analysis
22

3+
The following changes in version 1.22 affect C# analysis in all applications.
4+
35
## Changes to existing queries
46

57
| **Query** | **Expected impact** | **Change** |
68
|------------------------------|------------------------|-----------------------------------|
7-
| Added lines (`cs/vcs/added-lines-per-file`) | No results | Query has been removed. |
8-
| Churned lines (`cs/vcs/churn-per-file`) | No results | Query has been removed. |
99
| Constant condition (`cs/constant-condition`) | Fewer false positive results | Results have been removed for default cases (`_`) in switch expressions. |
10-
| Defect filter | No results | Query has been removed. |
11-
| Defect from SVN | No results | Query has been removed. |
12-
| Deleted lines (`cs/vcs/deleted-lines-per-file`) | No results | Query has been removed. |
1310
| Dispose may not be called if an exception is thrown during execution (`cs/dispose-not-called-on-throw`) | Fewer false positive results | Results have been removed where an object is disposed both by a `using` statement and a `Dispose` call. |
14-
| Files edited in pairs | No results | Query has been removed. |
15-
| Filter: only files recently edited | No results | Query has been removed. |
16-
| Large files currently edited | No results | Query has been removed. |
17-
| Metric from SVN | No results | Query has been removed. |
18-
| Number of authors (version control) (`cs/vcs/authors-per-file`) | No results | Query has been removed. |
19-
| Number of file-level changes (`cs/vcs/commits-per-file`) | No results | Query has been removed. |
20-
| Number of co-committed files (`cs/vcs/co-commits-per-file`) | No results | Query has been removed. |
21-
| Number of file re-commits (`cs/vcs/recommits-per-file`) | No results | Query has been removed. |
22-
| Number of recent file changes (`cs/vcs/recent-commits-per-file`) | No results | Query has been removed. |
23-
| Number of authors | No results | Query has been removed. |
24-
| Number of commits | No results | Query has been removed. |
25-
| Poorly documented files with many authors | No results | Query has been removed. |
26-
| Recent activity | No results | Query has been removed. |
2711
| Unchecked return value (`cs/unchecked-return-value`) | Fewer false positive results | Method calls that are expression bodies of `void` callables (for example, the call to `Foo` in `void Bar() => Foo()`) are no longer considered to use the return value. |
2812

13+
## Removal of old queries
14+
15+
The following historic queries are no longer available in the distribution:
16+
17+
* Added lines (`cs/vcs/added-lines-per-file`)
18+
* Churned lines (`cs/vcs/churn-per-file`)
19+
* Defect filter
20+
* Defect from SVN
21+
* Deleted lines (`cs/vcs/deleted-lines-per-file`)
22+
* Files edited in pairs
23+
* Filter: only files recently edited
24+
* Large files currently edited
25+
* Metric from SVN
26+
* Number of authors (version control) (`cs/vcs/authors-per-file`)
27+
* Number of file-level changes (`cs/vcs/commits-per-file`)
28+
* Number of co-committed files (`cs/vcs/co-commits-per-file`)
29+
* Number of file re-commits (`cs/vcs/recommits-per-file`)
30+
* Number of recent file changes (`cs/vcs/recent-commits-per-file`)
31+
* Number of authors
32+
* Number of commits
33+
* Poorly documented files with many authors
34+
* Recent activity
35+
2936
## Changes to code extraction
3037

3138
* The following C# 8 features are now extracted:
@@ -34,12 +41,18 @@
3441

3542
## Changes to QL libraries
3643

37-
* The new class `AnnotatedType` models types with type annotations, including nullability information, return kinds (`ref` and `readonly ref`), and parameter kinds (`in`, `out`, and `ref`)
38-
- The new predicate `Assignable.getAnnotatedType()` gets the annotated type of an assignable (such as a variable or a property)
39-
- The new predicates `Callable.getAnnotatedReturnType()` and `DelegateType.getAnnotatedReturnType()` get the annotated type of the return value
40-
- The new predicate `ArrayType.getAnnotatedElementType()` gets the annotated type of the array element
41-
- The new predicate `ConstructedGeneric.getAnnotatedTypeArgument()` gets the annotated type of a type argument
42-
- The new predicate `TypeParameterConstraints.getAnAnnotatedTypeConstraint()` gets a type constraint with type annotations
43-
* The new class `SuppressNullableWarningExpr` models suppress-nullable-warning expressions such as `x!`
44-
45-
## Changes to autobuilder
44+
* The new class `AnnotatedType` models types with type annotations, including nullability information, return kinds (`ref` and `readonly ref`), and parameter kinds (`in`, `out`, and `ref`).
45+
- The new predicate `Assignable.getAnnotatedType()` gets the annotated type of an assignable (such as a variable or a property).
46+
- The new predicates `Callable.getAnnotatedReturnType()` and `DelegateType.getAnnotatedReturnType()` gets the annotated type of the return value.
47+
- The new predicate `ArrayType.getAnnotatedElementType()` gets the annotated type of the array element.
48+
- The new predicate `ConstructedGeneric.getAnnotatedTypeArgument()` gets the annotated type of a type argument.
49+
- The new predicate `TypeParameterConstraints.getAnAnnotatedTypeConstraint()` gets a type constraint with type annotations.
50+
* The new class `SuppressNullableWarningExpr` models suppress-nullable-warning expressions such as `x!`.
51+
* The data-flow and taint-tracking libraries now support flow through fields. All existing configurations will have field-flow enabled by default, but it can be disabled by adding `override int fieldFlowBranchLimit() { result = 0 }` to the configuration class. Field assignments, `this.Foo = x`, object initializers, `new C() { Foo = x }`, and field initializers `int Foo = 0` are supported.
52+
* The possibility of specifying barrier edges using
53+
`isBarrierEdge`/`isSanitizerEdge` in data-flow and taint-tracking
54+
configurations has been replaced with the option of specifying in- and
55+
out-barriers on nodes by overriding `isBarrierIn`/`isSanitizerIn` and
56+
`isBarrierOut`/`isSanitizerOut`. This should be simpler to use effectively,
57+
as it does not require knowledge about the actual edges used internally by
58+
the library.

‎change-notes/1.22/analysis-java.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Improvements to Java analysis
22

3+
The following changes in version 1.22 affect Java analysis in all applications.
4+
35
## Changes to existing queries
46

57
| **Query** | **Expected impact** | **Change** |
@@ -13,7 +15,21 @@
1315

1416
* The virtual dispatch library has been updated to give more precise dispatch
1517
targets for `Object.toString()` calls. This affects all security queries and
16-
removes false positives that arose from paths through impossible `toString()`
18+
removes false positive results that arose from paths through impossible `toString()`
1719
calls.
1820
* The library `VCS.qll` and all queries that imported it have been removed.
19-
* The second copy of the interprocedural `TaintTracking` library has been renamed from `TaintTracking::Configuration2` to `TaintTracking2::Configuration`, and the old name is now deprecated. Import `semmle.code.java.dataflow.TaintTracking2` to access the new name.
21+
* The second copy of the interprocedural `TaintTracking` library has been
22+
renamed from `TaintTracking::Configuration2` to
23+
`TaintTracking2::Configuration`, and the old name is now deprecated. Import
24+
`semmle.code.java.dataflow.TaintTracking2` to access the new name.
25+
* The data-flow library now makes it easier to specify barriers/sanitizers
26+
arising from guards by overriding the predicate
27+
`isBarrierGuard`/`isSanitizerGuard` on data-flow and taint-tracking
28+
configurations respectively.
29+
* The possibility of specifying barrier edges using
30+
`isBarrierEdge`/`isSanitizerEdge` in data-flow and taint-tracking
31+
configurations has been replaced with the option of specifying in- and
32+
out-barriers on nodes by overriding `isBarrierIn`/`isSanitizerIn` and
33+
`isBarrierOut`/`isSanitizerOut`. This should be simpler to use effectively,
34+
as it does not require knowledge about the actual edges used internally by
35+
the library.

‎change-notes/1.22/analysis-javascript.md‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@
1616
* Support for tracking data flow and taint through getter functions (that is, functions that return a property of one of their arguments) and through the receiver object of method calls has been improved. This may produce more security alerts.
1717

1818
* Taint tracking through object property names has been made more precise, resulting in fewer false positive results.
19-
19+
20+
* Method calls are now resolved in more cases, due to improved class hierarchy analysis. This may produce more security alerts.
21+
22+
* Jump-to-definition now resolves calls to their definition in more cases, and supports jumping from a JSDoc type annotation to its definition.
23+
2024
## New queries
2125

2226
| **Query** | **Tags** | **Purpose** |
@@ -28,14 +32,19 @@
2832

2933
| **Query** | **Expected impact** | **Change** |
3034
|--------------------------------|------------------------------|---------------------------------------------------------------------------|
31-
| Shift out of range | Fewer false positive results | This rule now correctly handles BigInt shift operands. |
32-
| Conflicting HTML element attributes | Fewer results | Results are no longer shown on LGTM by default. |
33-
| Superfluous trailing arguments | Fewer false-positive results. | This rule no longer flags calls to placeholder functions that trivially throw an exception. |
35+
| Conflicting HTML element attributes (`js/conflicting-html-attribute`) | No changes to results | Results are no longer shown on LGTM by default. |
36+
| Shift out of range (`js/shift-out-of-range`| Fewer false positive results | This rule now correctly handles BigInt shift operands. |
37+
| Superfluous trailing arguments (`js/superfluous-trailing-arguments`) | Fewer false-positive results. | This rule no longer flags calls to placeholder functions that trivially throw an exception. |
38+
| Undocumented parameter (`js/jsdoc/missing-parameter`) | No changes to results | This rule is now run on LGTM, although its results are still not shown by default. |
3439

3540
## Changes to QL libraries
3641

37-
- The `getName()` predicate on functions and classes now gets a name
42+
- The `getName()` predicate on functions and classes now gets a name that is
3843
inferred from the context if the function or class was not declared with a name.
3944
- The two-argument and three-argument variants of `DataFlow::Configuration::isBarrier` and
4045
`TaintTracking::Configuration::isSanitizer` have been deprecated. Overriding them no
4146
longer has any effect. Use `isBarrierEdge` and `isSanitizerEdge` instead.
47+
- The QLDoc for most AST classes have been expanded with concrete syntax examples.
48+
- Tutorials on how to use [flow labels](https://help.semmle.com/QL/learn-ql/javascript/flow-labels.html)
49+
and [type tracking](https://help.semmle.com/QL/learn-ql/javascript/type-tracking.html) have been published,
50+
as well as a [data flow cheat sheet](https://help.semmle.com/QL/learn-ql/javascript/dataflow-cheat-sheet.html) for quick reference.

‎change-notes/1.22/analysis-python.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,29 @@
66
### Points-to
77
Tracking of "unknown" values from modules that are absent from the database has been improved. Particularly when an "unknown" value is used as a decorator, the decorated function is tracked.
88

9+
### Loop unrolling
10+
The extractor now unrolls a single iteration of loops that are known to run at least once. This improves analysis in cases like the following
911

10-
### Impact on existing queries.
12+
```python
13+
if seq:
14+
for x in seq:
15+
y = x
16+
y # y is defined here
17+
```
1118

19+
### Better API for function parameter annotations
20+
Instances of the `Parameter` and `ParameterDefinition` class now have a `getAnnotation` method that returns the corresponding parameter annotation, if one exists.
21+
22+
### Improvements to the Value API
23+
- The Value API has been extended with classes representing functions, classes, tuples, and other types.
24+
25+
- `Value::forInt(int x)` and `Value::forString(string s)` have been added to make it easier to refer to the `Value` entities for common constants.
26+
27+
### Other improvements
28+
29+
- Short flags for regexes (for example, `re.M` for multiline regexes) are now handled correctly.
30+
- Modules with multiple import roots no longer get multiple names.
31+
- A new `NegativeIntegerLiteral` class has been added as a subtype of `ImmutableLiteral`, so that `-1` is treated as an `ImmutableLiteral`. This means that queries looking for the use of constant integers will automatically handle negative numbers.
1232

1333
## New queries
1434

0 commit comments

Comments
 (0)