Merge remote-tracking branch 'upstream/master' into infinite-loops-visible

Moved the change note to 1.23.

Author: jbj

.codeqlmanifest.json

@@ -0,0 +1,3 @@
+{ "provide": [ "*/ql/src/qlpack.yml",
+               "misc/legacy-support/*/qlpack.yml",
+               "misc/suite-helpers/qlpack.yml" ] }

.github/ISSUE_TEMPLATE/lgtm-com---false-positive.md

@@ -0,0 +1,24 @@
+---
+name: LGTM.com - false positive
+about: Tell us about an alert that shouldn't be reported
+title: LGTM.com - false positive
+labels: false-positive
+assignees: ''
+
+---
+
+**Description of the false positive**
+
+<!-- Please explain briefly why you think it shouldn't be included. -->
+
+**URL to the alert on the project page on LGTM.com**
+
+<!--
+1. Open the project on LGTM.com.
+For example, https://lgtm.com/projects/g/pallets/click/.
+2. Switch to the `Alerts` tab. For example, https://lgtm.com/projects/g/pallets/click/alerts/.
+3. Scroll to the alert that you would like to report.
+4. Click on the right most icon `View this alert within the complete file`.
+5. A new browser tab opens. Copy and paste the page URL here.
+For example, https://lgtm.com/projects/g/pallets/click/snapshot/719fb7d8322b0767cdd1e5903ba3eb3233ba8dd5/files/click/_winconsole.py#xa08d213ab3289f87:1.
+-->

.github/ISSUE_TEMPLATE/ql---general.md

@@ -0,0 +1,14 @@
+---
+name: General issue
+about: Tell us if you think something is wrong or if you have a question
+title: General issue
+labels: question
+assignees: ''
+
+---
+
+**Description of the issue**
+
+<!-- Please explain briefly what is the problem.
+If it is about an LGTM project, please include its URL.-->
+

.gitignore

@@ -12,3 +12,6 @@
 # Visual studio temporaries, except a file used by QL4VS
 .vs/*
 !.vs/VSWorkspaceSettings.json
+
+# It's useful (though not required) to be able to unpack codeql in the ql checkout itself
+/codeql/

CODEOWNERS

@@ -2,4 +2,9 @@
 /java/ @Semmle/java
 /javascript/ @Semmle/js
 /cpp/ @Semmle/cpp-analysis
-/cpp/**/*.qhelp @semmledocs-ac
+/cpp/**/*.qhelp @hubwriter
+/csharp/**/*.qhelp @jf205
+/java/**/*.qhelp @felicitymay
+/javascript/**/*.qhelp @mchammer01
+/python/**/*.qhelp @felicitymay
+/docs/language/ @shati-patel @jf205

CODE_OF_CONDUCT.md

@@ -14,7 +14,7 @@ Our community strives to:
   * Posting, or threatening to post, people’s personally identifying information (“doxing”).
   * Insults, especially those using discriminatory terms or slurs.
   * Behavior that could be perceived as sexual attention.
-* Advocating for or encouraging any of the above behaviors.
+  * Advocating for or encouraging any of the above behaviors.
 * Understand disagreements: Disagreements, both social and technical, are useful learning opportunities. Seek to understand others’ viewpoints and resolve differences constructively.
 
 This code is not exhaustive or complete. It serves to capture our common understanding of a productive, collaborative environment. We expect the code to be followed in spirit as much as in the letter.

CONTRIBUTING.md

@@ -1,10 +1,50 @@
 # Contributing to QL
 
-We welcome contributions to our standard library and standard checks, got an idea for a new check, or how to improve an existing query? Then please go ahead an open a Pull Request!
+We welcome contributions to our standard library and standard checks. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request!
 
-Before we accept your pull request, we will require that you have agreed to our Contributor License Agreement, this is not something that you need to do before you submit your pull request, but until you've done so, we will be unable to accept your contribution.
+Before we accept your pull request, we require that you have agreed to our Contributor License Agreement, this is not something that you need to do before you submit your pull request, but until you've done so, we will be unable to accept your contribution.
 
-Please read our [QL Style Guide](docs/ql-style-guide.md) for information on how to format QL code in this repository.
+## Adding a new query
+
+If you have an idea for a query that you would like to share with other Semmle users, please open a pull request to add it to this repository. 
+Follow the steps below to help other users understand what your query does, and to ensure that your query is consistent with the other Semmle queries.
+
+1. **Consult the QL documentation for query writers**
+
+   There is lots of useful documentation to help you write QL, ranging from information about query file structure to language-specific tutorials. For more information on the documentation available, see [Writing QL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
+
+2. **Format your QL correctly**
+
+   All of Semmle's standard QL queries and libraries are uniformly formatted for clarity and consistency, so we strongly recommend that all QL contributions follow the same formatting guidelines. If you use QL for Eclipse, you can auto-format your query in the [QL editor](https://help.semmle.com/ql-for-eclipse/Content/WebHelp/ql-editor.html). For more information, see the [QL style guide](https://github.com/Semmle/ql/blob/master/docs/ql-style-guide.md).
+
+3. **Make sure your query has the correct metadata**
+
+   Query metadata is used by Semmle's analysis to identify your query and make sure the query results are displayed properly. 
+   The most important metadata to include are the `@name`, `@description`, and the `@kind`.
+   Other metadata properties (`@precision`, `@severity`, and `@tags`) are usually added after the query has been reviewed by Semmle staff.
+   For more information on writing query metadata, see the [Query metadata style guide](https://github.com/Semmle/ql/blob/master/docs/query-metadata-style-guide.md).
+
+4. **Make sure the `select` statement is compatible with the query type**
+
+   The `select` statement of your query must be compatible with the query type (determined by the `@kind` metadata property) for alert or path results to be displayed correctly in LGTM and QL for Eclipse.
+   For more information on `select` statement format, see [Introduction to query files](https://help.semmle.com/QL/learn-ql/writing-queries/introduction-to-queries.html#select-clause) on help.semmle.com.
+
+5. **Save your query in a `.ql` file in correct language directory in this repository**
+
+   There are five language-specific directories in this repository:
+   
+     * C/C++: `ql/cpp/ql/src`
+     * C#: `ql/csharp/ql/src`
+     * Java: `ql/java/ql/src`
+     * JavaScript: `ql/javascript/ql/src`
+     * Python: `ql/python/ql/src`
+
+   Each language-specific directory contains further subdirectories that group queries based on their `@tags` properties or purpose. Select the appropriate subdirectory for your new query, or create a new one if necessary. 
+
+6. **Write a query help file**
+
+   Query help files explain the purpose of your query to other users. Write your query help in a `.qhelp` file and save it in the same directory as your new query. 
+   For more information on writing query help, see the [Query help style guide](https://github.com/Semmle/ql/blob/master/docs/query-help-style-guide.md).
 
 ## Using your personal data
 

change-notes/1.22/analysis-cpp.md

@@ -1,25 +1,42 @@
 # Improvements to C/C++ analysis
 
-## General improvements
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
+The following changes in version 1.22 affect C/C++ analysis in all applications.
 
 ## Changes to existing queries
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
-| Expression has no effect (`cpp/useless-expression`) | Fewer false positive results | Calls to functions with the `weak` attribute are no longer considered to be side effect free, because they could be overridden with a different implementation at link time. |
-| No space for zero terminator (`cpp/no-space-for-terminator`) | Fewer false positive results | False positives involving strings that are not null-terminated have been excluded. |
+| Call to alloca in a loop (`cpp/alloca-in-loop`) | Fewer false positive results | The query no longer highlights code where the stack allocation could not be reached multiple times in the loop, typically due to a `break` or `return` statement. |
+| Continue statement that does not continue (`cpp/continue-in-false-loop`) | Fewer false positive results | Analysis is now restricted to `do`-`while` loops. This query is now run and displayed by default on LGTM. |
+| Expression has no effect (`cpp/useless-expression`) | Fewer false positive results | Calls to functions with the `weak` attribute are no longer considered to be side-effect free, because they could be overridden with a different implementation at link time. |
+| No space for zero terminator (`cpp/no-space-for-terminator`) | Fewer false positive results | False positive results for strings that are not null-terminated have been excluded. |
+| Non-constant format string (`cpp/non-constant-format`) | Fewer false positive results | The query was rewritten using the taint-tracking library. |
+| Sign check of bitwise operation (`cpp/bitwise-sign-check`) | Fewer false positive and more true positive results | The query now understands the direction of each comparison, making it more accurate. |
 | Suspicious pointer scaling (`cpp/suspicious-pointer-scaling`) | Lower precision | The precision of this query has been reduced to "medium". This coding pattern is used intentionally and safely in a number of real-world projects. Results are no longer displayed on LGTM unless you choose to display them. |
-| Non-constant format string (`cpp/non-constant-format`) | Fewer false positive results | Rewritten using the taint-tracking library. |
-| Comparison of narrow type with wide type in loop condition (`cpp/comparison-with-wider-type`) | Higher precision | The precision of this query has been increased to "high" as the alerts from this query have proved to be valuable on real-world projects. With this precision, results are now displayed by default in LGTM. |
+| Variable used in its own initializer (`cpp/use-in-own-initializer`) | Fewer false positive results | False positive results for constant variables with the same name in different namespaces have been removed. |
 
 ## Changes to QL libraries
 
+- The data flow library (`semmle.code.cpp.dataflow.DataFlow`) has had the
+  following improvements, all of which benefit the taint tracking library
+  (`semmle.code.cpp.dataflow.TaintTracking`) as well.
+  - This release includes preliminary support for interprocedural flow through
+    fields (non-static data members). In some cases, data stored in a field in
+    one function can now flow to a read of the same field in a different
+    function.
+  - The possibility of specifying barrier edges using
+    `isBarrierEdge`/`isSanitizerEdge` in data-flow and taint-tracking
+    configurations has been replaced with the option of specifying in- and
+    out-barriers on nodes by overriding `isBarrierIn`/`isSanitizerIn` and
+    `isBarrierOut`/`isSanitizerOut`. This should be simpler to use effectively,
+    as it does not require knowledge about the actual edges used internally by
+    the library.
+  - The library now models data flow through `std::swap`.
+  - Recursion through the `DataFlow` library is now always a compile error. Such recursion has been deprecated since release 1.16 in March 2018. If one `DataFlow::Configuration` needs to depend on the results of another, switch one of them to use one of the `DataFlow2` through `DataFlow4` libraries.
+- In the `semmle.code.cpp.dataflow.TaintTracking` library, the second copy of `Configuration` has been renamed from `TaintTracking::Configuration2` to `TaintTracking2::Configuration`, and the old name is now deprecated. Import `semmle.code.cpp.dataflow.TaintTracking2` to access the new name.
+- The `semmle.code.cpp.security.TaintTracking` library now considers a pointer difference calculation as blocking taint flow.
 - The predicate `Variable.getAnAssignedValue()` now reports assignments to fields resulting from aggregate initialization (` = {...}`).
 - The predicate `TypeMention.toString()` has been simplified to always return the string "`type mention`".  This may improve performance when using `Element.toString()` or its descendants.
-- The `semmle.code.cpp.security.TaintTracking` library now considers a pointer difference calculation as blocking taint flow.
-- Fixed the `LocalScopeVariableReachability.qll` library's handling of loops with an entry condition is both always true upon first entry, and where there is more than one control flow path through the loop condition.  This change increases the accuracy of the `LocalScopeVariableReachability.qll` library and queries which depend on it.
+- Fixed the `LocalScopeVariableReachability.qll` library's handling of loops where the entry condition is always true on first entry, and where there is more than one control flow path through the loop condition. This change increases the accuracy of the `LocalScopeVariableReachability.qll` library and queries that depend on it.
+- There is a new `Variable.isThreadLocal()` predicate. It can be used to tell whether a variable is `thread_local`.
+- C/C++ code examples have been added to QLDoc comments on many more classes in the QL libraries.

change-notes/1.22/analysis-csharp.md

@@ -1,29 +1,37 @@
 # Improvements to C# analysis
 
+The following changes in version 1.22 affect C# analysis in all applications.
+
 ## Changes to existing queries
 
 | **Query**                    | **Expected impact**    | **Change**                        |
 |------------------------------|------------------------|-----------------------------------|
-| Added lines (`cs/vcs/added-lines-per-file`) | No results | Query has been removed. |
-| Churned lines (`cs/vcs/churn-per-file`) | No results | Query has been removed. |
 | Constant condition (`cs/constant-condition`) | Fewer false positive results | Results have been removed for default cases (`_`) in switch expressions. |
-| Defect filter | No results | Query has been removed. |
-| Defect from SVN | No results | Query has been removed. |
-| Deleted lines (`cs/vcs/deleted-lines-per-file`) | No results | Query has been removed. |
 | Dispose may not be called if an exception is thrown during execution (`cs/dispose-not-called-on-throw`) | Fewer false positive results | Results have been removed where an object is disposed both by a `using` statement and a `Dispose` call. |
-| Files edited in pairs | No results | Query has been removed. |
-| Filter: only files recently edited | No results | Query has been removed. |
-| Large files currently edited | No results | Query has been removed. |
-| Metric from SVN | No results | Query has been removed. |
-| Number of authors (version control) (`cs/vcs/authors-per-file`) | No results | Query has been removed. |
-| Number of file-level changes (`cs/vcs/commits-per-file`) | No results | Query has been removed. |
-| Number of co-committed files (`cs/vcs/co-commits-per-file`) | No results | Query has been removed. |
-| Number of file re-commits (`cs/vcs/recommits-per-file`) | No results | Query has been removed. |
-| Number of recent file changes (`cs/vcs/recent-commits-per-file`) | No results | Query has been removed. |
-| Number of authors | No results | Query has been removed. |
-| Number of commits | No results | Query has been removed. |
-| Poorly documented files with many authors | No results | Query has been removed. |
-| Recent activity | No results | Query has been removed. |
+| Unchecked return value (`cs/unchecked-return-value`) | Fewer false positive results | Method calls that are expression bodies of `void` callables (for example, the call to `Foo` in `void Bar() => Foo()`) are no longer considered to use the return value. |
+
+## Removal of old queries
+
+The following historic queries are no longer available in the distribution:
+
+* Added lines (`cs/vcs/added-lines-per-file`)
+* Churned lines (`cs/vcs/churn-per-file`)
+* Defect filter
+* Defect from SVN
+* Deleted lines (`cs/vcs/deleted-lines-per-file`)
+* Files edited in pairs
+* Filter: only files recently edited
+* Large files currently edited
+* Metric from SVN
+* Number of authors (version control) (`cs/vcs/authors-per-file`)
+* Number of file-level changes (`cs/vcs/commits-per-file`)
+* Number of co-committed files (`cs/vcs/co-commits-per-file`)
+* Number of file re-commits (`cs/vcs/recommits-per-file`)
+* Number of recent file changes (`cs/vcs/recent-commits-per-file`)
+* Number of authors
+* Number of commits
+* Poorly documented files with many authors
+* Recent activity
 
 ## Changes to code extraction
 
@@ -33,12 +41,18 @@
 
 ## Changes to QL libraries
 
-* The new class `AnnotatedType` models types with type annotations, including nullability information, return kinds (`ref` and `readonly ref`), and parameter kinds (`in`, `out`, and `ref`)
-  - The new predicate `Assignable.getAnnotatedType()` gets the annotated type of an assignable (such as a variable or a property)
-  - The new predicates `Callable.getAnnotatedReturnType()` and `DelegateType.getAnnotatedReturnType()` get the annotated type of the return value
-  - The new predicate `ArrayType.getAnnotatedElementType()` gets the annotated type of the array element
-  - The new predicate `ConstructedGeneric.getAnnotatedTypeArgument()` gets the annotated type of a type argument
-  - The new predicate `TypeParameterConstraints.getAnAnnotatedTypeConstraint()` gets a type constraint with type annotations
-* The new class `SuppressNullableWarningExpr` models suppress-nullable-warning expressions such as `x!`
-
-## Changes to autobuilder
+* The new class `AnnotatedType` models types with type annotations, including nullability information, return kinds (`ref` and `readonly ref`), and parameter kinds (`in`, `out`, and `ref`).
+  - The new predicate `Assignable.getAnnotatedType()` gets the annotated type of an assignable (such as a variable or a property).
+  - The new predicates `Callable.getAnnotatedReturnType()` and `DelegateType.getAnnotatedReturnType()` gets the annotated type of the return value.
+  - The new predicate `ArrayType.getAnnotatedElementType()` gets the annotated type of the array element.
+  - The new predicate `ConstructedGeneric.getAnnotatedTypeArgument()` gets the annotated type of a type argument.
+  - The new predicate `TypeParameterConstraints.getAnAnnotatedTypeConstraint()` gets a type constraint with type annotations.
+* The new class `SuppressNullableWarningExpr` models suppress-nullable-warning expressions such as `x!`.
+* The data-flow and taint-tracking libraries now support flow through fields. All existing configurations will have field-flow enabled by default, but it can be disabled by adding `override int fieldFlowBranchLimit() { result = 0 }` to the configuration class. Field assignments, `this.Foo = x`, object initializers, `new C() { Foo = x }`, and field initializers `int Foo = 0` are supported.
+* The possibility of specifying barrier edges using
+  `isBarrierEdge`/`isSanitizerEdge` in data-flow and taint-tracking
+  configurations has been replaced with the option of specifying in- and
+  out-barriers on nodes by overriding `isBarrierIn`/`isSanitizerIn` and
+  `isBarrierOut`/`isSanitizerOut`. This should be simpler to use effectively,
+  as it does not require knowledge about the actual edges used internally by
+  the library.