Merge branch 'master' into python-unsafe-use-of-mktemp

Author: taus-semmle

README.md

@@ -4,8 +4,8 @@ This open source repository contains the standard QL libraries and queries that
 
 ## How do I learn QL and run queries?
 
-LGTM has [extensive documentation](https://lgtm.com/help/ql/introduction-to-ql) on getting started with writing QL.
-You can use the [interactive query console](https://lgtm.com/help/lgtm/using-query-console) or the [QL for Eclipse](https://lgtm.com/help/lgtm/running-queries-ide) plugin to try out your queries on any open-source project that's currently being analyzed.
+There is [extensive documentation](https://help.semmle.com/QL/learn-ql/) on getting started with writing QL.
+You can use the [interactive query console](https://lgtm.com/help/lgtm/using-query-console) on LGTM.com or the [QL for Eclipse](https://lgtm.com/help/lgtm/running-queries-ide) plugin to try out your queries on any open-source project that's currently being analyzed.
 
 ## Contributing
 

change-notes/1.19/support/framework-support.rst

@@ -0,0 +1,61 @@
+Frameworks and libraries
+########################
+
+The QL libraries and queries in this version have been explicitly checked against the libraries and frameworks listed below.
+
+.. pull-quote::
+
+    Tip
+    
+    If you're interested in other libraries or frameworks, you can extend the analysis to cover them. 
+    For example, by extending the data flow libraries to include data sources and sinks for additional libraries or frameworks.
+
+.. There is currently no built-in support for libraries or frameworks for C/C++.
+
+C# built-in support
+================================
+
+* ASP.Net MVC framework
+* ASP.NET Web API
+* ASP.NET Web Forms
+* ASP.NET Core
+* ASP.NET Core MVC
+* ASP.Net Core Razor
+* Razor templates
+
+
+COBOL built-in support
+===================================
+
+* Embedded SQL
+* Embedded CICS
+
+
+Java built-in support
+==================================
+
+.. csv-table:: 
+     :file: java-frameworks.csv
+     :header-rows: 1
+     :class: fullWidthTable
+     :widths: auto
+
+
+JavaScript and TypeScript built-in support
+=======================================================
+
+.. csv-table:: 
+     :file: javascript-typescript-frameworks.csv
+     :header-rows: 1
+     :class: fullWidthTable
+     :widths: auto
+
+
+Python built-in support
+====================================
+
+.. csv-table:: 
+     :file: python-frameworks.csv
+     :header-rows: 1
+     :class: fullWidthTable
+     :widths: auto

change-notes/1.19/support/java-frameworks.csv

@@ -0,0 +1,10 @@
+Name, Category
+Hibernate, Database
+iBatis / MyBatis, Database
+Java Persistence API (JPA), Database
+JDBC, Database
+Kryo deserialization, Serialization
+SnakeYaml, Serialization
+Spring JDBC, Database
+Spring MVC, Web application framework
+XStream, Serialization

change-notes/1.19/support/javascript-typescript-frameworks.csv

@@ -0,0 +1,22 @@
+Name, Category
+angularjs, HTML framework
+axios, Network communicator
+browser, Runtime environment
+electron, Runtime environment
+express, Server
+hapi, Server
+jquery, Utility library
+koa, Server
+lodash, Utility library
+mongodb, Database
+mssql, Database
+mysql, Database
+node, Runtime environment
+postgres, Database
+ramda, Utility library
+react, HTML framework
+request, Network communicator
+sequelize, Database
+sqlite3, Database
+superagent, Network communicator
+underscore, Utility library

change-notes/1.19/support/language-support.rst

@@ -0,0 +1,19 @@
+Languages and compilers
+#######################
+
+QL and LGTM version |version| support analysis of the following languages compiled by the following compilers.
+
+Note that where there are several versions or dialects of a language, the supported variants are listed.
+
+.. csv-table::
+     :file: versions-compilers.csv
+     :header-rows: 1
+     :widths: auto
+     :stub-columns: 1
+
+.. container:: footnote-group
+
+    .. [1] The best results are achieved with COBOL code that stays close to the ANSI 85 standard.  
+    .. [2] Java 11 refers to the language features used. Builds that execute on Java 6 or higher can be analyzed.
+    .. [3] JSX and Flow code, YAML, JSON, and HTML files may also be analyzed with JavaScript files. 
+    .. [4] TypeScript analysis is performed by running the JavaScript extractor with TypeScript enabled. This is the default for LGTM.   

change-notes/1.19/support/python-frameworks.csv

@@ -0,0 +1,7 @@
+Name, Category
+Django, Web application framework
+Flask, Microframework
+Pyramid, Web application framework
+Tornado, Web application framework and asynchronous networking library
+Twisted, Networking engine
+WebOb, WSGI request library

change-notes/1.19/support/versions-compilers.csv

@@ -0,0 +1,16 @@
+Language,Variants,Compilers,Extensions
+C/C++,"C89, C99, C11, C++98, C++03, C++11, C++14, C++17","Clang extensions (up to Clang 6.0)
+
+GNU extensions (up to GCC 7.3), 
+
+Microsoft extensions (up to VS 2017)","``.cpp``, ``.c++``, ``.cxx``, ``.hpp``, ``.hh``, ``.h++``, ``.hxx``, ``.c``, ``.cc``, ``.h``"
+C#,C# up to 7.2 together with .NET versions up to 4.7.1,"Microsoft Visual Studio up to 2017, 
+
+.NET Core up to 2.1","``.sln``, ``.csproj``, ``.cs``, ``.cshtml``, ``.xaml``"
+COBOL,ANSI 85 or newer [1]_.,Not applicable,"``.cbl``, ``.CBL``, ``.cpy``, ``.CPY``, ``.copy``, ``.COPY``"
+Java,"Java 11 [2]_. or lower","javac (OpenJDK and Oracle JDK)
+
+Eclipse compiler for Java (ECJ) batch compiler",``.java``
+JavaScript,ECMAScript 2018 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhm``, ``.xhtml``, ``.vue``, ``.json`` [3]_."
+Python,"2.7, 3.5, 3.6, 3.7",Not applicable,``.py``
+TypeScript [4]_.,"2.6, 2.7, 2.8, 2.9, 3.0, 3.1",Standard TypeScript compiler,"``.ts``, ``.tsx``"

change-notes/1.20/analysis-cpp.md

@@ -11,19 +11,29 @@
 | Use of string copy function in a condition (`cpp/string-copy-return-value-as-boolean`) | correctness | This query identifies calls to string copy functions used in conditions, where it's likely that a different function was intended to be called. |
 | Lossy function result cast (`cpp/lossy-function-result-cast`) | correctness | Finds function calls whose result type is a floating point type, which are implicitly cast to an integral type.  Newly available but not displayed by default on LGTM. |
 | Array argument size mismatch (`cpp/array-arg-size-mismatch`) | reliability | Finds function calls where the size of an array being passed is smaller than the array size of the declared parameter.  Newly displayed on LGTM. |
+| Returning stack-allocated memory (`cpp/return-stack-allocated-memory`) | reliability, external/cwe/cwe-825 | Finds functions that may return a pointer or reference to stack-allocated memory. This query existed already but has been rewritten from scratch to make the error rate low enough for use on LGTM. Displayed by default. |
 
 ## Changes to existing queries
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
 | Array argument size mismatch (`cpp/array-arg-size-mismatch`) | Fewer false positives | An exception has been added to this query for variable sized arrays. |
+| Call to memory access function may overflow buffer (`cpp/overflow-buffer`) | More correct results | This query now recognizes calls to `RtlCopyMemoryNonTemporal` and `RtlSecureZeroMemory`. |
+| Returning stack-allocated memory (`cpp/return-stack-allocated-memory`) | More correct results | Many more stack allocated expressions are now recognized. |
 | Suspicious add with sizeof (`cpp/suspicious-add-sizeof`) | Fewer false positives | Pointer arithmetic on `char * const` expressions (and other variations of `char *`) are now correctly excluded from the results. |
 | Suspicious pointer scaling (`cpp/suspicious-pointer-scaling`) | Fewer false positives | False positives involving types that are not uniquely named in the snapshot have been fixed. |
 | Call to memory access function may overflow buffer (`cpp/overflow-buffer`) | More correct results | Calls to `fread` are now examined by this query. |
 | Lossy function result cast (`cpp/lossy-function-result-cast`) | Fewer false positive results | The whitelist of rounding functions built into this query has been expanded. |
 | Unused static variable (`cpp/unused-static-variable`) | Fewer false positive results | Variables with the attribute `unused` are now excluded from the query. |
 | Resource not released in destructor (`cpp/resource-not-released-in-destructor`) | Fewer false positive results | Fix false positives where a resource is released via a virtual method call, function pointer, or lambda. |
+| 'new[]' array freed with 'delete' (`cpp/new-array-delete-mismatch`) | More correct results | Data flow through global variables for this query has been improved. |
+| 'new' object freed with 'delete[]' (`cpp/new-delete-array-mismatch`) | More correct results | Data flow through global variables for this query has been improved. |
+| Mismatching new/free or malloc/delete (`cpp/new-free-mismatch`) | More correct results | Data flow through global variables for this query has been improved. |
+| Use of inherently dangerous function (`cpp/potential-buffer-overflow`) | Cleaned up | This query no longer catches uses of `gets`, and has been renamed 'Potential buffer overflow'. |
+| Use of potentially dangerous function (`cpp/potentially-dangerous-function`) | More correct results | This query now catches uses of `gets`. |
 
 ## Changes to QL libraries
 
-There is a new `Namespace.isInline()` predicate, which holds if the namespace was declared as `inline namespace`.
+* There is a new `Namespace.isInline()` predicate, which holds if the namespace was declared as `inline namespace`.
+* The `Expr.isConstant()` predicate now also holds for _address constant expressions_, which are addresses that will be constant after the program has been linked. These address constants do not have a result for `Expr.getValue()`.
+* There are new `Function.isDeclaredConstexpr()` and `Function.isConstexpr()` predicates. They can be used to tell whether a function was declared as `constexpr`, and whether it actually is `constexpr`.

change-notes/1.20/analysis-csharp.md

@@ -11,11 +11,15 @@
 
 | *@name of query (Query ID)*  | *Impact on results*    | *How/why the query has changed*   |
 |------------------------------|------------------------|-----------------------------------|
-| Off-by-one comparison against container length (cs/index-out-of-bounds) | Fewer false positives | Results have been removed when there are additional guards on the index. |
-| Dereferenced variable is always null (cs/dereferenced-value-is-always-null) | Improved results | The query has been rewritten from scratch, and the analysis is now based on static single assignment (SSA) forms. The query is now enabled by default in LGTM. |
-| Dereferenced variable may be null (cs/dereferenced-value-may-be-null) | Improved results | The query has been rewritten from scratch, and the analysis is now based on static single assignment (SSA) forms. The query is now enabled by default in LGTM. |
-| SQL query built from user-controlled sources (cs/sql-injection), Improper control of generation of code (cs/code-injection), Uncontrolled format string (cs/uncontrolled-format-string), Clear text storage of sensitive information (cs/cleartext-storage-of-sensitive-information), Exposure of private information (cs/exposure-of-sensitive-information) | More results | Data sources have been added from user controls in `System.Windows.Forms`. |
-| Use of default ToString() (cs/call-to-object-tostring) | Fewer false positives | Results have been removed for `char` arrays passed to `StringBuilder.Append()`, which were incorrectly marked as using `ToString`. |
+| Off-by-one comparison against container length (`cs/index-out-of-bounds`) | Fewer false positives | Results have been removed when there are additional guards on the index. |
+| Dereferenced variable is always null (`cs/dereferenced-value-is-always-null`) | Improved results | The query has been rewritten from scratch, and the analysis is now based on static single assignment (SSA) forms. The query is now enabled by default in LGTM. |
+| Dereferenced variable may be null (`cs/dereferenced-value-may-be-null`) | Improved results | The query has been rewritten from scratch, and the analysis is now based on static single assignment (SSA) forms. The query is now enabled by default in LGTM. |
+| SQL query built from user-controlled sources (`cs/sql-injection`), Improper control of generation of code (`cs/code-injection`), Uncontrolled format string (`cs/uncontrolled-format-string`), Clear text storage of sensitive information (`cs/cleartext-storage-of-sensitive-information`), Exposure of private information (`cs/exposure-of-sensitive-information`) | More results | Data sources have been added from user controls in `System.Windows.Forms`. |
+| Use of default ToString() (`cs/call-to-object-tostring`) | Fewer false positives | Results have been removed for `char` arrays passed to `StringBuilder.Append()`, which were incorrectly marked as using `ToString`. |
+| Use of default ToString() (`cs/call-to-object-tostring`) | Fewer results | Results have been removed when the object is an interface or an abstract class. |
+| Unused format argument (`cs/format-argument-unused`) | Fewer false positives | Results have been removed where the format string is empty. This is often used as a default value and is not an interesting result. |
+| Double-checked lock is not thread-safe (`cs/unsafe-double-checked-lock`) | Fewer false positives, more true positives | Results have been removed where the underlying field was not updated in the `lock` statement, or where the field is a `struct`. Results have been added where there are other statements inside the `lock` statement. |
+| Using a package with a known vulnerability (`cs/use-of-vulnerable-package`) | More results | This query detects packages vulnerable to CVE-2019-0657. |
 
 ## Changes to code extraction
 
@@ -24,6 +28,7 @@
 
 ## Changes to QL libraries
 
-* The class `AccessorCall` (and subclasses `PropertyCall`, `IndexerCall`, and `EventCall`) have been redefined, so the expressions they represent are not necessarily the accesses themselves, but rather the expressions that give rise to the accessor calls. For example, in the property assignment `x.Prop = 0`, the call to the setter for `Prop` is no longer represented by the access `x.Prop`, but instead the whole assignment. Consequently, it is no longer safe to cast directly between `AccessorCall`s and `Access`es, and the predicate `AccessorCall::getAccess()` should be used instead.
+* The class `TrivialProperty` now includes library properties determined to be trivial using CIL analysis. This may increase the number of results for all queries that use data flow.
+* Taint-tracking steps have been added for the `Json.NET` package. This will improve results for queries that use taint-tracking.
 
 ## Changes to the autobuilder

change-notes/1.20/analysis-java.md

@@ -14,6 +14,7 @@
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
+| Arbitrary file write during archive extraction ("Zip Slip") (`java/zipslip`) | Fewer false positive results | Results involving a sanitization step that converts a destination `Path` to a `File` are no longer reported. |
 | Double-checked locking is not thread-safe (`java/unsafe-double-checked-locking`) | Fewer false positive results and more true positive results | Results that use safe publication through a `final` field are no longer reported. Results that initialize immutable types like `String` incorrectly are now reported. |
 | Result of multiplication cast to wider type (`java/integer-multiplication-cast-to-long`) | Fewer results | Results involving conversions to `float` or `double` are no longer reported, as they were almost exclusively false positives. |
 
@@ -24,5 +25,12 @@
   `semmle.code.java.dataflow.DataFlow`,
   `semmle.code.java.dataflow.TaintTracking`, and
   `semmle.code.java.dataflow.FlowSources` since 1.16.
+* Taint tracking now includes additional default data-flow steps through
+  collections, maps, and iterators. This affects all security queries, which
+  can report more results based on such paths.
+* The `FlowSources` and `TaintTracking` libraries are extended to cover additional remote user
+  input and taint steps from the Apache Thrift, Apache Struts, Guice and Protobuf frameworks.
+  This affects all security queries, which may yield additional results on projects
+  that use these frameworks.