Merge pull request #1046 from xiemaisi/rc/1.20

Merge rc/1.20 into master

Author: jbj

change-notes/1.20/analysis-cpp.md

@@ -33,9 +33,14 @@
 | Mismatching new/free or malloc/delete (`cpp/new-free-mismatch`) | More correct results | Data flow through global variables for this query has been improved. |
 | Use of inherently dangerous function (`cpp/potential-buffer-overflow`) | Cleaned up | This query no longer catches uses of `gets`, and has been renamed 'Potential buffer overflow'. |
 | Use of potentially dangerous function (`cpp/potentially-dangerous-function`) | More correct results | This query now catches uses of `gets`. |
+| Potential buffer overflow (`cpp/potential-buffer-overflow`) | Deprecated | This query has been deprecated. Use Potentially overrunning write (`cpp/overrunning-write`) and Potentially overrunning write with float to string conversion (`cpp/overrunning-write-with-float`) instead. |
 
 ## Changes to QL libraries
 
+* The `semmle.code.cpp.dataflow.DataFlow` library now supports _definition by reference_ via output parameters of known functions.
+    * Data flows through `memcpy` and `memmove` by default.
+    * Custom flow into or out of arguments assigned by reference can be modelled with the new class `DataFlow::DefinitionByReferenceNode`.
+    * The data flow library adds flow through library functions that are modeled in `semmle.code.cpp.models.interfaces.DataFlow`. Queries can add subclasses of `DataFlowFunction` to specify additional flow.
 * There is a new `Namespace.isInline()` predicate, which holds if the namespace was declared as `inline namespace`.
 * The `Expr.isConstant()` predicate now also holds for _address constant expressions_, which are addresses that will be constant after the program has been linked. These address constants do not have a result for `Expr.getValue()`.
 * There are new `Function.isDeclaredConstexpr()` and `Function.isConstexpr()` predicates. They can be used to tell whether a function was declared as `constexpr`, and whether it actually is `constexpr`.

change-notes/1.20/analysis-javascript.md

@@ -39,6 +39,7 @@
 | **Query**                                  | **Expected impact**          | **Change**                                                                   |
 |--------------------------------------------|------------------------------|------------------------------------------------------------------------------|
 | Ambiguous HTML id attribute                | Fewer false-positive results | This rule now treats templates more conservatively. Its precision has been revised to 'high'. |
+| Assignment to exports variable             | Fewer results                | This rule no longer flags code that is also flagged by the rule "Useless assignment to local variable". |
 | Client-side cross-site scripting           | More true-positive results, fewer false-positive results. | This rule now recognizes WinJS functions that are vulnerable to HTML injection. It no longer flags certain safe uses of jQuery, and recognizes custom sanitizers. |
 | Hard-coded credentials                     | Fewer false-positive results | This rule no longer flag the empty string as a hardcoded username. |
 | Insecure randomness | More results | This rule now flags insecure uses of `crypto.pseudoRandomBytes`. |

change-notes/1.20/extractor-javascript.md

@@ -18,7 +18,7 @@
 
 ## Changes to code extraction
 
-* Extraction of JavaScript files (but not TypeScript files) on LGTM is now parallelized. By default, the extractor uses as many threads as there are processors, but this can be overridden by setting the `LGTM_INDEX_THREADS` environment variable. In particular, setting `LGTM_INDEX_THREADS` to 1 disables parallel extraction.
+* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. The `LGTM_THREADS` environment variable can be set to indicate how many files should be extracted in parallel. If this variable is not set, parallel extraction is disabled.
 * The extractor now offers experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla.
 * The extractor now supports additional [Flow](https://flow.org/) syntax.
 * The extractor now supports [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions.

cpp/config/suites/security/cwe-242

@@ -12,7 +12,6 @@
 @import "cwe-134"
 @import "cwe-170"
 @import "cwe-190"
-@import "cwe-242"
 @import "cwe-253"
 @import "cwe-290"
 @import "cwe-311"

cpp/config/suites/security/default

@@ -46,32 +46,20 @@ predicate allocExprOrIndirect(Expr alloc, string kind) {
     alloc.(FunctionCall).getTarget() = rtn.getEnclosingFunction() and
     (
       allocExprOrIndirect(rtn.getExpr(), kind) or
-      allocReaches(rtn.getExpr(), _, kind)
+      allocReaches0(rtn.getExpr(), _, kind)
     )
   )
 }
 
 /**
- * Holds if `v` is assigned value `e`, and `e` is not known to be `0`.
+ * Holds if `v` is a non-local variable which is assigned with allocations of
+ * type `kind`.
  */
-private predicate nonNullGlobalAssignment(Variable v, Expr e) {
-  not v instanceof LocalScopeVariable and
-  v.getAnAssignedValue() = e and
-  not e.getValue().toInt() = 0
-}
-
-/**
- * Holds if `v` is a non-local variable which is assigned only with allocations of
- * type `kind` (it may also be assigned with NULL).
- */
-private predicate allocReachesVariable(Variable v, Expr alloc, string kind) {
+private pragma[nomagic] predicate allocReachesVariable(Variable v, Expr alloc, string kind) {
   exists(Expr mid |
-    nonNullGlobalAssignment(v, mid) and
-    allocReaches(mid, alloc, kind)
-  ) and
-  forall(Expr mid |
-    nonNullGlobalAssignment(v, mid) |
-    allocReaches(mid, _, kind)
+    not v instanceof LocalScopeVariable and
+    v.getAnAssignedValue() = mid and
+    allocReaches0(mid, alloc, kind)
   )
 }
 
@@ -80,22 +68,35 @@ private predicate allocReachesVariable(Variable v, Expr alloc, string kind) {
  * result of a previous memory allocation `alloc`.  `kind` is a
  * string describing the type of that allocation.
  */
-predicate allocReaches(Expr e, Expr alloc, string kind) {
+private predicate allocReaches0(Expr e, Expr alloc, string kind) {
   (
     // alloc
     allocExprOrIndirect(alloc, kind) and
     e = alloc
   ) or exists(SsaDefinition def, LocalScopeVariable v |
     // alloc via SSA
-    allocReaches(def.getAnUltimateDefiningValue(v), alloc, kind) and
+    allocReaches0(def.getAnUltimateDefiningValue(v), alloc, kind) and
     e = def.getAUse(v)
   ) or exists(Variable v |
-    // alloc via a singly assigned global
+    // alloc via a global
     allocReachesVariable(v, alloc, kind) and
     e.(VariableAccess).getTarget() = v
   )
 }
 
+/**
+ * Holds if `e` is an expression which may evaluate to the
+ * result of previous memory allocations `alloc` only of type
+ * `kind`.
+ */
+predicate allocReaches(Expr e, Expr alloc, string kind) {
+  allocReaches0(e, alloc, kind) and
+  not exists(string k2 |
+    allocReaches0(e, _, k2)  and
+    kind != k2
+  )
+}
+
 /**
  * Holds if `free` is a use of free or delete.  `freed` is the
  * expression that is freed / deleted and `kind` is a string

cpp/ql/src/Critical/NewDelete.qll

@@ -9,6 +9,9 @@
  * @tags reliability
  *       security
  *       external/cwe/cwe-676
+ * @deprecated This query is deprecated, use
+ *             Security/CWE/CWE-120/OverrunWrite.ql and
+ *             Security/CWE/CWE-120/OverrunWriteFloat.ql instead.
  */
 import cpp
 import semmle.code.cpp.commons.Buffer

cpp/ql/src/Likely Bugs/Memory Management/PotentialBufferOverflow.ql

@@ -7,11 +7,11 @@
  * @id cpp/non-virtual-destructor
  * @problem.severity warning
  * @tags reliability
- * @deprecated
+ * @deprecated This query is deprecated, and replaced by
+ *             jsf/4.10 Classes/AV Rule 78.ql, which has far fewer false
+ *             positives on typical code.
  */
 
-// This query is deprecated, and replaced by jsf/4.10 Classes/AV Rule 78.ql, which has far fewer false positives on typical code.
-
 import cpp
 
 from Class base, Destructor d1, Class derived, Destructor d2

cpp/ql/src/Likely Bugs/OO/NonVirtualDestructor.ql

@@ -3,11 +3,9 @@
  * @description Query to help investigate mysterious results with ReturnStackAllocatedObject
  * @kind table
  * @id cpp/points-to/debug
- * @deprecated
+ * @deprecated This query is not suitable for production use and has been deprecated.
  */
 
-// This query is not suitable for production use and has been deprecated.
-
 import cpp
 import semmle.code.cpp.pointsto.PointsTo
 

cpp/ql/src/PointsTo/Debug.ql

@@ -3,11 +3,9 @@
  * @description Query to force evaluation of staged points-to predicates
  * @kind table
  * @id cpp/points-to/prepared-staged-points-to
- * @deprecated
+ * @deprecated This query is not suitable for production use and has been deprecated.
  */
 
-// This query is not suitable for production use and has been deprecated.
-
  import semmle.code.cpp.pointsto.PointsTo
 
  select count(int set, Element location | setlocations(set, unresolveElement(location))),