Merge branch 'master' of github.com:Semmle/ql into rdmarsh/cpp/ir-buffer-read-call-se

Author: Robert Marsh

.codeqlmanifest.json

@@ -0,0 +1,5 @@
+{ "provide": [ "*/ql/src/qlpack.yml",
+               "*/upgrades/qlpack.yml",
+               "misc/legacy-support/*/qlpack.yml",
+               "misc/suite-helpers/qlpack.yml",
+               "codeql/.codeqlmanifest.json" ] }

change-notes/1.23/analysis-cpp.md

@@ -22,6 +22,7 @@ The following changes in version 1.23 affect C/C++ analysis in all applications.
 | Too few arguments to formatting function (`cpp/wrong-number-format-arguments`) | Fewer false positive results | Fixed false positives resulting from mistmatching declarations of a formatting function. |
 | Too many arguments to formatting function (`cpp/too-many-format-arguments`) | Fewer false positive results | Fixed false positives resulting from mistmatching declarations of a formatting function. |
 | Unclear comparison precedence (`cpp/comparison-precedence`) | Fewer false positive results | False positives involving template classes and functions have been fixed. |
+| Comparison of narrow type with wide type in loop condition (`cpp/comparison-with-wider-type`) | Higher precision | The precision of this query has been increased to "high" as the alerts from this query have proved to be valuable on real-world projects. With this precision, results are now displayed by default in LGTM. |
 
 ## Changes to QL libraries
 
@@ -32,6 +33,7 @@ The following changes in version 1.23 affect C/C++ analysis in all applications.
   picture of the partial flow paths from a given source. The feature is
   disabled by default and can be enabled for individual configurations by
   overriding `int explorationLimit()`.
+* The data-flow library now supports flow out of C++ reference parameters.
 * The data-flow library now allows flow through the address-of operator (`&`).
 * The `DataFlow::DefinitionByReferenceNode` class now considers `f(x)` to be a
   definition of `x` when `x` is a variable of pointer type. It no longer

change-notes/1.23/analysis-java.md

@@ -2,6 +2,12 @@
 
 The following changes in version 1.23 affect Java analysis in all applications.
 
+## New queries
+
+| **Query**                   | **Tags**  | **Purpose**                                                        |
+|-----------------------------|-----------|--------------------------------------------------------------------|
+| Continue statement that does not continue (`java/continue-in-false-loop`) | correctness | Finds `continue` statements in `do { ... } while (false)` loops. |
+
 ## Changes to existing queries
 
 | **Query**                    | **Expected impact**    | **Change**                        |
@@ -11,6 +17,7 @@ The following changes in version 1.23 affect Java analysis in all applications.
 | Query built from user-controlled sources (`java/sql-injection`) | More results | The query now identifies arguments to `Statement.executeLargeUpdate` and `Connection.prepareCall` as SQL expressions sinks. |
 | Query built from local-user-controlled sources (`java/sql-injection-local`) | More results | The query now identifies arguments to `Statement.executeLargeUpdate` and `Connection.prepareCall` as SQL expressions sinks. |
 | Query built without neutralizing special characters (`java/concatenated-sql-query`) | More results | The query now identifies arguments to `Statement.executeLargeUpdate` and `Connection.prepareCall` as SQL expressions sinks. |
+| Useless comparison test (`java/constant-comparison`) | Fewer false positives | Additional overflow check patterns are now recognized and no longer reported. |
 
 ## Changes to QL libraries
 

change-notes/1.23/analysis-javascript.md

@@ -2,6 +2,8 @@
 
 ## General improvements
 
+* Suppor for `globalThis` has been added.
+
 * Support for the following frameworks and libraries has been improved:
   - [firebase](https://www.npmjs.com/package/firebase)
   - [mongodb](https://www.npmjs.com/package/mongodb)
@@ -10,15 +12,20 @@
 
 * The call graph has been improved to resolve method calls in more cases. This may produce more security alerts.
 
+* TypeScript 3.6 features are supported.
+
+
 ## New queries
 
 | **Query**                                                                 | **Tags**                                                          | **Purpose**                                                                                                                                                                            |
 |---------------------------------------------------------------------------|-------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | Unused index variable (`js/unused-index-variable`)                        | correctness                                                       | Highlights loops that iterate over an array, but do not use the index variable to access array elements, indicating a possible typo or logic error. Results are shown on LGTM by default. |
-| Loop bound injection (`js/loop-bound-injection`)                          | security, external/cwe/cwe-834                                      | Highlights loops where a user-controlled object with an arbitrary .length value can trick the server to loop indefinitely. Results are not shown on LGTM by default. |
+| Loop bound injection (`js/loop-bound-injection`)                          | security, external/cwe/cwe-834                                      | Highlights loops where a user-controlled object with an arbitrary .length value can trick the server to loop indefinitely. Results are shown on LGTM by default. |
 | Suspicious method name (`js/suspicious-method-name-declaration`)          | correctness, typescript, methods                                  | Highlights suspiciously named methods where the developer likely meant to write a constructor or function. Results are shown on LGTM by default. |
+| Shell command built from environment values (`js/shell-command-injection-from-environment`) | correctness, security, external/cwe/cwe-078, external/cwe/cwe-088 | Highlights shell commands that may change behavior inadvertently depending on the execution environment, indicating a possible violation of [CWE-78](https://cwe.mitre.org/data/definitions/78.html). Results are shown on LGTM by default.|
 | Use of returnless function (`js/use-of-returnless-function`)              | maintainability, correctness                                      | Highlights calls where the return value is used, but the callee never returns a value. Results are shown on LGTM by default. |
 | Useless regular expression character escape (`js/useless-regexp-character-escape`) | correctness, security, external/cwe/cwe-20 | Highlights regular expression strings with useless character escapes, indicating a possible violation of [CWE-20](https://cwe.mitre.org/data/definitions/20.html). Results are shown on LGTM by default. |
+| Unreachable method overloads (`js/unreachable-method-overloads`)          | correctness, typescript                                           | Highlights method overloads that are impossible to use from client code. Results are shown on LGTM by default. |
 
 ## Changes to existing queries
 
@@ -36,7 +43,27 @@
 | Reflected cross-site scripting (`js/reflected-xss`) | Fewer false-positive results | The query now recognizes more sanitizers. |
 | Stored cross-site scripting (`js/stored-xss`) | Fewer false-positive results | The query now recognizes more sanitizers. |
 | Uncontrolled command line (`js/command-line-injection`) | More results | This query now treats responses from servers as untrusted. |
+| Uncontrolled data used in path expression (`js/path-injection`) | Fewer false-positive results | This query now recognizes calls to Express `sendFile` as safe in some cases. |
+| Unknown directive (`js/unknown-directive`) | Fewer false positive results | This query no longer flags uses of ":", which is sometimes used like a directive. |
 
 ## Changes to QL libraries
 
 * `Expr.getDocumentation()` now handles chain assignments.
+
+## Removal of deprecated queries
+
+The following queries (deprecated since 1.17) are no longer available in the distribution:
+
+* Builtin redefined (js/builtin-redefinition)
+* Inefficient method definition (js/method-definition-in-constructor)
+* Bad parity check (js/incomplete-parity-check)
+* Potentially misspelled property or variable name (js/wrong-capitalization)
+* Unknown JSDoc tag (js/jsdoc/unknown-tag-type)
+* Invalid JSLint directive (js/jslint/invalid-directive)
+* Malformed JSLint directive (js/jslint/malformed-directive)
+* Use of HTML comments (js/html-comment)
+* Multi-line string literal (js/multi-line-string)
+* Octal literal (js/octal-literal)
+* Reserved word used as variable name (js/use-of-reserved-word)
+* Trailing comma in array or object expressions (js/trailing-comma-in-array-or-object)
+* Call to parseInt without radix (js/parseint-without-radix)

change-notes/1.23/analysis-python.md

@@ -12,3 +12,11 @@
 | Clear-text logging of sensitive information (`py/clear-text-logging-sensitive-data`) | security, external/cwe/cwe-312 | Finds instances where sensitive information is logged without encryption or hashing. Results are shown on LGTM by default. |
 | Clear-text storage of sensitive information (`py/clear-text-storage-sensitive-data`) | security, external/cwe/cwe-312 | Finds instances where sensitive information is stored without encryption or hashing. Results are shown on LGTM by default. |
 | Binding a socket to all network interfaces (`py/bind-socket-all-network-interfaces`) | security | Finds instances where a socket is bound to all network interfaces. Results are shown on LGTM by default. |
+
+
+## Changes to existing queries
+
+| **Query**                  | **Expected impact**    | **Change** |
+|----------------------------|------------------------|------------|
+| Unreachable code | Fewer false positives | Analysis now accounts for uses of `contextlib.suppress` to suppress exceptions. |
+| `__iter__` method returns a non-iterator | Better alert message | Alert now highlights which class is expected to be an iterator. |

change-notes/1.23/extractor-javascript.md

@@ -5,6 +5,17 @@
 ## Changes to code extraction
 
 * Asynchronous generator methods are now parsed correctly and no longer cause a spurious syntax error.
+* Files in `node_modules` and `bower_components` folders are no longer extracted by default. If you still want to extract files from these folders, you can add the following filters to your `lgtm.yml` file (or add them to existing filters):
+
+```yaml
+extraction:
+  javascript:
+    index:
+      filters:
+        - include: "**/node_modules"
+        - include: "**/bower_components"
+```
+
 * Recognition of CommonJS modules has improved. As a result, some files that were previously extracted as
   global scripts are now extracted as modules.
 * Top-level `await` is now supported.

config/identical-files.json

@@ -76,7 +76,11 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/Operand.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll",
     "csharp/ql/src/semmle/code/csharp/ir/implementation/raw/Operand.qll",
-    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/Operand.qll"
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/Operand.qll" 
+  ],
+  "IR IRType": [
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/IRType.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/IRType.qll"
   ],
   "IR Operand Tag": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/OperandTag.qll",
@@ -169,22 +173,39 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/PrintIRImports.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/PrintIRImports.qll"
   ],
+  "C++ SSA SSAConstructionImports": [
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SSAConstructionImports.qll",
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstructionImports.qll"
+  ],
   "C++ SSA AliasAnalysis": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/AliasAnalysis.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/AliasAnalysis.qll"
   ],
-  "C++ SSA SSAConstruction": [
+  "C++ IR ValueNumberingImports": [
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll",
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/internal/ValueNumberingImports.qll"
+  ],
+  "IR SSA SimpleSSA": [
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll"
+  ],
+  "IR SSA SSAConstruction": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SSAConstruction.qll",
-    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstruction.qll"
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstruction.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/internal/SSAConstruction.qll"
   ],
-  "C++ SSA PrintSSA": [
+  "IR SSA PrintSSA": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/PrintSSA.qll",
-    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/PrintSSA.qll"
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/PrintSSA.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/internal/PrintSSA.qll"
   ],
-  "C++ IR ValueNumber": [
+  "IR ValueNumber": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/gvn/ValueNumbering.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/ValueNumbering.qll",
-    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/ValueNumbering.qll"
+    "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/ValueNumbering.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/raw/gvn/ValueNumbering.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/gvn/ValueNumbering.qll"
   ],
   "C++ IR ConstantAnalysis": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/constant/ConstantAnalysis.qll",
@@ -235,5 +256,9 @@
   "C# IR PrintIRImports": [
     "csharp/ql/src/semmle/code/csharp/ir/implementation/raw/internal/PrintIRImports.qll",
     "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/internal/PrintIRImports.qll"
+  ],
+  "C# IR ValueNumberingImports": [
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
+    "csharp/ql/src/semmle/code/csharp/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll"
   ]
 }

cpp/ql/src/Likely Bugs/Arithmetic/BadAdditionOverflowCheck.qhelp

@@ -2,36 +2,39 @@
   "-//Semmle//qhelp//EN"
   "qhelp.dtd">
 <qhelp>
-  <overview>
-    <p>
-      Checking for overflow of integer addition needs to be done with
-      care, because automatic type promotion can prevent the check
-      from working correctly.
-    </p>
-  </overview>
-  <recommendation>
-    <p>
-      Use an explicit cast to make sure that the result of the addition is
-      not implicitly converted to a larger type.
-    </p>
-  </recommendation>
-  <example>
-    <sample src="BadAdditionOverflowCheckExample1.cpp" />
-    <p>
-      On a typical architecture where <tt>short</tt> is 16 bits
-      and <tt>int</tt> is 32 bits, the operands of the addition are
-      automatically promoted to <tt>int</tt>, so it cannot overflow
-      and the result of the comparison is always false.
-    </p>
-    <p>
-      The code below implements the check correctly, by using an
-      explicit cast to make sure that the result of the addition
-      is <tt>unsigned short</tt>.
-    </p>
-    <sample src="BadAdditionOverflowCheckExample2.cpp" />
-  </example>
-  <references>
-    <li><a href="http://c-faq.com/expr/preservingrules.html">Preserving Rules</a></li>
-    <li><a href="https://www.securecoding.cert.org/confluence/plugins/servlet/mobile#content/view/20086942">Understand integer conversion rules</a></li>
-  </references>
+
+<overview>
+<p>
+Checking for overflow of integer addition needs to be done with
+care, because automatic type promotion can prevent the check
+from working as intended, with the same value (<code>true</code>
+or <code>false</code>) always being returned.
+</p>
+</overview>
+<recommendation>
+<p>
+Use an explicit cast to make sure that the result of the addition is
+not implicitly converted to a larger type.
+</p>
+</recommendation>
+<example>
+<sample src="BadAdditionOverflowCheckExample1.cpp" />
+<p>
+On a typical architecture where <code>short</code> is 16 bits
+and <code>int</code> is 32 bits, the operands of the addition are
+automatically promoted to <code>int</code>, so it cannot overflow
+and the result of the comparison is always false.
+</p>
+<p>
+The code below implements the check correctly, by using an
+explicit cast to make sure that the result of the addition
+is <code>unsigned short</code> (which may overflow, in which case
+the comparison would evaluate to <code>true</code>).
+</p>
+<sample src="BadAdditionOverflowCheckExample2.cpp" />
+</example>
+<references>
+<li><a href="http://c-faq.com/expr/preservingrules.html">Preserving Rules</a></li>
+<li><a href="https://www.securecoding.cert.org/confluence/plugins/servlet/mobile#content/view/20086942">Understand integer conversion rules</a></li>
+</references>
 </qhelp>

cpp/ql/src/Likely Bugs/Arithmetic/BadAdditionOverflowCheckExample1.cpp

@@ -1,3 +1,4 @@
 bool checkOverflow(unsigned short x, unsigned short y) {
-  return (x + y < x);  // BAD: x and y are automatically promoted to int.
+  // BAD: comparison is always false due to type promotion
+  return (x + y < x);  
 }

cpp/ql/src/Likely Bugs/Arithmetic/ComparisonPrecedence.ql

@@ -18,4 +18,4 @@ where
   co.getAChild() = chco and
   not chco.isParenthesised() and
   not co.isFromUninstantiatedTemplate(_)
-select co, "Check the comparison operator precedence."
+select co, "Comparison as an operand to another comparison."