Merge pull request #9369 from asgerf/python/api-graph-api

Python: API graph renaming and documentation

Author: asgerf

python/ql/lib/change-notes/2022-06-28-api-graph-api.md

@@ -0,0 +1,10 @@
+---
+category: deprecated
+---
+
+- The documentation of API graphs (the `API` module) has been expanded, and some of the members predicates of `API::Node`
+  have been renamed as follows:
+  - `getAnImmediateUse` -> `asSource`
+  - `getARhs` -> `asSink`
+  - `getAUse` -> `getAValueReachableFromSource`
+  - `getAValueReachingRhs` -> `getAValueReachingSink`

python/ql/lib/semmle/python/ApiGraphs.qll

@@ -12,76 +12,165 @@ import semmle.python.dataflow.new.DataFlow
 private import semmle.python.internal.CachedStages
 
 /**
- * Provides classes and predicates for working with APIs used in a database.
+ * Provides classes and predicates for working with the API boundary between the current
+ * codebase and external libraries.
+ *
+ * See `API::Node` for more in-depth documentation.
  */
 module API {
   /**
-   * An abstract representation of a definition or use of an API component such as a function
-   * exported by a Python package, or its result.
+   * A node in the API graph, representing a value that has crossed the boundary between this
+   * codebase and an external library (or in general, any external codebase).
+   *
+   * ### Basic usage
+   *
+   * API graphs are typically used to identify "API calls", that is, calls to an external function
+   * whose implementation is not necessarily part of the current codebase.
+   *
+   * The most basic use of API graphs is typically as follows:
+   * 1. Start with `API::moduleImport` for the relevant library.
+   * 2. Follow up with a chain of accessors such as `getMember` describing how to get to the relevant API function.
+   * 3. Map the resulting API graph nodes to data-flow nodes, using `asSource` or `asSink`.
+   *
+   * For example, a simplified way to get the first argument of a call to `json.dumps` would be
+   * ```ql
+   * API::moduleImport("json").getMember("dumps").getParameter(0).asSink()
+   * ```
+   *
+   * The most commonly used accessors are `getMember`, `getParameter`, and `getReturn`.
+   *
+   * ### API graph nodes
+   *
+   * There are two kinds of nodes in the API graphs, distinguished by who is "holding" the value:
+   * - **Use-nodes** represent values held by the current codebase, which came from an external library.
+   *   (The current codebase is "using" a value that came from the library).
+   * - **Def-nodes** represent values held by the external library, which came from this codebase.
+   *   (The current codebase "defines" the value seen by the library).
+   *
+   * API graph nodes are associated with data-flow nodes in the current codebase.
+   * (API graphs are designed to work when external libraries are not part of the database,
+   * so we do not associate with concrete data-flow nodes from the external library).
+   * - **Use-nodes** are associated with data-flow nodes where a value enters the current codebase,
+   *   such as the return value of a call to an external function.
+   * - **Def-nodes** are associated with data-flow nodes where a value leaves the current codebase,
+   *   such as an argument passed in a call to an external function.
+   *
+   *
+   * ### Access paths and edge labels
+   *
+   * Nodes in the API graph are associated with a set of access paths, describing a series of operations
+   * that may be performed to obtain that value.
+   *
+   * For example, the access path `API::moduleImport("json").getMember("dumps")` represents the action of
+   * importing `json` and then accessing the member `dumps` on the resulting object.
+   *
+   * Each edge in the graph is labelled by such an "operation". For an edge `A->B`, the type of the `A` node
+   * determines who is performing the operation, and the type of the `B` node determines who ends up holding
+   * the result:
+   * - An edge starting from a use-node describes what the current codebase is doing to a value that
+   *   came from a library.
+   * - An edge starting from a def-node describes what the external library might do to a value that
+   *   came from the current codebase.
+   * - An edge ending in a use-node means the result ends up in the current codebase (at its associated data-flow node).
+   * - An edge ending in a def-node means the result ends up in external code (its associated data-flow node is
+   *   the place where it was "last seen" in the current codebase before flowing out)
+   *
+   * Because the implementation of the external library is not visible, it is not known exactly what operations
+   * it will perform on values that flow there. Instead, the edges starting from a def-node are operations that would
+   * lead to an observable effect within the current codebase; without knowing for certain if the library will actually perform
+   * those operations. (When constructing these edges, we assume the library is somewhat well-behaved).
+   *
+   * For example, given this snippet:
+   * ```python
+   * import foo
+   * foo.bar(lambda x: doSomething(x))
+   * ```
+   * A callback is passed to the external function `foo.bar`. We can't know if `foo.bar` will actually invoke this callback.
+   * But _if_ the library should decide to invoke the callback, then a value will flow into the current codebase via the `x` parameter.
+   * For that reason, an edge is generated representing the argument-passing operation that might be performed by `foo.bar`.
+   * This edge is going from the def-node associated with the callback to the use-node associated with the parameter `x`.
    */
   class Node extends Impl::TApiNode {
     /**
-     * Gets a data-flow node corresponding to a use of the API component represented by this node.
+     * Gets a data-flow node where this value may flow after entering the current codebase.
      *
-     * For example, `import re; re.escape` is a use of the `escape` function from the
-     * `re` module, and `import re; re.escape("hello")` is a use of the return of that function.
-     *
-     * This includes indirect uses found via data flow, meaning that in
-     * ```python
-     * def f(x):
-     *     pass
-     *
-     * f(obj.foo)
-     * ```
-     * both `obj.foo` and `x` are uses of the `foo` member from `obj`.
+     * This is similar to `asSource()` but additionally includes nodes that are transitively reachable by data flow.
+     * See `asSource()` for examples.
      */
-    DataFlow::Node getAUse() {
+    DataFlow::Node getAValueReachableFromSource() {
       exists(DataFlow::LocalSourceNode src | Impl::use(this, src) |
         Impl::trackUseNode(src).flowsTo(result)
       )
     }
 
     /**
-     * Gets a data-flow node corresponding to the right-hand side of a definition of the API
-     * component represented by this node.
+     * Gets a data-flow node where this value leaves the current codebase and flows into an
+     * external library (or in general, any external codebase).
      *
-     * For example, in the property write `foo.bar = x`, variable `x` is the the right-hand side
-     * of a write to the `bar` property of `foo`.
+     * Concretely, this is either an argument passed to a call to external code,
+     * or the right-hand side of an attribute write on an object flowing into such a call.
      *
-     * Note that for parameters, it is the arguments flowing into that parameter that count as
-     * right-hand sides of the definition, not the declaration of the parameter itself.
-     * Consequently, in :
+     * For example:
      * ```python
-     * from mypkg import foo;
+     * import foo
+     *
+     * # 'x' is matched by API::moduleImport("foo").getMember("bar").getParameter(0).asSink()
      * foo.bar(x)
+     *
+     * # 'x' is matched by API::moduleImport("foo").getMember("bar").getParameter(0).getMember("prop").asSink()
+     * obj.prop = x
+     * foo.bar(obj);
      * ```
-     * `x` is the right-hand side of a definition of the first parameter of `bar` from the `mypkg.foo` module.
+     *
+     * This predicate does not include nodes transitively reaching the sink by data flow;
+     * use `getAValueReachingSink` for that.
      */
-    DataFlow::Node getARhs() { Impl::rhs(this, result) }
+    DataFlow::Node asSink() { Impl::rhs(this, result) }
 
     /**
-     * Gets a data-flow node that may interprocedurally flow to the right-hand side of a definition
-     * of the API component represented by this node.
+     * Gets a data-flow node that transitively flows to an external library (or in general, any external codebase).
+     *
+     * This is similar to `asSink()` but additionally includes nodes that transitively reach a sink by data flow.
+     * See `asSink()` for examples.
      */
-    DataFlow::Node getAValueReachingRhs() { result = Impl::trackDefNode(this.getARhs()) }
+    DataFlow::Node getAValueReachingSink() { result = Impl::trackDefNode(this.asSink()) }
 
     /**
-     * Gets an immediate use of the API component represented by this node.
+     * Gets a data-flow node where this value enters the current codebase.
      *
-     * For example, `import re; re.escape` is a an immediate use of the `escape` member
-     * from the `re` module.
+     * For example:
+     * ```python
+     * # API::moduleImport("re").asSource()
+     * import re
+     *
+     * # API::moduleImport("re").getMember("escape").asSource()
+     * re.escape
+     *
+     * # API::moduleImport("re").getMember("escape").getReturn().asSource()
+     * re.escape()
+     * ```
      *
-     * Unlike `getAUse()`, this predicate only gets the immediate references, not the indirect uses
-     * found via data flow. This means that in `x = re.escape` only `re.escape` is a reference
-     * to the `escape` member of `re`, neither `x` nor any node that `x` flows to is a reference to
-     * this API component.
+     * This predicate does not include nodes transitively reachable by data flow;
+     * use `getAValueReachableFromSource` for that.
      */
-    DataFlow::LocalSourceNode getAnImmediateUse() { Impl::use(this, result) }
+    DataFlow::LocalSourceNode asSource() { Impl::use(this, result) }
+
+    /** DEPRECATED. This predicate has been renamed to `getAValueReachableFromSource()`. */
+    deprecated DataFlow::Node getAUse() { result = this.getAValueReachableFromSource() }
+
+    /** DEPRECATED. This predicate has been renamed to `asSource()`. */
+    deprecated DataFlow::LocalSourceNode getAnImmediateUse() { result = this.asSource() }
+
+    /** DEPRECATED. This predicate has been renamed to `asSink()`. */
+    deprecated DataFlow::Node getARhs() { result = this.asSink() }
+
+    /** DEPRECATED. This predicate has been renamed to `getAValueReachingSink()`. */
+    deprecated DataFlow::Node getAValueReachingRhs() { result = this.getAValueReachingSink() }
 
     /**
      * Gets a call to the function represented by this API component.
      */
-    CallNode getACall() { result = this.getReturn().getAnImmediateUse() }
+    CallNode getACall() { result = this.getReturn().asSource() }
 
     /**
      * Gets a node representing member `m` of this API component.
@@ -306,7 +395,7 @@ module API {
   class CallNode extends DataFlow::CallCfgNode {
     API::Node callee;
 
-    CallNode() { this = callee.getReturn().getAnImmediateUse() }
+    CallNode() { this = callee.getReturn().asSource() }
 
     /** Gets the API node for the `i`th parameter of this invocation. */
     pragma[nomagic]
@@ -319,14 +408,14 @@ module API {
      * Gets an API node where a RHS of the node is the `i`th argument to this call.
      */
     pragma[noinline]
-    private Node getAParameterCandidate(int i) { result.getARhs() = this.getArg(i) }
+    private Node getAParameterCandidate(int i) { result.asSink() = this.getArg(i) }
 
     /** Gets the API node for a parameter of this invocation. */
     Node getAParameter() { result = this.getParameter(_) }
 
     /** Gets the object that this method-call is being called on, if this is a method-call */
     Node getSelfParameter() {
-      result.getARhs() = this.(DataFlow::MethodCallNode).getObject() and
+      result.asSink() = this.(DataFlow::MethodCallNode).getObject() and
       result = callee.getSelfParameter()
     }
 
@@ -346,13 +435,13 @@ module API {
 
     pragma[noinline]
     private Node getAKeywordParameterCandidate(string name) {
-      result.getARhs() = this.getArgByName(name)
+      result.asSink() = this.getArgByName(name)
     }
 
     /** Gets the API node for the return value of this call. */
     Node getReturn() {
       result = callee.getReturn() and
-      result.getAnImmediateUse() = this
+      result.asSource() = this
     }
 
     /**

python/ql/lib/semmle/python/filters/Tests.qll

@@ -9,7 +9,7 @@ class UnitTestClass extends TestScope, Class {
       testCaseString.matches("%TestCase") and
       testCaseClass = any(API::Node mod).getMember(testCaseString)
     |
-      this.getParent() = testCaseClass.getASubclass*().getAnImmediateUse().asExpr()
+      this.getParent() = testCaseClass.getASubclass*().asSource().asExpr()
     )
   }
 }

python/ql/lib/semmle/python/frameworks/Aiohttp.qll

@@ -243,9 +243,7 @@ module AiohttpWebModel {
 
   /** A class that has a super-type which is an aiohttp.web View class. */
   class AiohttpViewClassFromSuperClass extends AiohttpViewClass {
-    AiohttpViewClassFromSuperClass() {
-      this.getParent() = View::subclassRef().getAnImmediateUse().asExpr()
-    }
+    AiohttpViewClassFromSuperClass() { this.getParent() = View::subclassRef().asSource().asExpr() }
   }
 
   /** A class that is used in a route-setup, therefore being considered an aiohttp.web View class. */
@@ -628,7 +626,8 @@ module AiohttpWebModel {
         // and just go with the LHS
         this.asCfgNode() = subscript
       |
-        subscript.getObject() = aiohttpResponseInstance().getMember("cookies").getAUse().asCfgNode() and
+        subscript.getObject() =
+          aiohttpResponseInstance().getMember("cookies").getAValueReachableFromSource().asCfgNode() and
         value.asCfgNode() = subscript.(DefinitionNode).getValue() and
         index.asCfgNode() = subscript.getIndex()
       )
@@ -686,8 +685,8 @@ private module AiohttpClientModel {
         DataFlow::Node disablingNode, DataFlow::Node argumentOrigin
       ) {
         exists(API::Node param | param = this.getKeywordParameter(["ssl", "verify_ssl"]) |
-          disablingNode = param.getARhs() and
-          argumentOrigin = param.getAValueReachingRhs() and
+          disablingNode = param.asSink() and
+          argumentOrigin = param.getAValueReachingSink() and
           // aiohttp.client treats `None` as the default and all other "falsey" values as `False`.
           argumentOrigin.asExpr().(ImmutableLiteral).booleanValue() = false and
           not argumentOrigin.asExpr() instanceof None

python/ql/lib/semmle/python/frameworks/Aiomysql.qll

@@ -53,7 +53,7 @@ private module Aiomysql {
   class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
     CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
 
-    override DataFlow::Node getSql() { result = this.getParameter(0, "operation").getARhs() }
+    override DataFlow::Node getSql() { result = this.getParameter(0, "operation").asSink() }
   }
 
   /**
@@ -63,7 +63,7 @@ private module Aiomysql {
   class AwaitedCursorExecuteCall extends SqlExecution::Range {
     CursorExecuteCall executeCall;
 
-    AwaitedCursorExecuteCall() { this = executeCall.getReturn().getAwaited().getAnImmediateUse() }
+    AwaitedCursorExecuteCall() { this = executeCall.getReturn().getAwaited().asSource() }
 
     override DataFlow::Node getSql() { result = executeCall.getSql() }
   }
@@ -94,7 +94,7 @@ private module Aiomysql {
   class SAConnectionExecuteCall extends SqlConstruction::Range, API::CallNode {
     SAConnectionExecuteCall() { this = saConnection().getMember("execute").getACall() }
 
-    override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
+    override DataFlow::Node getSql() { result = this.getParameter(0, "query").asSink() }
   }
 
   /**
@@ -104,7 +104,7 @@ private module Aiomysql {
   class AwaitedSAConnectionExecuteCall extends SqlExecution::Range {
     SAConnectionExecuteCall execute;
 
-    AwaitedSAConnectionExecuteCall() { this = execute.getReturn().getAwaited().getAnImmediateUse() }
+    AwaitedSAConnectionExecuteCall() { this = execute.getReturn().getAwaited().asSource() }
 
     override DataFlow::Node getSql() { result = execute.getSql() }
   }

python/ql/lib/semmle/python/frameworks/Aiopg.qll

@@ -53,7 +53,7 @@ private module Aiopg {
   class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
     CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
 
-    override DataFlow::Node getSql() { result = this.getParameter(0, "operation").getARhs() }
+    override DataFlow::Node getSql() { result = this.getParameter(0, "operation").asSink() }
   }
 
   /**
@@ -63,7 +63,7 @@ private module Aiopg {
   class AwaitedCursorExecuteCall extends SqlExecution::Range {
     CursorExecuteCall execute;
 
-    AwaitedCursorExecuteCall() { this = execute.getReturn().getAwaited().getAnImmediateUse() }
+    AwaitedCursorExecuteCall() { this = execute.getReturn().getAwaited().asSource() }
 
     override DataFlow::Node getSql() { result = execute.getSql() }
   }
@@ -90,7 +90,7 @@ private module Aiopg {
   class SAConnectionExecuteCall extends SqlConstruction::Range, API::CallNode {
     SAConnectionExecuteCall() { this = saConnection().getMember("execute").getACall() }
 
-    override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
+    override DataFlow::Node getSql() { result = this.getParameter(0, "query").asSink() }
   }
 
   /**
@@ -100,7 +100,7 @@ private module Aiopg {
   class AwaitedSAConnectionExecuteCall extends SqlExecution::Range {
     SAConnectionExecuteCall excute;
 
-    AwaitedSAConnectionExecuteCall() { this = excute.getReturn().getAwaited().getAnImmediateUse() }
+    AwaitedSAConnectionExecuteCall() { this = excute.getReturn().getAwaited().asSource() }
 
     override DataFlow::Node getSql() { result = excute.getSql() }
   }