Merge pull request #1917 from jf205/recent-changes-122

docs: port some recent changes to rc/1.22 branch

Author: shati-patel

docs/language/README.rst

@@ -21,8 +21,7 @@ The QL language documentation currently consists of the following Sphinx project
 - ``ql-handbook``–a user-friendly guide to the QL language
 - ``ql-spec``–formal descriptions of the QL language and QLDoc comments
 - ``support``–the languages and frameworks currently supported in QL analysis
-- ``training``–QL for variant analysis training material
-- ``ql-training-rst``–QL for variant analysis slide shows
+- ``ql-training``–source files for the QL training and variant analysis examples slide decks
 
 Each project contains:
 
@@ -37,8 +36,8 @@ This directory also contains any other files, such as templates and stylesheets,
 that are used by multiple projects.
 Images used in the QL documentation are located in the ``images`` directory.
 
-The ``ql-training-rst`` project contains the source files, themes, and static files 
-used to generate the QL training presentations. 
+The ``ql-training`` project contains the source files, themes, and static files 
+used to generate the QL training and variant analysis presentations. 
 It uses a different configuration from the other projects, and is built using an 
 extension specifically designed for HTML slide shows. 
 For more information, see  
@@ -48,9 +47,12 @@ For more information, see
 Building and previewing the QL language documentation
 *****************************************************
 
-To build and preview the QL documentation locally, you need to install Sphinx. 
+To build and preview the QL documentation and QL training presentations locally, you need to 
+install Sphinx 1.7.9. More recent versions of Sphinx do not work with hieroglyph, 
+the Sphinx extension that we use to generate HTML slides, as explained below. 
 For installation options, see https://github.com/sphinx-doc/sphinx.
 
+
 Using ``sphinx-build``
 ----------------------
 
@@ -69,11 +71,13 @@ For example, to generate the HTML output for a project in the
  
   Add the ``-W`` flag to turn *warnings* into *errors* during the build process. 
   You can use errors reported during the build to debug problems in your source 
-  code, such as broken internal links and malformed tables.
+  code, such as broken internal links and malformed tables. You can also check 
+  external links using Sphinx's `external link builder 
+  <http://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.linkcheck.CheckExternalLinksBuilder>`__.
 
   Add the ``-a`` flag to regenerate all output files. By default, only files that 
   have changed are rebuilt.
-
+  
 Using the reStructuredText Extension for Visual Studio Code
 -----------------------------------------------------------
 
@@ -95,8 +99,10 @@ After installing hieroglyph, you can build the QL training presentations by runn
   sphinx-build -b slides . <slides-output>
 
 generates html slide shows in the ``<slides-output>`` directory when run from
-the ``ql-training-rst`` source directory.
+the ``ql-training`` source directory.
 
+For more information about creating slides for QL training and variant analysis 
+examples, see the `template slide deck <https://github.com/Semmle/ql/blob/master/docs/language/ql-training/template.rst>`__.
 
 Viewing the current version of the QL language documentation
 ************************************************************

docs/language/global-sphinx-files/_templates/layout.html

@@ -60,7 +60,7 @@
             <div class="linkcontainer">
                 <div class="linkbar">
                     <a href="https://help.semmle.com/QL/learn-ql/" target="_blank">Learn QL</a>
-                    <a href="https://help.semmle.com/QL/ql-training/training-index.html" target="_blank">QL training</a>
+                    <a href="https://help.semmle.com/QL/learn-ql/ql-training.html" target="_blank">QL for variant analysis</a>
                     <a href="https://help.semmle.com/QL/ql-tools.html" target="_blank">QL tools</a>
                     <a href="https://help.semmle.com/QL/ql-explore-queries.html" target="_blank">Queries</a>
                     <a href="https://help.semmle.com/QL/ql-reference-topics.html" target="_blank">Reference</a>

docs/language/learn-ql/cpp/introduce-libraries-cpp.rst

@@ -10,6 +10,8 @@ QL queries are easy to write and share–visit the topics below and `our open so
 You can also try out QL in the `query console <https://lgtm.com/query>`__ on `LGTM.com <https://lgtm.com>`__. 
 Here, you can write QL code to query open source projects directly, without having to download snapshots and libraries. 
 
+.. _getting-started:
+
 Getting started
 ***************
 
@@ -22,6 +24,18 @@ If you are new to QL, start by looking at the following topics:
    about-ql
    beginner/ql-tutorials
 
+QL training and variant analysis examples
+******************************************
+
+To start learning how to use QL in variant analysis for a specific language, see:
+
+.. toctree::
+   :maxdepth: -1
+
+   ql-training
+
+.. _writing-ql-queries:
+
 Writing QL queries
 ******************
 

docs/language/learn-ql/index.rst

@@ -20,8 +20,8 @@ The QL Python library incorporates a large number of classes, each class corresp
 
 -  **Syntactic** - classes that represent entities in the Python source code.
 -  **Control flow** - classes that represent entities from the control flow graphs.
--  **Data flow** - classes that assist in performing data flow analyses on Python source code.
--  **Type inference** - classes that represent the inferred types of entities in the Python source code.
+-  **Type inference** - classes that represent the inferred values and types of entities in the Python source code.
+-  **Taint tracking** -  classes that represent the source, sinks and kinds of taint used to implement taint-tracking queries.
 
 Syntactic classes
 ~~~~~~~~~~~~~~~~~
@@ -289,41 +289,52 @@ The classes in the control-flow part of the library are:
 -  `ControlFlowNode <https://help.semmle.com/qldoc/python/semmle/python/Flow.qll/type.Flow$ControlFlowNode.html>`__ – A control-flow node. There is a one-to-many relation between AST nodes and control-flow nodes.
 -  `BasicBlock <https://help.semmle.com/qldoc/python/semmle/python/Flow.qll/type.Flow$BasicBlock.html>`__ – A non branching list of control-flow nodes.
 
-Data flow
-~~~~~~~~~
-
-The ``SsaVariable`` class represents `static single assignment form <http://en.wikipedia.org/wiki/Static_single_assignment_form>`__ variables (SSA variables). There is a one-to-many relation between variables and SSA variables. The ``SsaVariable`` class provides an accurate and fast means of tracking data flow from definition to use; the ``SsaVariable`` class is an important element for building data flow analyses, including type inference.
 
 Type-inference classes
 ----------------------
 
-The QL library for Python also supplies some classes for accessing the inferred types of values. The classes ``Object`` and ``ClassObject`` allow you to query the possible classes that an expression may have at runtime. For example, which ``ClassObjects`` are iterable can be determined using the query:
+The QL library for Python also supplies some classes for accessing the inferred types of values. The classes ``Value`` and ``ClassValue`` allow you to query the possible classes that an expression may have at runtime. For example, which ``ClassValue``\ s are iterable can be determined using the query:
 
-**Find iterable ``ClassObjects``**
+**Find iterable "ClassValue"s**
 
 .. code-block:: ql
 
    import python
 
-   from ClassObject cls
+   from ClassValue cls
    where cls.hasAttribute("__iter__")
    select cls
 
-➤ `See this in the query console <https://lgtm.com/query/688180005/>`__ This query returns a list of classes for the projects analyzed. If you want to include the results for `builtin classes <http://docs.python.org/library/stdtypes.html>`__, which do not have any Python source code, show the non-source results.
+➤ `See this in the query console <https://lgtm.com/query/5151030165280978402/>`__ This query returns a list of classes for the projects analyzed. If you want to include the results for `builtin classes <http://docs.python.org/library/stdtypes.html>`__, which do not have any Python source code, show the non-source results.
 
 Summary
 ~~~~~~~
 
--  `Object <https://help.semmle.com/qldoc/python/semmle/python/types/Object.qll/type.Object$Object.html>`__
+-  `Value <https://help.semmle.com/qldoc/python/semmle/python/objects/ObjectAPI.qll/type.ObjectAPI$Value.html>`__
 
-   -  ``ClassObject``
-   -  ``FunctionObject``
-   -  ``ModuleObject``
+   -  ``ClassValue``
+   -  ``CallableValue``
+   -  ``ModuleValue``
 
 These classes are explained in more detail in :doc:`Tutorial: Points-to analysis and type inference <pointsto-type-infer>`.
 
+Taint-tracking classes
+----------------------
+
+The QL library for Python also supplies classes to specify taint-tracking analyses. The ``Configuration`` class can be overrridden to specify a taint-tracking analysis, by specifying source, sinks, sanitizers and additional flow steps. For those analyses that require additional types of taint to be tracked the ``TaintKind`` class can be overridden.
+
+
+Summary
+~~~~~~~
+
+- `TaintKind <https://help.semmle.com/qldoc/python/semmle/python/security/TaintTracking.qll/type.TaintTracking$TaintKind.html>`__
+- `Configuration <https://help.semmle.com/qldoc/python/semmle/python/security/TaintTracking.qll/type.TaintTracking$TaintTracking$Configuration.html>`__
+
+These classes are explained in more detail in :doc:`Tutorial: Taint tracking and data flow analysis in Python <taint-tracking>`.
+
+
 What next?
 ----------
 
--  Experiment with the worked examples in the QL for Python tutorial topics: :doc:`Functions <functions>`, :doc:`Statements and expressions <statements-expressions>`, :doc:`Control flow <control-flow>` and :doc:`Points-to analysis and type inference <pointsto-type-infer>`.
+-  Experiment with the worked examples in the QL for Python tutorial topics: :doc:`Functions <functions>`, :doc:`Statements and expressions <statements-expressions>`, :doc:`Control flow <control-flow>`, :doc:`Points-to analysis and type inference <pointsto-type-infer>` and :doc:`Taint tracking and data flow analysis in Python <taint-tracking>`.
 -  Find out more about QL in the `QL language handbook <https://help.semmle.com/QL/ql-handbook/index.html>`__ and `QL language specification <https://help.semmle.com/QL/QLLanguageSpecification.html>`__.

docs/language/learn-ql/python/introduce-libraries-python.rst

@@ -101,12 +101,10 @@ Each kind of Python expression has its own class. Here is the full class hierarc
    -  ``BoolExpr`` – Short circuit logical operations, ``x and y``, ``x or y``
    -  ``Bytes`` – A bytes literal, ``b"x"`` or (in Python 2) ``"x"``
    -  ``Call`` – A function call, ``f(arg)``
-   -  ``ClassExpr`` – An artificial expression representing the right hand side a ``ClassDef`` assignment
    -  ``Compare`` – A comparison operation, ``0 < x < 10``
    -  ``Dict`` – A dictionary literal, ``{'a': 2}``
    -  ``DictComp`` – A dictionary comprehension, ``{k: v for ...}``
    -  ``Ellipsis`` – An ellipsis expression, ``...``
-   -  ``FunctionExpr`` – An artificial expression representing the right hand side a ``FunctionDef`` assignment
    -  ``GeneratorExp`` – A generator expression
    -  ``IfExp`` – A conditional expression, ``x if cond else y``
    -  ``ImportExpr`` – An artificial expression representing the module imported
@@ -255,9 +253,9 @@ checks that the value of the attribute (the expression to the left of the dot in
 Class and function definitions
 ------------------------------
 
-As Python is a dynamically typed language, class, and function definitions are executable statements. This means that a class statement is both a statement and a scope containing statements. To represent this cleanly the class definition is broken into a number of parts. At runtime, when a class definition is executed a class object is created and then assigned to a variable of the same name in the scope enclosing the class. This class is created from a code-object representing the source code for the body of the class. To represent this the ``ClassDef`` class (which represents a ``class`` statement) subclasses ``Assign``. The right hand side of the ``ClassDef`` is a ``ClassExpr`` representing the creation of the class. The ``Class`` class, which represents the body of the class, can be accessed via the ``ClassExpr.getInnerScope()``
+As Python is a dynamically typed language, class, and function definitions are executable statements. This means that a class statement is both a statement and a scope containing statements. To represent this cleanly the class definition is broken into a number of parts. At runtime, when a class definition is executed a class object is created and then assigned to a variable of the same name in the scope enclosing the class. This class is created from a code-object representing the source code for the body of the class. To represent this the ``ClassDef`` class (which represents a ``class`` statement) subclasses ``Assign``. The ``Class`` class, which represents the body of the class, can be accessed via the ``ClassDef.getDefinedClass()``
 
-``FunctionDef``, ``FunctionExpr`` and ``Function`` are handled similarly.
+``FunctionDef``, ``Function`` are handled similarly.
 
 Here is the relevant part of the class hierarchy:
 
@@ -268,12 +266,6 @@ Here is the relevant part of the class hierarchy:
       -  ``ClassDef``
       -  ``FunctionDef``
 
--  ``Expr``
-
-   -  ``ClassExp``
-
-      -  ``FunctionExpr``
-
 -  ``Scope``
 
    -  ``Class``
@@ -283,4 +275,4 @@ What next?
 ----------
 
 -  Experiment with the worked examples in the QL for Python tutorial topics: :doc:`Control flow <control-flow>`, :doc:`Points-to analysis and type inference <pointsto-type-infer>`.
--  Find out more about QL in the `QL language handbook <https://help.semmle.com/QL/ql-handbook/index.html>`__ and `QL language specification <https://help.semmle.com/QL/QLLanguageSpecification.html>`__.
+-  Find out more about QL in the `QL language handbook <https://help.semmle.com/QL/ql-handbook/index.html>`__ and `QL language specification <https://help.semmle.com/QL/QLLanguageSpecification.html>`__.

docs/language/learn-ql/python/statements-expressions.rst

@@ -96,7 +96,7 @@ The sink is defined by using a custom ``TaintTracking::Sink`` class.
     class UnsafeSink extends TaintTracking::Sink {
 
         UnsafeSink() {
-            exists(FunctionObject unsafe |
+            exists(FunctionValue unsafe |
                 unsafe.getName() = "unsafe" and
                 unsafe.getACall().(CallNode).getAnArg() = this
             )
@@ -172,7 +172,7 @@ Thus, our example query becomes:
     class UnsafeSink extends TaintTracking::Sink {
 
         UnsafeSink() {
-            exists(FunctionObject unsafe |
+            exists(FunctionValue unsafe |
                 unsafe.getName() = "unsafe" and
                 unsafe.getACall().(CallNode).getAnArg() = this
             )
@@ -255,4 +255,4 @@ What next?
 ----------
 
 -  Experiment with the worked examples in the QL for Python tutorial topics: :doc:`Control flow <control-flow>`, and :doc:`Points-to analysis and type inference <pointsto-type-infer>`.
--  Find out more about QL in the `QL language handbook <https://help.semmle.com/QL/ql-handbook/index.html>`__ and `QL language specification <https://help.semmle.com/QL/QLLanguageSpecification.html>`__.
+-  Find out more about QL in the `QL language handbook <https://help.semmle.com/QL/ql-handbook/index.html>`__ and `QL language specification <https://help.semmle.com/QL/QLLanguageSpecification.html>`__.