Python: Add non-alert data for lines of code

`py/summary/lines-of-code` is just a port of the C++/JS queries added in:

- #5271 (C++)
- #5304 (JS)

We are the first to implement the `lines-of-user-code` query, so nothing to
compare with in other languages -- but it makes a lot of sense to do for Python 👍

Author: RasmusWL

python/ql/src/Summary/LinesOfCode.ql

@@ -0,0 +1,13 @@
+/**
+ * @id py/summary/lines-of-code
+ * @name Total lines of Python code in the database
+ * @description The total number of lines of Python code across all files, including
+ *   external libraries and auto-generated files. This is a useful metric of the size of a
+ *   database. This query counts the lines of code, excluding whitespace or comments.
+ * @kind metric
+ * @tags summary
+ */
+
+import python
+
+select sum(Module m | | m.getMetrics().getNumberOfLinesOfCode())

python/ql/src/Summary/LinesOfUserCode.ql

@@ -0,0 +1,21 @@
+/**
+ * @id py/summary/lines-of-user-code
+ * @name Total lines of user written Python code in the database
+ * @description The total number of lines of Python code from the source code directory,
+ *   excluding auto-generated files. This query counts the lines of code, excluding
+ *   whitespace or comments. Note: If external libraries are included in the codebase
+ *   either in a checked-in virtual environment or as vendored code, that will currently
+ *   be counted as user written code.
+ * @kind metric
+ * @tags summary
+ */
+
+import python
+import semmle.python.filters.GeneratedCode
+
+select sum(Module m |
+    exists(m.getFile().getRelativePath()) and
+    not m.getFile() instanceof GeneratedFile
+  |
+    m.getMetrics().getNumberOfLinesOfCode()
+  )

python/ql/test/query-tests/Summary/LinesOfCode.expected

@@ -0,0 +1 @@
+| 38 |

python/ql/test/query-tests/Summary/LinesOfCode.qlref

@@ -0,0 +1 @@
+Summary/LinesOfCode.ql

python/ql/test/query-tests/Summary/LinesOfUserCode.expected

@@ -0,0 +1 @@
+| 11 |

python/ql/test/query-tests/Summary/LinesOfUserCode.qlref

@@ -0,0 +1 @@
+Summary/LinesOfUserCode.ql

python/ql/test/query-tests/Summary/also_python_code

@@ -0,0 +1,7 @@
+#!/usr/bin/env python
+
+# although this is actually Python code, it is not included by the extractor by default.
+
+print("this is also code")
+
+print("but just dummy code")

python/ql/test/query-tests/Summary/my_file.py

@@ -0,0 +1,26 @@
+"""
+module level docstring
+
+is not included
+"""
+# this line is not code
+
+# `tty` was chosen for stability over python versions (so we don't get diffrent results
+# on different computers, that has different versions of Python).
+#
+# According to https://github.com/python/cpython/tree/master/Lib (at 2021-04-23) `tty`
+# was last changed in 2001, so chances of this being changed in the future are slim.
+import tty
+
+s = """
+all these lines are code
+"""
+
+print(s)
+
+def func():
+    """
+    this string is a doc-string. Although the module-level docstring is not considered
+    code, this one apparently is ¯\_(ツ)_/¯
+    """
+    pass

python/ql/test/query-tests/Summary/not_python

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+# Although this is valid python code, it should not be counted as such.
+
+print("foo")