Merge pull request #3953 from RasmusWL/python-more-call-graph-tracing

Approved by tausbn

Author: codeql-ci

python/tools/recorded-call-graph-metrics/.flake8

@@ -0,0 +1,6 @@
+# As described in https://github.com/psf/black/blob/master/docs/compatible_configs.md#flake8
+# and https://black.readthedocs.io/en/stable/the_black_code_style.html#line-length
+[flake8]
+max-line-length = 88
+select = C,E,F,W,B,B950
+ignore = E203, E501, W503

python/tools/recorded-call-graph-metrics/.gitignore

@@ -0,0 +1,13 @@
+# Example DB
+cg-trace-example-db/
+
+# Tests artifacts
+tests/python-traces/
+tests/cg-trace-test-db
+
+# Artifact from building `pip install -e .`
+src/cg_trace.egg-info/
+
+projects/
+
+venv/

python/tools/recorded-call-graph-metrics/.isort.cfg

@@ -0,0 +1,6 @@
+[settings]
+multi_line_output = 3
+include_trailing_comma = True
+force_grid_wrap = 0
+use_parentheses = True
+line_length = 88

python/tools/recorded-call-graph-metrics/README.md

@@ -4,14 +4,113 @@ also known as _call graph tracing_.
 
 Execute a python program and for each call being made, record the call and callee. This allows us to compare call graph resolution from static analysis with actual data -- that is, can we statically determine the target of each actual call correctly.
 
-This is still in the early stages, and currently only supports a very minimal working example (to show that this approach might work).
+Using the call graph tracer does incur a heavy toll on the performance. Expect 10x longer to execute the program.
 
-The next hurdle is being able to handle multiple calls on the same line, such as
+Number of calls recorded vary a little from run to run. I have not been able to pinpoint why.
 
-- `foo(); bar()`
-- `foo(bar())`
-- `foo().bar()`
+## Running against real projects
 
-## How do I give it a spin?
+Currently it's possible to gather metrics from traced runs of the standard test suite of a few projects (defined in [projects.json](./projects.json)): `youtube-dl`, `wcwidth`, and `flask`.
 
-Run the `recreate-db.sh` script to create the database `cg-trace-example-db`, which will include the `example/simple.xml` trace from executing the `example/simple.py` code. Then run the queries inside the `ql/` directory.
+To run against all projects, use
+
+```bash
+$ ./helper.sh all $(./helper.sh projects)
+```
+
+To view the results, use
+```
+$ head -n 100 projects/*/Metrics.txt
+```
+
+### Expanding set of projects
+
+It should be fairly straightforward to expand the set of projects. Most projects use `tox` for running their tests against multiple python versions. I didn't look into any kind of integration, but have manually picked out the instructions required to get going.
+
+As an example, compare the [`tox.ini`](https://github.com/pallets/flask/blob/21c3df31de4bc2f838c945bd37d185210d9bab1a/tox.ini) file from flask with the configuration
+
+```json
+    "flask": {
+        "repo": "https://github.com/pallets/flask.git",
+        "sha": "21c3df31de4bc2f838c945bd37d185210d9bab1a",
+        "module_command": "pytest -c /dev/null tests examples",
+        "setup": [
+            "pip install -r requirements/tests.txt",
+            "pip install -q -e examples/tutorial[test]",
+            "pip install -q -e examples/javascript[test]"
+        ]
+    }
+```
+
+## Local development
+
+### Setup
+
+1. Ensure you have at least Python 3.7
+
+2. Create virtual environment `python3 -m venv venv` and activate it
+
+3. Install dependencies `pip install -r --upgrade requirements.txt`
+
+4. Install this codebase as an editable package `pip install -e .`
+
+5. Setup your editor. If you're using VS Code, create a new project for this folder, and
+   use these settings for correct autoformatting of code on save:
+  ```
+  {
+      "python.pythonPath": "venv/bin/python",
+      "python.linting.enabled": true,
+      "python.linting.flake8Enabled": true,
+      "python.formatting.provider": "black",
+      "editor.formatOnSave": true,
+      "[python]": {
+          "editor.codeActionsOnSave": {
+              "source.organizeImports": true
+          }
+      },
+      "python.autoComplete.extraPaths": [
+          "src"
+      ]
+  }
+  ```
+
+6. Enjoy writing code, and being able to run `cg-trace` on your command line :tada:
+
+### Using it
+
+After following setup instructions above, you should be able to reproduce the example trace by running
+
+```
+cg-trace --xml example/simple.xml example/simple.py
+```
+
+You can also run traces for all tests and build a database by running `tests/create-test-db.sh`. Then run the queries inside the `ql/` directory.
+
+## Tracing Limitations
+
+### Multi-threading
+
+Should be possible by using [`threading.setprofile`](https://docs.python.org/3.8/library/threading.html#threading.setprofile), but that hasn't been done yet.
+
+### Code that uses `sys.setprofile`
+
+Since that is our mechanism for recording calls, any code that uses `sys.setprofile` will not work together with the call-graph tracer.
+
+### Class instantiation
+
+Does not always fire off an event in the `sys.setprofile` function (neither in `sys.settrace`), so is not recorded. Example:
+
+```
+r = range(10)
+```
+
+when disassembled (`python -m dis <file>`):
+
+```
+  9          48 LOAD_NAME                7 (range)
+             50 LOAD_CONST               5 (10)
+             52 CALL_FUNCTION            1
+             54 STORE_NAME               8 (r)
+```
+
+but no event :disappointed: