VDK Structlog Integration

Use Structlog in vdk-core, plugins and data jobs

If we make structlog a dependency of vdk-core, it raises two follow-up questions.

• how do we make sure plugins use the correct logger
• how do we make sure user code uses the correct logger

The naive solution is to use structlog as is in vdk-core code and pass it inside a wrapper for plugins and use code. For plugins, the wrapper can be inside CommonContext. For user code, we can expose an object instance, similar to job_input and have user logs inside data jobs be done through the custom object instance.

PRO: Simple and straightforward, doesn't add too much complexity to our existing code base.

CON: There is absolutely no guarantee that users and plugin devs will use the objects we provide and not opt for logging.getLogger().

CON: This is too much of a breaking change for existing setups. We would have to create the configuration mechanism as well and ship the two features together.

Use structlog for vdk-core, use native logging for plugins and user code

https://www.structlog.org/en/stable/api.html#structlog.stdlib.render_to_log_kwargs https://www.structlog.org/en/stable/standard-library.html#rendering-using-logging-based-formatters

We can use structlog in vdk-core and benefit from the build-in configuration system, event dicts, etc. At the same time, we can do the formatting with regular logging by writing and passing our own formatters, or already existing ones, e.g. JSON. This allows plugins and user code to still use logging.getLogger()

PRO: Ensures that users will always benefit from structlog, e.g. they can use structlog IF they want to, for example if plugins want to add to the event dict, but still get some of the benefits if they don't.

CON: Structlog is still a dependency of core

CON: We potentially have to write our own formatters

Create a vdk-structlog plugin and use native logging for vdk-core, plugins and user code

This approach is similar to the above one, but exports all the structlog configuration to a vdk-plugin. We can create the plugin that adds structlog and then use native logging in the whole vdk-codebase.

PRO: Structlog is not a dependency of vdk-core

PRO: Ensures users will always benefit from structlog

CON: Hard to pass values in event dicts outside of configuration, e.g. we can add stuff in the processor chain, but for passing key-value pairs on the fly, we would have to somehow fetch the extra params, get the event dict and add to it.

CON: Requires POC, not sure if config will work as a plugin