Pipelines as a code (pipeline configuration files)

[saul-data]

Background

For Devops / Dataops best practice, pipeline code should be saved and edited as configuration files. This allows for version control in Git and to facilitate import / export of code between local development through to production.

Suggestions

1. Design a pipeline configuration file as either json or yaml
2. API to import and export pipeline configuration file
3. Commit pipeline files to git

[northcode7]

Maybe it is also interesting to add import/export functionality? So you could export a created datapipeline (configuration files and python files).

[saul-data]

Yes agreed! I am busy looking into to it. We had to implement an algorithm to position the nodes in the graph because if you implement data pipelines as code, it won’t know the node positions but we are getting closer to this :)