Skip to content

Shared JavaScript config file is only applied to first action #2051

@BenJackGill

Description

@BenJackGill

I'm trying to share some config properties via a centralised JavaScript file.

I write some centralised settings into configs.js and spread it into each sqlx config as needed.

Environment: Dataform core (3.0.39), BigQuery

Reproduction (minimal):

  1. In includes/configs.js, export a shared config with partitioning:
const cleanTable = {
  type: "incremental",
  schema: "clean",
  bigquery: {
    partitionBy: "DATE(event_ts)",
    requirePartitionFilter: true,
  },
};
module.exports = { cleanTable };
  1. Define two "clean" table models that spread the shared config:

definitions/clean/foo.sqlx

config {
  ...configs.cleanTable,
  name: "foo"
}
select 1 as event_ts;

definitions/clean/bar.sqlx

config {
  ...configs.cleanTable,
  name: "bar"
}
select 1 as event_ts;
  1. Run dataform run --full-refresh.

Expected:

Both clean.foo and clean.bar are created partitioned on event_ts with requirePartitionFilter: true.

Actual:

Only the first table created (foo) is partitioned; the second (bar) is created without partitioning/requirePartitionFilter.

Question:

Is there a "more correct" way to share config setting between sqlx files?

It would be very helpful to have a way of sharing standardised configs across similar tables. Can it be fixed so that it works?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions