Is it possible to set priority of dataform execution to BATCH

Google Cloud Data Analytics
2

While Dataform doesn’t have a direct setting for default BigQuery priority, you can achieve this through configuration:

1. Edit dataform.json Configuration:

Open your dataform.json file and add or modify the defaultConfig section:

{
  "defaultConfig": {
    "bigquery": {
      "priority": "BATCH"
    }
  }
}

This sets BATCH as the default priority for all BigQuery operations in your Dataform project.

2. Configure in config Block (Granular Control):

For specific SQLX scripts, you can set the priority directly in the config block:

config {
  type: "table",
  bigquery: {
    priority: "BATCH"
  }
}

-- Your SQL code here

This allows you to control which scripts use BATCH priority.

Considerations:

  • Performance: BATCH priority is slower, designed for less time-sensitive workloads.

  • Cost: BATCH can be more cost-effective for large queries.

Potential Workarounds:

  • Custom Profiles: Create Dataform profiles with BATCH priority set, using --profile when running.

  • Pre/Post Operations: Use scripts to adjust priority before or after execution (requires scripting).

  • Feature Request: Suggest this feature to the Dataform team for future releases.