Dataform test cannot see the dataset. But the dataset is actually exist.

Google Cloud Data Analytics
1

To make the question simple. I create a dataset call test_dataset. And I can see it inside Bigquery after I created it.

Then I wrote the test of Dataform, like this:

config {
type: “test”,
dataset: “test_dataset”
}
The error is:
Dataset test_dataset could not be found.
If I use type: “table”, it can be found.
It should be straghtforward, but not. Can you help me with this? Thanks.

2

Hi @hao-wang ,

Welcome to Google Cloud Community!

The error “Dataset test_dataset could not be found” during a Dataform test execution indicates that Dataform is unable to locate the specified dataset when running the test. This typically happens due to a misunderstanding of how Dataform handles datasets during testing versus during the creation of tables/views.

Here are some approaches that you may try:

  • Ensure that the test_dataset is in the same location as your Dataform project. BigQuery datasets are location-specific, and mismatched locations can cause errors.
  • Verify that the service account or credentials used by Dataform have the necessary permissions to access the test_dataset. Specifically, it should have bigquery.datasets.get and bigquery.tables.get permissions.
  • Double-check your Dataform configuration file dataform.json to ensure that the default location matches the location of your dataset. If there’s a mismatch, update the defaultLocation field in the configuration.
  • The "test" type in Dataform might have specific requirements or limitations. If it works with "table", it could indicate that "test" is not fully supported for your use case.
  • Ensure that the dataset name is correctly referenced in your Dataform script. If the dataset is in a different project, you may need to include the project ID.

Feel free to explore the following documentations for more guidance:

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

3

Hi, Thanks for the reply!

  1. Both the dataset and Dataform project are in same location (i.e. us-east1)

  2. The service account 'dataform@ava-backend.iam.gserviceaccount.com’ has role Bigquery Data Editor role. So it should able to access datasets in the GCP project.

  3. The dataform.json file looks like:

{
“defaultDatabase”: “ava-backend”,
“defaultLocation”: “us-east1”,
“warehouse”: “bigquery”,
“defaultSchema”: “dataform”,
“assertionSchema”: “dataform_assertions”
}
4. The dataset is in the same GCP project with the Dataform project
Is there anything that I m missing?

BR,

Hao