System error when starting to transfer asset [db/schema/table], try again later.

Google Cloud Database
1

I want to transfer Cloud SQL for PostgreSQL data to BigQuery.
So, I am setting BigQuery Data Transfer Service.
https://cloud.google.com/bigquery/docs/salesforce-transfer
Premise
Cloud SQL does not allow 0.0.0.0/0

resource "google_sql_database_instance" "postgres" {
  database_version = "POSTGRES_14"
  
  settings {
    ip_configuration {
      ipv4_enabled    = var.public_access_enabled
      private_network = var.vpc_id
      ssl_mode        = "ENCRYPTED_ONLY"
      
      authorized_networks {
        name  = "internal"
        value = "company_ip"
      }
    }
  }
}

Description
BigQuery Data Transfer Service has below error.
「Transfer run failed with system error code FAILED_PRECONDITION.」
「“System error when starting to transfer asset [db/schema/table], try again later.”」

resource "google_bigquery_dataset" "cloud_sql_to_bigquery" {
  dataset_id = "cloud_sql_to_bigquery"
  location   = "asia-northeast1"
  
  access {
    role          = "OWNER"
    user_by_email = "terraform@project.iam.gserviceaccount.com"
  }
  
  access {
    role   = "READER"
    domain = "comany_domain"
  }
}

resource "google_bigquery_data_transfer_config" "cloud_sql_to_bigquery" {
  display_name           = "cloud_sql_to_bigquery"
  data_source_id         = "postgresql"
  location               = "asia-northeast1"
  schedule               = "every 1 hours"
  destination_dataset_id = google_bigquery_dataset.cloud_sql_to_bigquery.dataset_id
  
  params = {
    "connector.endpoint.host"           = "private_ip"
    "connector.endpoint.port"           = "5432"
    "connector.database"                = var.TF_POSTGRES_DATABASE
    "connector.authentication.username" = var.TF_POSTGRES_USER
    "connector.authentication.password" = var.TF_POSTGRES_PASSWORD
    "connector.schema"                  = "public"
    "connector.networkAttachment"       = "projects/.../regions/asia-northeast1/networkAttachments/..."
    "assets"                           = jsonencode([
      "${var.TF_POSTGRES_DATABASE}/public/table_name"
    ])
  }
}

Additional explanation
Before adding 「connector.networkAttachment」and
connector.endpoint.host is public_ip, BigQuery Data Transfer Service has below error.

「INVALID_ARGUMENT: The connection attempt failed.」
「Invalid datasource configuration provided when starting to transfer asset [db/schema/table] INVALID_ARGUMENT: The connection attempt failed.」

Images

スクリーンショット 2025-07-09 19.43.48.png

スクリーンショット 2025-07-09 19.36.46.png

スクリーンショット 2025-07-09 19.48.34.png
スクリーンショット 2025-07-09 19.48.34.png665×515 69.4 KB

So, i think not network error.Thank for watching the post.

2

Hi @imaharu,

Welcome to Google Cloud Community!

The errors you encountered may indicate network configuration issues or missing roles/permissions required for the BigQuery Data Transfer Service. Please ensure that you have configured the necessary roles and permissions to enable the transfer of Cloud SQL for PostgreSQL data to BigQuery. For network configuration, SSL-enabled connection with EncryptionMode set to FULL is currently not supported by the connector for Cloud SQL PostgreSQL instances. As a workaround, you can use a Private IP with EncryptionMode set to DISABLE for a more secure connection. You can also track this issue via the public issue tracker, which will allow customers to connect to their instances with EncryptionMode set to FULL. Please note that I can’t provide any details or timelines at this moment. For future updates, please keep an eye on the release notes for any updates or new features related to BigQuery Data Transfer Service.

Additionally, the load PostgreSQL data into BigQuery is currently in Preview, which means it is still under development, and may not yet offer the expected quality, and might have limited support. Preview features often come with certain limitations and might not provide the full range of quality or functionality available in the final product. It may have bugs or unexpected behaviors. However, you can expect the quality to improve as the feature matures.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

3

Hi, marckevin.

Thank you for replying.

After I allow non ssl connection in cloud sql, transfer succeed.

I’m looking forward to Google support ssl connection in BigQuery Data Transfer Service.