DataStream unable to start due to BigQuery connection profile

Google Cloud Serverless Applications
1

I use pulumi as IaaS and yesterday we migrated from GCP’s v6 to v7 and now the stream is unable to start.

This stream is running for almost 6 months syncing data from CloudSQL Postgres to Big Query and never had a problem.

When I try to start the stream I get the following:

The stream is connected to non-existent connectionProfile

I tried to debug it through Logs Explorer and there is no more information than that.

After the pulumi update, all the steps to create the stream have passed. Based on the GCP’s operation cli response:

validationResult:> validations:> - code: POSTGRES_VALIDATE_CONNECTIVITY> description: Validates that Datastream can connect to the source Postgres database.> state: PASSED> - code: POSTGRES_VALIDATE_PUBLICATION> description: Validates that the publication exists and configured for the required> tables.> state: PASSED> - code: POSTGRES_VALIDATE_LOGICAL_DECODING> description: Validates that logical decoding is properly configured on the database.> state: PASSED> - code: POSTGRES_VALIDATE_REPLICATION_SLOT> description: Validates that the replication slot exists and not lost.> state: PASSED> - code: POSTGRES_VALIDATE_BACKFILL_PERMISSIONS> description: Validates that the given Postgres user has the select permissions> on the required customer tables> state: PASSED> - code: BIGQUERY_VALIDATE_API_ENABLED> description: Validates that the BigQuery API is enabled in the current project.> state: PASSED> - code: BIGQUERY_VALIDATE_DESTINATION_PERMISSIONS> description: Validates that Datastream has permissions to write data into BigQuery.> state: PASSED> - code: BIGQUERY_VALIDATE_DYNAMIC_DATASET_LOCATION> description: Validates that Datastream can write data into BigQuery datasets> in the specified location.> state: PASSED> verb: create

I checked the IAM policies and all the BigQuery-related permissions are granted to the DataStream service account. Reference: https://www.googlecloudcommunity.com/gc/Serverless/Unable-to-connect-datastream-to-a-bigquery-destination-in/m-p/646046

I also checked the KMS key for BQ encryption/decryption and the same assigned on the BigQuery dataset is assigned on the DataStream destination configuration.

How can I overcome this issue?

1 Like
2

It seems like the issue you’re encountering is related to the connection profile for your Datastream stream not being recognized or available after migrating from GCP’s v6 to v7. Here are some steps you can take to troubleshoot and potentially resolve the issue:

  • Verify Connection Profile Configuration: Double-check the configuration of your connection profile for the Datastream stream. Ensure that it is correctly configured to connect to your CloudSQL Postgres database and BigQuery destination. Pay special attention to any changes that might have been introduced during the migration process.
  • Recreate Connection Profile: If you suspect that the connection profile might have been corrupted or lost during the migration, try recreating it. Make sure to follow the correct steps and configuration parameters.
  • Check Datastream Configuration: Review the configuration settings for your Datastream stream. Ensure that all necessary configurations, such as source and destination settings, are correctly set up. It’s possible that there might be a misconfiguration causing the stream to fail to start.
  • Review Logs: Although you mentioned that you didn’t find much information in Logs Explorer, it’s still worth reviewing all available logs for any potential error messages or clues as to what might be causing the issue. Look for any errors or warnings related to the stream startup process.
  • Test Connectivity: Manually test the connectivity between your CloudSQL Postgres database and BigQuery destination using tools like psql or bq command-line utilities. This can help verify that there are no network or authentication issues preventing Datastream from connecting to either source or destination.
1 Like