BigQuery Sink Connector Configuration Errors

Google Cloud Data Analytics
1

I am trying to ingest data from MySQL to BigQuery. I am using Debezium components running on Docker for this purpose.
Anytime I try to deploy the BigQuery sink connector to Kafka connect, I am getting this error:

{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nFailed to construct GCS client: Failed to access JSON key file\nAn unexpected error occurred while validating credentials for BigQuery: Failed to access JSON key file\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}

It shows it’s an issue with the service account key.

I granted the service account BigQuery Admin and Editor permissions, but the error persists.

This is my BigQuery connector configuration file:

{
    "name": "kcbq-connect1",
    "config": {
      "connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
      "tasks.max" : "1",
      "topics" : "kcbq-quickstart1",
      "sanitizeTopics" : "true",
      "autoCreateTables" : "true",
      "autoUpdateSchemas" : "true",
      "schemaRetriever" : "com.wepay.kafka.connect.bigquery.retrieve.IdentitySchemaRetriever",
      "schemaRegistryLocation":"http://localhost:8081",
      "bufferSize": "100000",
      "maxWriteSize":"10000",
      "tableWriteWait": "1000",
      "project" : "dummy-production-overview",
      "defaultDataset" : "debeziumtest",
      "keyfile" : " /Users/Oladayo/Desktop/Debezium-Learning/key.json"
    }
}

Can anyone help?

Thank you

2

@Oladayo

Just a doubt , there is space at the start of “keyfile” attribute value at the beginning of the path " /user/***" is that intended?

3

It’s a mistake. It didn’t work when I removed the space either.

My other question is.. since I am running on docker, do I have to mount the file in docker and specify the docker path rather the local path?

4

Hi all, I just want to say that I had to mount the service account key to the Kafka connect container and then specified the path in the container rather than the local path of the service account key.