We are facing an issue with cloud spanner and spring cloud spanner in a spring boot microservice deployed in GKE.
Here, I am trying to explain the issue-
I have two projects say-
gke-project
microservice-project
I have enabled spanner api, created spanner instance and database under microservice-project
created a java microservice using spring cloud spanner and deployed into GKE cluster under gke-project.
Now, the problem is; the microservice always referring gke-project and asking to enable cloud spanner api but we have created spanner instance in microservice-project and already enabled cloud spanner api there.
For your information-
I have assigned proper IAM policy (spanner admin role) to gke service account and defined below configuration in microservice
It appears that you have already configured the spring.cloud.gcp.spanner.project-id property in your Spring Boot application to point to microservice-project. This should indeed direct the Spring Cloud Spanner client to use the microservice-project for all its operations.
However, since you are still facing the issue, it may be related to other factors such as the service account permissions, credentials configuration, or potential inconsistencies in the property naming.
Verify Property Name: Ensure that the property name is correct. The correct property name is spring.cloud.gcp.spanner.project-id, not spring.cloud.gcp.spanner.project. Make sure there are no typos or extra spaces in the property name or value.
Service Account and Permissions: Double-check the service account permissions and make sure that the GKE cluster in gke-project is using a service account with the necessary permissions to access the Spanner instance in microservice-project.
Credentials Configuration: Verify the credentials configuration as mentioned in the previous response. You may need to explicitly specify the path to the service account key file.
Logging and Debugging: Enable detailed logging to get more insights into what might be going wrong.
Thanks for reply. I already checked all the configuration, settings, permissions, credentials multiple times and all are correct.
One strange thing which I observed that when I enable Cloud Spanner API in gke-project everything started working fine. There is no more issue, and all the data are saving properly into spanner under microservice-project. Looks like there is some problem with Spanner configuration and Health indicator because it is looking for Spanner API in gke-project. do you have any workaround to solve this issue without enabling Cloud Spanner API ingke-project?
It’s indeed strange behavior that enabling the Cloud Spanner API in the gke-project resolved the issue, even though your Spanner instance is in the microservice-project. This could be indicative of an underlying issue with how the projects and permissions are being resolved.
Here are a few potential workarounds and additional checks you might consider:
Explicitly Specify Credentials: If you haven’t already, try explicitly specifying the service account credentials for the microservice-project in your Spring Boot application. This can be done using the spring.cloud.gcp.credentials.location property, pointing to the JSON key file for the service account with the necessary permissions.
Customize Health Indicator: If the issue is specifically related to the health indicator, you might consider customizing or disabling the Spanner health indicator in your Spring Boot application. This can be done by adding the following property to your application.properties file: management.health.spanner.enabled=false
This will disable the Spanner health indicator, which might bypass the issue if it’s specifically related to the health check.
Use Different Service Accounts: Consider creating a new service account in the microservice-project with the necessary permissions and configuring your GKE cluster and Spring Boot application to use this new service account. This might help isolate the issue if it’s related to a specific service account configuration.
Check Environment Variables: Verify that there are no conflicting environment variables that might be affecting the Google Cloud SDK’s default project or credentials. This includes checking both the GKE cluster’s environment variables and any environment variables set in your Spring Boot application.
Upgrade Libraries: Ensure that you are using the latest versions of the Spring Cloud GCP and Google Cloud client libraries. Sometimes, issues like this can be related to bugs in specific versions of the libraries, and upgrading to the latest versions might resolve the issue.