Cloud Run to Cloud SQL (Postgres) - Persistent "password authentication failed"

Google Cloud Database
1

Hello,

I’m seeking help with a persistent database connection issue between a Cloud Run service and a Cloud SQL for PostgreSQL instance that I have been unable to solve after extensive troubleshooting.

The Problem

My Python/Flask application, deployed on Cloud Run, fails to connect to my Cloud SQL for PostgreSQL database.

  • The application log shows the connection fails with: (psycopg2.OperationalError) connection to server on socket … failed: FATAL: password authentication failed for user “cloudrunuser”.

  • The Cloud SQL database log shows the corresponding rejection: FATAL: password authentication failed for user “cloudrunuser”.

Troubleshooting Steps Performed

We have systematically worked through every potential cause and have confirmed the following:

  1. Correct User Type: The cloudrunuser is a standard SQL user with a password, not an IAM database user.

  2. Correct Permissions: The cloudrunuser has been made the owner of all tables in the database using REASSIGN OWNED and can successfully see all tables during a manual login.

  3. Correct Connection Method: The application correctly uses the secure Unix socket for the connection, which is confirmed by host=[local] in the database logs.

  4. Clean Builds: Every deployment is built using gcloud builds submit --no-cache to ensure the latest code is always used.

  5. Removed Secret Manager Override: We discovered an early issue where the deployment was configured to use a password from Secret Manager. We have since deployed multiple times using the –remove-secrets=DB_PASS flag to ensure this override is no longer active.

  6. Application-Level Debugging: I have added a diagnostic test directly into my application’s startup code (app/init.py) to verify the credentials it’s using.

The Core Contradiction

This is the part I cannot solve. The logs from my most recent deployment show a direct contradiction:

  • Fact 1: My app log proves it has the correct password. My diagnostic code prints the password it is about to use, and the log confirms it is correct (e.g., Password starts with: ‘XXXX’).

  • Fact 2: My app log proves the connection is rejected. My diagnostic code then catches the database error and prints my custom message: — :cross_mark: DATABASE CONNECTION FAILED: PASSWORD REJECTED :cross_mark: —

  • Fact 3: My manual test with the exact same password succeeds. When I log in to the database manually from Cloud Shell (psql) as cloudrunuser and type the exact same password that is in my code, I can connect successfully and see all my tables.

My question is: How is this possible? How can the database reject a password from the application, when the application log shows it is using the correct password, and a manual test with that same password works?

Any insight into what could cause this discrepancy between a manual psql connection and a psycopg2 connection from within the Cloud Run environment would be greatly appreciated.

Thank you.

2

Hi @RatVibeCoder ,

Welcome to Google Cloud Community!

Since the password works manually but fails in Cloud Run, it’s likely an environment issue, possibly a hidden newline or space in the password from an env var or Secret Manager. Try logging the password using repr() or [ord(c) for c in password] to spot anything weird. Also, rotating to a new simple password (just letters and numbers) might be a quick way to rule that out. Lastly, note that some drivers need specific parameters for Unix socket connections. For example, psycopg2 uses the path set to host to connect properly.

For reference, here’s the official Cloud SQL guide for connecting from Cloud Run.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

3

Thanks @mcbsalceda . I did follow the steps but unfortunately the error remain the same. I have switched now to IAM and its working fine