Cloud Run can't connect to MySQL after Google Cloud SQL Maintenance

Google Cloud Database
1

We are seeing a super weird issue regarding MySQL server connectivity from Cloud Run.

Our setup is two separate PHP applications in Cloud Run (one Wordpress, one Laravel).

Today, Google performed maintenance on the SQL server. After the maintenance was performed, the cloud run applications could not connect anymore. There have been no changes to code/env variables or anything.

The Laravel app had somewhat verbose logging and indicated that the connection resulted in access denied, so at that point it sounds more like an auth issue than connectivity.

Checking the Cloud SQL server MySQL error logs shows this to be the case as well:

INFO 2025-04-22T07:01:45.444819Z 2025-04-22T07:01:45.444524Z 686 [Note] [MY-010926] [Server] Access denied for user 'redacteduser'@'cloudsqlproxy~34.34.30.5' (using password: YES)

For every attempt to use the app, another log entry is produced.

We are using “standard” MySQL authentication and users. In other words, not IAM users.

This is the really weird part:

We can “resurrect” connectivity in one of the following ways from our testing:

  • Connect with the affected MySQL user with the exact same credentials, using GCP console “Cloud SQL Studio”.
  • Initiate a connection from our local computers, for example set up a simple PHP script that connects.
  • Deploy a new Cloud Run revision, with NO changes to code/environment variables.

After this, the apps work fine, without redeploying or making any changes whatsoever!

This is not intuitive at all, so I’ll emphasize:
The mere act of connecting from somewhere else seems to “flush” some state somewhere, and from that point on connections are accepted.

The real problem from this is that maintenance may be redirected to “off-hours” and not disturb us/our clients, but it puts the apps in a state of not working, until someone makes a connection manually from somewhere.

Any tips on how to troubleshoot this?

2 Likes
2

Hi @KalleDelorean ,

Welcome to Google Cloud Community!

It sounds like the issue could be related to the state of the Cloud SQL connection or a temporary hiccup caused by the maintenance, especially if you’re seeing “access denied” messages despite no changes in code or environment variables. One potential cause is that after maintenance, Cloud SQL might have had issues properly recognizing or refreshing the connection state, which could explain why connecting externally (from your local machine or Cloud SQL Studio) or redeploying helps “reset” things. I suggest checking the Cloud SQL troubleshooting guide for debugging connection issues, as it can help identify issues like authentication mismatches, firewall rules, or proxy issues.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

1 Like
3

Hello @mcbsalceda ,

This is not an issue regarding my connection setup, it turns out it is only happening on MySQL 8.4, see this thread:

https://www.googlecloudcommunity.com/gc/Databases/Cloud-SQL-DB-access-denied-unless-also-logged-in-to-Cloud-SQL/m-p/862515

Important notes regarding the behavior of GCP regarding this issue:

  1. If you create a new MySQL user, it will NOT be able to connect over Cloud SQL Proxy,. We can regard this user as “dead” currently.
  2. If you initiate a connection via Cloud SQL studio or similar means (custom connection script to the server), you will essentially “revive” the user so it can now connect via Cloud Run for example.
  3. If you run the SQL command “FLUSH PRIVILEGES;” on the server, it will make ALL users “dead”. These can then be revived individually by doing what’s described in step 2.
  4. If GCP performs maintenance on the server, this is EQUIVALENT to running FLUSH PRIVILEGES.

What is the important takeaway here?

Any GCP projects using MySQL 8.4 are ticking timebombs that will detonate at the time of maintenance being done.

DO NOT USE MySQL 8.4!!! I think GCP should hide this option for now until a solution has been implemented.

@mcbsalceda Since you are staff, could you advice me on what to do from here? Is there someone I should email about this?

I can confirm this is an issue that is quite serious, but I cannot persuade my managers to buy an expensive Google Support subscription to simply report what is essentially an error on Googles end (we users cannot mitigate this error).

Should I create a new thread?

5 Likes
4

I have had exactly the same issue and have been pulling my hair out over it.

I can also confirm that after the maintenance update tonight that connections would no longer work from all apps to the Cloud SQL database. To solve the problem, I created a new password - however at that point it still did not work, and I had to login with the new password before the app would connect successfully via cloudproxy.

1 Like
5

Our team has sporadically encountered this issue but we all thought we were going a little mad.

Yesterday evening we had maintenance done on a MySQL 8.4 instance and Cloud Run started experiencing “access denied” errors. @KalleDelorean has done a great job explaining the behavior and the worrisome workarounds required to get users connected again.

Could @mcbsalceda or a Google staffer please provide an update on this issue? This does appear to be a rather serious issue that has consequences for production workflows. Downgrading to MySQL 8 feels like a shoddy solution.

6

I think we need to buy a google cloud support package to be able to contact them properly to ask them to fix their problem that affects us in an unreasonable manner.

Not ideal, but this has been a problem for many months and I don’t think there’s any incentive for them to fix it unless we pay for it somehow.

We will just make sure not to use version 8.4 for any MySQL project continuing, this is also not ideal but at least it doesn’t lead to service outage.

7

I opened a case yesterday and today I heard back. They’re aware of this issue and here’s the Github issue they referenced: Support `caching_sha2_password` authentication plugin with Proxy in Unix Socket mode · Issue #2317 · GoogleCloudPlatform/cloud-sql-proxy · GitHub

Rough timeline for a fix is October 2025.

2 Likes
8

This problem persists and has not yet been resolved. It always occurs after maintenance; after the initial connection via SQL Proxy on my computer, other services like Cloud Run and Compute Engine also reconnect. I’m using MySQL 8.0; it works correctly with version 5.6.

9

I can confirm, this issue persist since some time now. It doesn’t matter much if you are using Cloud Run or Compute Engine, after the Instance maintenance activity, MY-010926 will show up and no connections are being accepted from the SQL Server.

Error started showing up right after the maintenance finished:

logs
logs2228Ă—462 31.1 KB

Maintenance finished:

update
update1944Ă—256 22.6 KB

I have multiple servers using different version, and i can confirm this affects only 8.4, instead 8.0 runs fine after the maintenance/restart.

Speechless :frowning:

10

Running MySQL in google cloud should probably be seen as a pretty niche solution and not for regular use cases.

It is clearly not a high priority from google to have basic functionality like connecting to the MySQL server working reliably.

11

I was also bitten by this error after an automatic maintenance update.

Does this relate to the disabled mysql_native_password as described in the 8.4.x release notes?

I did not find a spot in Cloud SQL to add the mysql_native_password=ON config or command option. Since this was a production instance, the only solution I saw, was to dump the DB and spin up a new 8.0 instance and import the dump.

12

You can just login through cloud sql studio with the credentials, and that will do the trick (if you have multiple users/db, you need to do the same for all).

Screenshot 2026-01-10 at 19.40.46
Screenshot 2026-01-10 at 19.40.461178Ă—1084 61.1 KB

1 Like
14

To clarify if anyone reads your comment and having a takeaway that this is a solution to anything, be aware that it is not.

As soon as maintenance happens on the server or privileges are flushed due to any other reason, your connections will start failing again.

1 Like
15

To any mods on the forum, please watch out for this weird hacking attempt via pdf

edit: thank you for removing that malicious comment, mods. Might want to acknowledge that somehow so my own comment doesn’t seem out of place.
Adding this edit to fix that for you, so you can spend team with the team working on a solution on the main issue with MySQL being broken since v8.4.

16

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.