When designing multi-region, cross-project Disaster Recovery (DR) or data distribution architectures on Google Cloud, Cloud SQL External Server Replication (using a Source Representation instance and a Cloud SQL Read Replica) allows you to replicate across independent projects and VPCs. However, combining Private Services Access (PSA) with Cloud SQL for MySQL introduces three critical infrastructure challenges that must be addressed for replication to succeed:
-
1. Non-Transitive VPC Peering Across PSA Networks (i/o timeout): Both the Source Cloud SQL instance and the Replica Cloud SQL instance run inside Google-managed Service Producer VPCs peered to your customer VPCs. If your customer VPCs are connected via VPC Network Peering, traffic must cross three peering boundaries—which Google Cloud drops silently. By default, you should connect the VPCs via HA VPN with BGP custom route advertisements so no proxy VMs are required. Only when HA VPN is unavailable should you fall back to a Two-Hop TCP Proxy VM chain.
-
2. MySQL Authentication & TLS (Error 1045: Access denied): MySQL permanently removed the legacy mysql_native_password plugin and defaults to caching_sha2_password. Over an unencrypted connection, caching_sha2_password requires an RSA public-key exchange that fails during external sync verification unless the source’s Server CA certificate is attached to negotiate TLS—returning a misleading Error 1045 (using password: YES) even with a valid password.
-
3. Version & Edition Constraints (INCOMPATIBLE_DATABASE_VERSION): Cloud SQL for MySQL defaults to the ENTERPRISE_PLUS edition (which rejects db-custom-* tiers) and enforces strict major version parity across both the source representation and the replica.
1. The Networking Challenge: Why Direct VPC Peering Fails
When a Cloud SQL instance uses Private IP via Private Services Access (PSA), it resides inside a Google-managed Service Producer VPC (servicenetworking) peered to your customer VPC. If your Source VPC ($SOURCE_VPC) and Replica VPC ($REPLICA_VPC) are connected via VPC Network Peering, the path between the two databases spans three peering hops:
Figure 1: Direct traffic between Cloud SQL instances across peered VPCs spans three peering boundaries and is blackholed due to non-transitive VPC peering.
Because Google Cloud VPC Network Peering is strictly non-transitive, a VPC only exports its own local subnet routes to its PSA Service Producer network—never routes learned from another peered VPC. Pointing the Source Representation directly at $SOURCE_DB_PRIVATE_IP over VPC Peering results in: dial tcp <SOURCE_PSA_IP>:3306: i/o timeout.
2. Choosing Your Connectivity Architecture: HA VPN (Default) vs. Proxy VMs (Fallback)
Option A (Default & Recommended): Direct Connectivity via HA VPN + BGP Custom Routes
When connecting $SOURCE_VPC and $REPLICA_VPC using HA VPN and Cloud Router (BGP), the link between your two customer VPCs is a routed BGP boundary rather than a VPC Peering hop. Consequently, each path only crosses a single PSA peering hop on either side:
Figure 2 (Default Architecture): HA VPN with BGP Custom Route advertisements enables direct PSA-to-PSA routing between the Cloud SQL Replica and Source Cloud SQL instance without any Proxy VMs.
-
Zero Proxy VMs Required: Cloud Router in $SOURCE_VPC advertises $SOURCE_PSA_CIDR over BGP to $REPLICA_VPC, and Cloud Router in $REPLICA_VPC advertises $REPLICA_PSA_CIDR back to $SOURCE_VPC.
-
PSA Custom Route Export/Import: By enabling --export-custom-routes and --import-custom-routes on the servicenetworking-googleapis-com peering in both VPCs, each Google-managed Cloud SQL producer network learns the remote PSA route natively. The Source Representation’s hostPort points directly to $SOURCE_DB_PRIVATE_IP:$DB_PORT.
Option B (Fallback Only When HA VPN is Not Available): Two-Hop HAProxy TCP Proxy Chain
If organizational constraints, existing network design, or dev/test budget limits prevent deploying HA VPN, and you must rely on existing VPC Network Peering between $SOURCE_VPC and $REPLICA_VPC, you can bridge the non-transitive peering gap using two lightweight HAProxy VMs placed in the native subnets of each VPC:
Figure 3 (Fallback Architecture): Two-Hop HAProxy TCP Forwarding Chain used only when HA VPN is unavailable.
-
Hop 1 ($HOP1_PROXY_NAME in $REPLICA_VPC): Receives traffic from the Cloud SQL Replica across Peering #1 (since it sits in a native subnet of $REPLICA_VPC) and forwards TCP packets across the VPC peering link to Hop 2 ($HOP2_PROXY_IP:$DB_PORT).
-
Hop 2 ($HOP2_PROXY_NAME in $SOURCE_VPC): Receives traffic from Hop 1 in a native subnet of $SOURCE_VPC and forwards TCP packets across Peering #3 to the Source Cloud SQL PSA IP ($SOURCE_DB_PRIVATE_IP:$DB_PORT).
| Criteria | Option A: HA VPN + BGP (Default / Recommended) | Option B: Two-Hop TCP Proxy VMs (Fallback) |
|---|---|---|
| When to Use | Default choice for production & enterprise cross-VPC deployments. | Fallback only when HA VPN cannot be provisioned and VPC Peering must be used. |
| Proxy VMs Needed | None (0 VMs) — 100% cloud-native BGP routing. | 2 VMs (Hop 1 in $REPLICA_VPC + Hop 2 in $SOURCE_VPC). |
| Source Representation hostPort | Direct Source PSA IP ($SOURCE_DB_PRIVATE_IP:$DB_PORT). | Hop 1 Proxy VM IP ($HOP1_PROXY_IP:$DB_PORT). |
| High Availability | Built-in 99.99% SLA with active/active HA VPN tunnels. | Requires Regional MIG + Internal NLB on both hops to avoid single-VM SPOF. |
4. Step-by-Step Implementation Using gcloud
Step 1: Retrieve Source Database Private IP
export SOURCE_DB_PRIVATE_IP=$(gcloud sql instances describe "${SOURCE_INSTANCE_NAME}" \\
--project="${SOURCE_PROJECT_ID}" \\
--format="value(ipAddresses.filter(type:PRIVATE).ipAddress)")
echo "Source Cloud SQL Private IP: ${SOURCE_DB_PRIVATE_IP}"
Step 2A (Default Architecture): Configure HA VPN & PSA Custom Route Exchange (No Proxy VMs)
By default, use HA VPN with BGP custom route advertisements so the Cloud SQL Replica can reach $SOURCE_DB_PRIVATE_IP directly. (If HA VPN is already provisioned between your VPCs, simply run parts 1 and 2 below to ensure the PSA CIDR ranges are advertised and exported).
1. Enable Custom Route Export & Import on PSA Peerings in BOTH Projects
gcloud compute networks peerings update "${PSA_PEERING_NAME}" \\
--network="${SOURCE_VPC}" \\
--project="${SOURCE_PROJECT_ID}" \\
--export-custom-routes \\
--import-custom-routes
gcloud compute networks peerings update "${PSA_PEERING_NAME}" \\
--network="${REPLICA_VPC}" \\
--project="${REPLICA_PROJECT_ID}" \\
--export-custom-routes \\
--import-custom-routes
2. Create Cloud Routers Advertising Local Subnets + PSA CIDRs via BGP
gcloud compute routers create "${SOURCE_ROUTER_NAME}" \\
--project="${SOURCE_PROJECT_ID}" \\
--network="${SOURCE_VPC}" \\
--region="${SOURCE_REGION}" \\
--asn="${SOURCE_ROUTER_ASN}" \\
--advertisement-mode=CUSTOM \\
--set-advertisement-groups=ALL_SUBNETS \\
--set-advertisement-ranges="${SOURCE_PSA_CIDR}"
gcloud compute routers create "${REPLICA_ROUTER_NAME}" \\
--project="${REPLICA_PROJECT_ID}" \\
--network="${REPLICA_VPC}" \\
--region="${REPLICA_REGION}" \\
--asn="${REPLICA_ROUTER_ASN}" \\
--advertisement-mode=CUSTOM \\
--set-advertisement-groups=ALL_SUBNETS \\
--set-advertisement-ranges="${REPLICA_PSA_CIDR}"
3. Create HA VPN Gateways (if not already deployed)
gcloud compute vpn-gateways create "${SOURCE_VPN_GW}" \\
--project="${SOURCE_PROJECT_ID}" \\
--network="${SOURCE_VPC}" \\
--region="${SOURCE_REGION}"
gcloud compute vpn-gateways create "${REPLICA_VPN_GW}" \\
--project="${REPLICA_PROJECT_ID}" \\
--network="${REPLICA_VPC}" \\
--region="${REPLICA_REGION}"
4. Create HA VPN Tunnels (Interface 0 & 1) Between Source and Replica Gateways
for IFACE in 0 1; do
gcloud compute vpn-tunnels create "tunnel-source-to-replica-${IFACE}" \\
--project="${SOURCE_PROJECT_ID}" \\
--region="${SOURCE_REGION}" \\
--vpn-gateway="${SOURCE_VPN_GW}" \\
--interface="${IFACE}" \\
--peer-gcp-gateway="projects/${REPLICA_PROJECT_ID}/regions/${REPLICA_REGION}/vpnGateways/${REPLICA_VPN_GW}" \\
--router="${SOURCE_ROUTER_NAME}" \\
--ike-version=2 \\
--shared-secret="${VPN_SHARED_SECRET}"
gcloud compute vpn-tunnels create "tunnel-replica-to-source-${IFACE}" \\
--project="${REPLICA_PROJECT_ID}" \\
--region="${REPLICA_REGION}" \\
--vpn-gateway="${REPLICA_VPN_GW}" \\
--interface="${IFACE}" \\
--peer-gcp-gateway="projects/${SOURCE_PROJECT_ID}/regions/${SOURCE_REGION}/vpnGateways/${SOURCE_VPN_GW}" \\
--router="${REPLICA_ROUTER_NAME}" \\
--ike-version=2 \\
--shared-secret="${VPN_SHARED_SECRET}"
done
5. Set Target HostPort Directly to the Source Cloud SQL Private IP
export TARGET_HOST_PORT="${SOURCE_DB_PRIVATE_IP}:${DB_PORT}"
‘’‘Skip Step 2B if Using HA VPN: With HA VPN and PSA custom routes enabled above, TARGET_HOST_PORT points directly to SOURCE_DB_PRIVATE_IP:${DB_PORT}. Proceed directly to Step 3! Only execute Step 2B below if HA VPN is unavailable in your environment.’‘’
Step 2B (Fallback Only When HA VPN is Unavailable): Deploy Two-Hop HAProxy VMs
Run this step ONLY if HA VPN is not available and your VPCs are connected exclusively via VPC Network Peering. This deploys Hop 2 in $SOURCE_VPC and Hop 1 in $REPLICA_VPC:
1. Create Firewall Rules for Proxy VMs in Both VPCs
gcloud compute firewall-rules create allow-mysql-proxy-source \\
--project="${SOURCE_PROJECT_ID}" \\
--network="${SOURCE_VPC}" \\
--direction=INGRESS \\
--action=ALLOW \\
--rules="tcp:${DB_PORT}" \\
--source-ranges="${REPLICA_SUBNET_CIDR},${SOURCE_SUBNET_CIDR}" \\
--target-tags="${PROXY_TAG}"
gcloud compute firewall-rules create allow-mysql-proxy-replica \\
--project="${REPLICA_PROJECT_ID}" \\
--network="${REPLICA_VPC}" \\
--direction=INGRESS \\
--action=ALLOW \\
--rules="tcp:${DB_PORT}" \\
--source-ranges="${REPLICA_PSA_CIDR},${REPLICA_SUBNET_CIDR}" \\
--target-tags="${PROXY_TAG}"
2. Deploy Hop 2 Proxy VM (Inside Source VPC)
cat > startup-hop2.sh <<EOF
#!/bin/bash
apt-get update && apt-get install -y haproxy
cat << 'CFG' > /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend mysql_front
bind \*:${DB_PORT}
default_backend mysql_back
backend mysql_back
server source_db ${SOURCE_DB_PRIVATE_IP}:${DB_PORT} check
CFG
systemctl enable haproxy && systemctl restart haproxy
EOF
gcloud compute instances create "${HOP2_PROXY_NAME}" \\
--project="${SOURCE_PROJECT_ID}" \\
--zone="${SOURCE_ZONE}" \\
--machine-type="${PROXY_MACHINE_TYPE}" \\
--subnet="${SOURCE_SUBNET}" \\
--private-network-ip="${HOP2_PROXY_IP}" \\
--no-address \\
--tags="${PROXY_TAG}" \\
--shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring \\
--metadata-from-file=startup-script=startup-hop2.sh
3. Deploy Hop 1 Proxy VM (Inside Replica VPC)
cat > startup-hop1.sh <<EOF
#!/bin/bash
apt-get update && apt-get install -y haproxy
cat << 'CFG' > /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend mysql_front
bind \*:${DB_PORT}
default_backend mysql_back
backend mysql_back
server hop2_proxy ${HOP2_PROXY_IP}:${DB_PORT} check
CFG
systemctl enable haproxy && systemctl restart haproxy
EOF
gcloud compute instances create "${HOP1_PROXY_NAME}" \\
--project="${REPLICA_PROJECT_ID}" \\
--zone="${REPLICA_ZONE}" \\
--machine-type="${PROXY_MACHINE_TYPE}" \\
--subnet="${REPLICA_SUBNET}" \\
--private-network-ip="${HOP1_PROXY_IP}" \\
--no-address \\
--tags="${PROXY_TAG}" \\
--shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring \\
--metadata-from-file=startup-script=startup-hop1.sh
4. Verify the Proxy Chain & Point TARGET_HOST_PORT to Hop 1
gcloud compute ssh "${HOP1_PROXY_NAME}" \\
--zone="${REPLICA_ZONE}" \\
--project="${REPLICA_PROJECT_ID}" \\
--tunnel-through-iap --quiet \\
--command="exec 3<>/dev/tcp/127.0.0.1/${DB_PORT}; timeout 5 dd bs=1 count=60 <&3 2>/dev/null | tr -cd '\[:print:\]'; echo"
export TARGET_HOST_PORT="${HOP1_PROXY_IP}:${DB_PORT}"
Step 3: Export the Source Cloud SQL Server CA Certificate
Why TLS CA is Mandatory on MySQL (Both HA VPN & Proxy Approaches): MySQL removed mysql_native_password and enforces caching_sha2_password. Over a non-TLS connection, caching_sha2_password requires an RSA public-key exchange that fails during external replica verification with Error 1045 (28000): Access denied. Supplying the source Server CA certificate ensures the replica negotiates TLS end-to-end.
gcloud sql instances describe "${SOURCE_INSTANCE_NAME}" \\
--project="${SOURCE_PROJECT_ID}" \\
--format="value(serverCaCert.cert)" > "${SOURCE_CA_FILE}"
# Optional: Verify replication user grants & SSL cipher
mysql -h "${SOURCE_DB_PRIVATE_IP}" -u "${REPL_USER}" -p"${REPL_PASSWORD}" --connect-timeout=10 \\
-e "SHOW GRANTS FOR CURRENT_USER(); SHOW STATUS LIKE 'Ssl_cipher';"
Step 4: Create the Source Representation Instance
Create the Source Representation using the Cloud SQL Admin API and gcloud auth print-access-token. Notice how hostPort automatically uses TARGET_HOST_PORT ({SOURCE_DB_PRIVATE_IP}:{DB_PORT} for HA VPN, or {HOP1_PROXY_IP}:${DB_PORT} for the proxy fallback):
python3 -c "
import json, os
ca = open(os.environ\['SOURCE_CA_FILE'\]).read()
payload = {
'name': os.environ\['SOURCE_REPR_NAME'\],
'region': os.environ\['REPLICA_REGION'\],
'databaseVersion': os.environ\['DB_VERSION'\],
'onPremisesConfiguration': {
'hostPort': os.environ\['TARGET_HOST_PORT'\],
'username': os.environ\['REPL_USER'\],
'password': os.environ\['REPL_PASSWORD'\],
'caCertificate': ca
}
}
with open('repr.json', 'w') as f:
json.dump(payload, f)
"
curl -s -X POST \\
-H "Authorization: Bearer $(gcloud auth print-access-token)" \\
-H "Content-Type: application/json" \\
-d @repr.json \\
"https://sqladmin.googleapis.com/v1/projects/${REPLICA_PROJECT_ID}/instances"
rm -f repr.json
# Confirm creation with gcloud
gcloud sql instances describe "${SOURCE_REPR_NAME}" \\
--project="${REPLICA_PROJECT_ID}"
Step 5: Create the Cloud SQL Read Replica at MySQL
Create the replica linked to $SOURCE_REPR_NAME. Explicitly specify --edition=$REPLICA_EDITION (ENTERPRISE) if using custom machine tiers (db-custom-*), as MySQL defaults to ENTERPRISE_PLUS which only accepts predefined db-perf-optimized-N-* tiers:
gcloud sql instances create "${REPLICA_INSTANCE_NAME}" \\
--project="${REPLICA_PROJECT_ID}" \\
--master-instance-name="${SOURCE_REPR_NAME}" \\
--database-version="${DB_VERSION}" \\
--edition="${REPLICA_EDITION}" \\
--region="${REPLICA_REGION}" \\
--tier="${REPLICA_TIER}" \\
--storage-size="${REPLICA_STORAGE_SIZE}" \\
--storage-type="${REPLICA_STORAGE_TYPE}" \\
--network="projects/${REPLICA_PROJECT_ID}/global/networks/${REPLICA_VPC}" \\
--no-assign-ip
Step 6: Verify External Sync Settings & Start Replication
Once the replica finishes provisioning (4–8 minutes), verify the sync settings and start replication:
export ACCESS_TOKEN=$(gcloud auth print-access-token)
export SQL_API_BASE="https://sqladmin.googleapis.com/v1/projects/${REPLICA_PROJECT_ID}/instances/${REPLICA_INSTANCE_NAME}"
# 1. Verify External Sync Settings (Must return an empty error list)
curl -s -X POST \\
-H "Authorization: Bearer ${ACCESS_TOKEN}" \\
-H "Content-Type: application/json" \\
-d '{"syncMode":"ONLINE"}' \\
"${SQL_API_BASE}/verifyExternalSyncSettings" | python3 -m json.tool
# Expected response:
# { "kind": "sql#externalSyncSettingErrorList" }
# 2. Start External Sync
curl -s -X POST \\
-H "Authorization: Bearer ${ACCESS_TOKEN}" \\
-H "Content-Type: application/json" \\
-d '{"syncMode":"ONLINE"}' \\
"${SQL_API_BASE}/startExternalSync" | python3 -m json.tool
5. Confirming Replication is Genuinely Running
Caution — RUNNABLE Does Not Mean Replicating: An instance state of RUNNABLE in the Google Cloud Console only indicates the database VM is up. Always verify that START_EXTERNAL_SYNC completed without errors and inspect the replica’s [Repl] log stream.
\# 1. Confirm START_EXTERNAL_SYNC operation is DONE with no errors
gcloud sql operations list \\
--instance="${REPLICA_INSTANCE_NAME}" \\
--project="${REPLICA_PROJECT_ID}" \\
--limit=5 \\
--format="table(operationType,status,error.errors\[0\].message:label=ERROR)"
# 2. Inspect the MySQL Replica Receiver & SQL threads in Cloud Logging
gcloud logging read \\
"resource.type=cloudsql_database AND resource.labels.database_id=\\"${REPLICA_PROJECT_ID}:${REPLICA_INSTANCE_NAME}\\"" \\
--project="${REPLICA_PROJECT_ID}" \\
--limit=100 \\
--format="value(textPayload)" | grep "\\\[Repl\\\]"
Healthy replication output in Cloud Logging looks like:
\[System\] \[MY-014002\] \[Repl\] Replica receiver thread for channel '':
connected to source 'repl_user@<TARGET_HOST_PORT>' with
server_uuid=..., server_id=.... Starting GTID-based replication.
\[Note\] \[MY-013498\] \[Repl\] Replica SQL thread for channel '' initialized,
starting replication in log 'mysql-bin.000003' at position 3396
6. Clean Teardown / Rollback
If you need to tear down the replica and source representation (and optional proxy VMs if the fallback was used), delete the Replica first:
# 1. Delete the Cloud SQL Replica and Source Representation
gcloud sql instances delete "${REPLICA_INSTANCE_NAME}" \\
--project="${REPLICA_PROJECT_ID}" --quiet
gcloud sql instances delete "${SOURCE_REPR_NAME}" \\
--project="${REPLICA_PROJECT_ID}" --quiet
# 2. (Only if Fallback Step 2B was used) Delete the Proxy VMs
gcloud compute instances delete "${HOP1_PROXY_NAME}" \\
--zone="${REPLICA_ZONE}" \\
--project="${REPLICA_PROJECT_ID}" --quiet
gcloud compute instances delete "${HOP2_PROXY_NAME}" \\
--zone="${SOURCE_ZONE}" \\
--project="${SOURCE_PROJECT_ID}" --quiet
rm -f startup-hop1.sh startup-hop2.sh "${SOURCE_CA_FILE}"`


