Multiple SQL Server Datastreams intermittently enter FAILED for exactly 60 seconds in asia-northeast3

Hello,

We are experiencing frequent and intermittent state changes across multiple Google Cloud Datastream streams in the asia-northeast3 region.

Environment:

  • Source: Self-managed Microsoft SQL Server
  • Connectivity: Forward SSH tunnel through a self-managed bastion host
  • Destination: BigQuery
  • CDC source: SQL Server transaction logs
  • Region: asia-northeast3

Symptoms:

Starting on October 27, 2026, multiple streams began transitioning from RUNNING to FAILED and then returning automatically to RUNNING almost exactly 60 seconds later.

We observed:

  • More than 60 FAILED transitions on October 27
  • More than 200 FAILED transitions on October 28
  • Multiple independent streams and source databases are affected
  • Some streams repeat the transition at intervals of approximately 7 minutes
  • In some cases, different streams enter FAILED at nearly the same time
  • No stream has entered FAILED_PERMANENTLY

A typical state transition looks like this:

2026-10-28 14:35:06 KST — RUNNING → FAILED
2026-10-28 14:36:06 KST — FAILED → RUNNING

Cloud Logging only reports:

eventCode: STREAM_STATE_CHANGED
eventMessage: Stream state has changed.
newState: FAILED

No underlying connectivity, authentication, CDC, LSN, permission, or BigQuery error is included.

What we have verified:

  1. SSH authentication succeeds.

The bastion host logs show successful authentication entries. We do not see authentication failures, access denials, or MaxStartups throttling messages.

  1. Connectivity from the SSH bastion to SQL Server remains available.

We continuously monitored the SQL Server TCP port from the bastion host at five-second intervals. No TCP failures or timeouts were recorded while Datastream continued to enter FAILED.

  1. CDC processing continues successfully.

For example, shortly before one FAILED transition, the logs showed:

  • SQL_SERVER_CDC_FETCH_STARTED
  • SQL_SERVER_CDC_FETCH_COMPLETED
  • BIGQUERY_WRITER_WRITE_COMPLETED

The stream then entered FAILED a few minutes later and returned to RUNNING after 60 seconds.

  1. BigQuery writes are succeeding.

We found successful BigQuery writer completion events and no corresponding BigQuery write errors.

  1. There were no relevant configuration changes.

We checked Cloud Audit Logs and found no Datastream connection profile, stream, firewall, or related Compute configuration changes around the time the issue began.

  1. FetchErrors also returns an internal unknown error.

Calling the Datastream FetchErrors API for affected streams returns:

code: 2
reason: UNKNOWN
message: Unknown Error.
originalMessage: “”

This happens for multiple affected streams.

  1. The source connection can currently be discovered successfully.

Datastream connection profile discovery completes successfully, and all affected streams currently report RUNNING between these brief failures.

Questions:

  1. Could this be a Datastream health-check or control-plane issue in asia-northeast3?
  2. Why would a stream enter FAILED for exactly 60 seconds when SSH connectivity, SQL Server CDC fetches, and BigQuery writes remain successful?
  3. Is there another API, metric, or diagnostic log that can expose the actual reason behind STREAM_STATE_CHANGED?
  4. Has there been a recent Datastream infrastructure or health-check behavior change that could cause this state flapping?
  5. Should these brief FAILED transitions be treated as actual replication outages when CDC checkpoints and BigQuery writes continue successfully?

We have retained the affected stream names, operation UUIDs, project information, and detailed timestamps and can provide them privately to Google Cloud Support if required.

Thank you.

I have been got it that ERROR situlation!

The issue appears to be related to the OpenSSH MaxStartups setting on our shared SSH bastion.

Many Datastream streams use the same Forward SSH tunnel host. The bastion was using the default setting:

MaxStartups 10:30:100

This causes sshd to begin randomly dropping new unauthenticated connections when 10 concurrent authentication attempts are reached.

We changed it to:

MaxStartups 40:30:100

After reloading sshd, we have not observed any further RUNNING → FAILED state transitions. CDC fetches and BigQuery writes continue normally.

We are still monitoring the environment, so this is currently a likely cause rather than a confirmed final resolution.

Thank you !