Apigee OPDK 4.53.01 Multi-DC Cassandra Schema Split (system_schema.columns missing entries in DC1)

Google Cloud Apigee
,
1

  1. Apigee version: 4.53.01

  2. 2 DCs, 3 Cassandra nodes per DC

  3. nodetool describecluster showing two schema versions

  4. DC1 schema UUID

  5. DC2 schema UUID

  6. Example tables with missing metadata:

    • apimodel_v2.apis_revision
    • apimodel_v2.method
    • apimodel_v2.resource
    • audit.audits
    • kms.oauth_20_access_tokens

  7. Error: SchemaKeyspace$MissingColumns

    Columns not found in schema table

system_schema.tables contains entries

system_schema.columns contains 0 rows

all 6 nodes are UP and reachable.

2

Hi @RakeshTalanki We are currently building a new Apigee OPDK 4.53.01 production environment with 2 DCs (12 nodes total). During installation we encountered a Cassandra schema mismatch issue where nodes reported different schema versions even though all nodes were UP/UN and inter-DC connectivity was healthy. After extensive troubleshooting, including verification of ZooKeeper, Cassandra gossip, schema versions, keyspaces, and replication, we rebuilt the environment from scratch. Has anyone else experienced schema divergence during initial multi-DC installation of OPDK 4.53.01 ? Were there any specific Cassandra schema synchronization or installation sequencing recommendations from Apigee Support?

My current assessment

At this point the evidence strongly suggests:

Cassandra cluster formation is successful.
Schema agreement exists before MS setup.
During apimodel-schema.cql, system_schema metadata becomes inconsistent.
Cassandra starts rejecting schema merges.
Schema UUIDs diverge.
Management Server waits forever for schema agreement.

The PROD comparison conclusively proves that apis_revision should have 27 column definitions and audit.audits should have 19 column definitions,
whereas your new cluster currently has zero, which is exactly the condition producing the SchemaKeyspace$MissingColumns exceptions.

This is the clearest root-cause evidence we’ve seen so far. The next critical step is to determine which nodes have 27/19 versus 0/0.
That will tell us where the corruption first appears.*

This is not a Management Server issue anymore.

This is a Cassandra schema synchronization/corruption issue occurring during schema creation.

SchemaKeyspace$MissingColumns
Columns not found in schema table for apimodel_v2.apis_revision
Unable to merge schema from /100.102.11.XX1
Unable to merge schema from /100.102.11.XX2

3

Hi @ukmakineni, thank you for providing such a detailed breakdown of the issue you’re seeing during your Apigee OPDK setup, providing your troubleshooting steps and root-cause evidence is incredibly helpful.

I am leaving this open for our community experts and engineers to share their insights or experiences with multi-datacenter installations.

While we wait for the community to weigh in, I’d also like to invite you to review the recordings from our community techtalks :right_arrow: Google Cloud Apigee Community TechTalks

5

below is the error:–

ERROR SchemaKeyspace.java:919 - No partition columns found for table apimodel_v2.apis_revision in system_schema.columns

ERROR SchemaKeyspace.java:919 - No partition columns found for table audit.audits in system_schema.columns

ERROR CassandraDaemon.java:289 - Error while loading schema
ERROR CassandraDaemon.java:915 - Exception encountered during startup

Your Cassandra schema metadata is corrupted.

Specifically, Cassandra has entries in:system_schema.tables for tables such as: apimodel_v2.apis_revision , audit.audits but their corresponding definitions are missing from:system_schema.columns

6

SchemaKeyspace$MissingColumns:
Columns not found in schema table for apimodel_v2.apis_revision
Unable to merge schema from
100.102.xx.x
100.102.xx.x
100.102.xx.x
nodetool describecluster shows 3 schema versions after apimodel-schema.cql execution.

7

This kind of issue usually points to schema disagreement between Cassandra nodes rather than a simple missing-row problem. In a multi-DC Apigee OPDK setup, I’d first compare the schema versions across all Cassandra nodes and confirm that replication and gossip are healthy in both data centers.

If system_schema.columns entries exist in DC2 but are missing in DC1, check nodetool describecluster, nodetool status, and the Cassandra logs for schema migration or replication errors. It’s also worth confirming that every node is running the same Cassandra/Apigee version and that no node was unavailable during the most recent schema update.

I would avoid manually inserting records into system_schema. The safer approach is to identify the node with the correct schema, resolve any connectivity or version mismatch, and allow Cassandra’s schema propagation to synchronize naturally. If the disagreement remains, Apigee support or the documented OPDK schema-repair procedure would be the safest next step._