When developing and maintaining API platforms with Apigee Hybrid, ensuring sub-millisecond response times is a top priority for technical architects and API developers. However, when an API proxy experiences performance degradation or unexplained latency during key security checks—specifically during the execution of the VerifyAPIKey policy—the root cause often lies deep within the datastore layer.
Because Apigee Hybrid uses Apache Cassandra as its distributed runtime datastore, any slowdown in querying client credentials, API products, or developer profiles directly impacts the API proxy’s response times.
In this article, we will walk through a step-by-step diagnostic guide to installing a Cassandra debugging client inside your Kubernetes cluster, querying Cassandra 4.0’s real-time in-memory performance tables, and interpreting key indicators to resolve API key verification bottlenecks.
Technical Overview: the mTLS connection barrier
By default, Apigee Hybrid enforces strict security by requiring Mutual TLS (mTLS) encryption for all internal traffic, including connections to the Cassandra database. To connect a debugging client tool (like cqlsh) to Cassandra, you cannot simply run a standard client container; you must supply a trusted security certificate.
To resolve this, we will locate the TLS secret dynamically generated by the Cassandra user setup job (apigee-cassandra-user-setup) and mount it directly into our custom client pod.
Step-by-Step Diagnostic Guide
Step 1: Locate the Kubernetes TLS Secret
Execute the following command to retrieve the exact name of the TLS-type Kubernetes Secret containing the required security certificates for your Apigee namespace:
kubectl get secrets -n apigee
--field-selector ``type=kubernetes.io/tls`` | grep apigee-cassandra-user-setup | awk '{print $1}'
This command filters out other configuration secrets and returns the specific dynamically generated TLS secret name (e.g., apigee-cassandra-user-setup-tls).
Step 2: Define and Deploy the Cassandra Client Pod
Create a deployment file named cass-client.yaml. This manifest defines a debugging pod running the official Apigee Cassandra client image. It automatically imports the required credentials and mounts the TLS secret retrieved in Step 1 to /opt/apigee/ssl.
Make sure to replace PUT_THE_NAME_OF_YOUR_SECRET_HERE with your actual secret name. Use the version of the Apigee Cassandra client (version 1.16.5 in the example below) regarding the version of the Apigee Hybrid runtime you are using:
apiVersion: v1
kind: Pod
metadata:
labels:
name: my-cassandra-client
namespace: apigee
spec:
containers:
- name: my-cassandra-client
image: "``gcr.io/apigee-release/hybrid/apigee-hybrid-cassandra-client:1.16.5``"
imagePullPolicy: Always
command:
- sleep
- "3600"
env:
- name: CASSANDRA_SEEDS
value: apigee-cassandra-default.apigee.svc.cluster.local
- name: APIGEE_DML_USER
valueFrom:
secretKeyRef:
key: dml.user
name: apigee-datastore-default-creds
- name: APIGEE_DML_PASSWORD
valueFrom:
secretKeyRef:
key: dml.password
name: apigee-datastore-default-creds
volumeMounts:
- mountPath: /opt/apigee/ssl
name: tls-volume
readOnly: true
volumes:
- name: tls-volume
secret:
defaultMode: 420
secretName: PUT_THE_NAME_OF_YOUR_SECRET_HERE
restartPolicy: Never
Deploy the pod and open an interactive bash session inside the container:
kubectl apply -f cass-client.yaml -n apigee
kubectl exec -n apigee my-cassandra-client -it -- bash
Step 3: Connect to Cassandra via cqlsh
Once inside the client pod, establish a secure connection using cqlsh and your Cassandra administrative credentials:
cqlsh ${CASSANDRA_SEEDS} -u admin_user -p <your_admin_password> --ssl
(Note: If you are using a fresh development environment, the default admin password is iloveapis123. Otherwise, make sure to extract your active admin password from the apigee-datastore-default-creds secret).
Inspecting Keyspaces and Table Latencies
Once connected, your command prompt should change to:
admin_user@cqlsh>
1. Identify Your KMS Keyspace
Apigee segregates its datastore into functional keyspaces. Run the following query to identify your specific Key Management System (KMS) keyspace:
SELECT keyspace_name FROM system_schema.keyspaces;
Look for the keyspace starting with kms_ (e.g., kms_apigee_h_116_hybrid). This keyspace is responsible for validating all API keys, client credentials, and resource authorizations.
2. Query Real-Time Read Latency Metrics
Cassandra 4.0 introduces Virtual Tables (exposed under the system_views keyspace). These tables do not read from disk; they fetch JMX metrics in memory with extremely low overhead, making them perfectly safe to run on production environments.
Run this query to isolate read latencies within your KMS keyspace:
SELECT table_name, count, max_ms, p50th_ms, p99th_ms
FROM system_views.local_read_latency
WHERE keyspace_name = '<your-kms-keyspace>'
ALLOW FILTERING;
Here is an example of results you get:
Key Indicators to Monitor (KMS & API Key Focus)
When assessing the results of your query, pay close attention to the following tables. They are queried sequentially when a client presents an API key:
| Target Table | Role in VerifyAPIKey Policy | Operational Impact |
|---|---|---|
| app_credential | Holds the raw consumer API keys, client secrets, and status (approved/revoked). | Causes direct latency spikes during the key validation phase. |
| api_product | Contains API Products associated with the key to authorize resource paths. | Slows down product matching, blocking request processing. |
| app | Contains developer app definitions, custom attributes, and application status. | Leads to read amplification (searching through too many SSTables). |
| developer | Contains developer profile details (such as email, status, and custom metadata). | Delays the retrieval of developer metadata attached to the context. |
Interpreting the Results
-
High
p99th_ms(> 10ms) onapp_credential: This indicates that the database engine is struggling to find credential data. This is typically caused by disk I/O bottlenecks on the underlying persistent volumes or compaction backlog (high SSTable count). -
High
max_mswith Lowp50th_ms: This points to intermittent latency spikes, which are frequently correlated with JVM Garbage Collection (GC) pauses on specific Cassandra pods or heavy table compactions. -
Count is 0 across all tables on the local node: The node you are currently querying is not acting as the active coordinator or replica for those keys. Run the diagnostic steps across other Cassandra pods in your statefulset to locate where the queries are active.
References & Official Documentation
For deeper architectural guidance, refer to the following official resources:
Tailor your solution with help from a Google Cloud Sales Specialist