Error DataStoreConfig cache-datastore doesn't exists or not initialised yet for APIKey policy with a new Edge

Google Cloud Apigee
1

I have add new environment with new router and message processor but Cassandra and zookeeper are not install into new router and message. they are in already exist 2 gateway and management servers.

When I access Management UI then select new environment I got following message
Error fetching analytics data
Data Access Error - Data Access Error

and when I add policy verify API key, I got following error at runtime in message processor log file - /opt/apigee/var/log/edge-message-processor/logs

exception when fetching entry 888-20180830 - ignoring it…
com.apigee.datastore.DataStoreNotFoundException: DataStoreConfig cache-datastore doesnt exists or not initialized yet.
at com.apigee.datastore.client.astyanax.AstyanaxCassandraClient.getContext(AstyanaxCassandraClient.java:2775) ~[datastore-service-1.0.0.jar:na]
at com.apigee.datastore.client.astyanax.AstyanaxCassandraClient.getGreatestLongTypeColumn(AstyanaxCassandraClient.java:2373) ~[datastore-service-1.0.0.jar:na]
at com.apigee.cache.distributed.SecondLevelCacheManagerImpl.getSequenceId(SecondLevelCacheManagerImpl.java:621) ~[cache-1.0.0.jar:na]
at com.apigee.cache.distributed.SecondLevelCacheManagerImpl.internalGetRowKeyInColumnFamily(SecondLevelCacheManagerImpl.java:211) ~[cache-1.0.0.jar:na]
at com.apigee.cache.distributed.SecondLevelCacheManagerImpl.internalGetEntry(SecondLevelCacheManagerImpl.java:382) ~[cache-1.0.0.jar:na]
at com.apigee.cache.distributed.SecondLevelCacheManagerImpl.getEntry(SecondLevelCacheManagerImpl.java:376) ~[cache-1.0.0.jar:na]
at com.apigee.cache.distributed.DistributedCacheManagerImpl.getEntry(DistributedCacheManagerImpl.java:314) ~[cache-1.0.0.jar:na]
at com.apigee.steps.keyvaluemapoperations.KeyValueMapOperationsExecution.execute0(KeyValueMapOperationsExecution.java:135) [keyvaluemap-operations-1.0.0.jar:na]
at com.apigee.steps.keyvaluemapoperations.KeyValueMapOperationsExecution.execute(KeyValueMapOperationsExecution.java:68) [keyvaluemap-operations-1.0.0.jar:na]
at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:151) [message-processor-1.0.0.jar:na]
at com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask.call(AbstractAsyncExecutionStrategy.java:74) [message-flow-1.0.0.jar:na]
at com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask.call(AbstractAsyncExecutionStrategy.java:45) [message-flow-1.0.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]

2

Did you configure the new router/message processor with the same CASS_HOSTS and ZK_CLIENT_HOSTS as your other, working router/message processors?

https://docs.apigee.com/private-cloud/v4.18.05/edge-configuration-file-reference?hl=en

3

Yes, My configuration are describe in your link but I found out later that firewall was block from Cassandra back to Edge but Edge to Cassandra is ok but has no error during installation

is that the cause of problem ??

4

After asking to Apigee support, I’ve resolved the problem on Management UI - Data Access error by running following command to check

verify that new environment correctly associate with axgroup

apigee-adminapi.sh orgs envs analytics status -o -e <new_env> --host localhost --port 8080 --debug --admin <admin_user> --pwd <admin_password>

to associate new environment with axgroup

apigee-adminapi.sh orgs envs analytics scopes add -o -e <new_env> -g axgroup-001 --host localhost --port 8080 --debug --admin <admin_user> --pwd <admin_password>

5

We had the same issue, and the reason was that our Gateway Pod did not have the datastore assigned.

You can check it using the command:

curl -u ADMINEMAIL:PW "http://MSIP:port/v1/servers?pod=GATEWAY_POD"

Datastores are shown like this:

“type” : [ “cache-datastore”, “dc-datastore”, “kms-datastore”, “keyvaluemap-datastore”, “counter-datastore” ],

If you dont see it, you need to add it and after restart the Message-Processor, make sure that you add all the IPs if you have more than one server.

Command to add the datastore is:

curl -u ADMINEMAIL:PW "http://MSIP:port/v1/servers -d \
  "Type=kms-datastore&Type=dc-datastore&Type=keyvaluemap-datastore&Type=counter-datastore&Type=cache-datastore \
  &InternalIP=NEWIP®ion=REGION&pod=GATEWAY_POD" -H 'content-type: \
  application/x-www-form-urlencoded' -X POST

More info: https://docs.apigee.com/private-cloud/v4.50.00/moving-apigee-servers?xyzallow#changingtheipaddressofacassandranode-updatedatastoreregistrations

6

If your firewall was stateless and thus required rules to permit the response traffic then yes, this could definitely cause it.