Cassandra PoolTimeoutException while running apigee-migrate-tool

Google Cloud Apigee
1

We are trying to export from a 17.05 OPDK all our proxies with apigee-migrate-tool but we are seeing this error.

Fatal error: Cannot read property 'length' of undefined

The error happens after some proxies are already exported. We have around 200 proxies and the size varies between 12MB to a few KB.

Looking at the management-server logs (system.log) there are quite a few PoolTimeoutException like this one

2018-03-29 16:16:22,346 org:org-example  pool-3-thread-2490 ERROR DATASTORE.CASSANDRA - AstyanaxCassandraClient.fetchDynamicCompositeColumns() : Error while querying columnfamily : [api_proxy_revisions_r21, org-example@proxy-example] for rowkey:{}
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=None(0.0.0.0):0, latency=2017(2017), attempts=2]Timed out waiting for connection
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231) ~[astyanax-core-1.56.43-apigee.jar:na]
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198) ~[astyanax-core-1.56.43-apigee.jar:na]
        at com.netflix.astyanax.connectionpool.impl.LeastOutstandingExecuteWithFailover.borrowConnection(LeastOutstandingExecuteWithFailover.java:74) ~[astyanax-core-1.56.43-apigee.jar:na]

The above error starts with the proxy that failed to be exported.

This post seems somewhat related though it was while doing a load test for the KVM. I was looking to change that same property (maxactive_cassandra_connections) though which of all these files I should change and which could be a good number to run the script? (Assuming that the change has to be done in the management-server, please correct me if am wrong)

/opt/apigee/edge-management-server/conf/cps-management.properties:kvm.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-management.properties:kms.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-management.properties:quota.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/repository.properties:cassandra.defaults.maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/auth-datastore.properties:maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/cache.properties:distributed.maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/keyvaluemap-datastore.properties:maxactive_cassandra_connections=5
/opt/apigee/edge-management-server/conf/keymanagement-datastore-config.properties:maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/cps-runtime.properties:kvm.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-runtime.properties:kms.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-runtime.properties:l2cache.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-runtime.properties:timeseries.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/cps-runtime.properties:quota.maxactive_cassandra_connections=8
/opt/apigee/edge-management-server/conf/audit.properties:maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/developer-datastore.properties:maxactive_cassandra_connections=10
/opt/apigee/edge-management-server/conf/counter.properties:maxactive_cassandra_connections=10

We had ran the export all proxies in a 15.07 without problems in the past, was any of those configuration changed with the recent versions?

2

@Maruti Chand @Baba Krishnankutty

Any help would be greatly appreciated.

3

Hi @Matias,

management servers are not intended for the load; you may want to wait for few sec-min (sleep) before each proxy download.

IIRC you should modify the app repo keyspace related property file to increase the maxactive_cassandra_connections if you want to see raising that fixes your issue for now.

4

Thanks for the reply @Maruti Chand.

management servers are not intended for the load

Good to know. It’s unfortunate that the script from that repo doesn’t follow those guidelines.

I ran a test with more connections in the repository.properties though it didn’t work.

Does Apigee has any other public repo to migrate proxies from one cluster to another one?