Data not showing up in the analytics dashboard. Qpid is not able to push data to postgres db

Google Cloud Apigee
1

Unable to see data in the dashboard. Observed from the qpid stats, the msgout is 0 . Restarting all qpid/postgres* didn’t solve the issue.

[root@XXX ~]# apigee-all status
+ apigee-service apigee-postgresql status
apigee-service: apigee-postgresql: OK
+ apigee-service apigee-qpidd status
apigee-service: apigee-qpidd: OK
+ apigee-service edge-postgres-server status
apigee-service: edge-postgres-server: OK
+ apigee-service edge-qpid-server status
apigee-service: edge-qpid-server: O

[root@XXX ~]# qpid-stat -q
Queues
  queue                                     dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  =========================================================================================================================
  701562ec-e6b4-4419-aa15-7156d0c832ee:0.0       Y        Y        0     0      0       0      0        0         1     2
  ax-q-axgroup-001-consumer-group-001       Y                     75    75      0     147k   147k       0         0     2
  ax-q-axgroup-001-consumer-group-001-dl    Y                      0     0      0       0      0        0         0     2


v1/analytics/groups/ax/axgroup-001
{
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "xxx~test", "xxx~prod" ],
  "uuids" : {
    "qpid-server" : [ "36da7c82-6388-4036-858e-107ba7b74708" ],
    "aries-datastore" : [ ],
    "postgres-server" : [ "d2c861b2-38a5-4baf-8f54-9186b1eba2ec" ],
    "dw-server" : [ ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "36da7c82-6388-4036-858e-107ba7b74708" ],
    "datastores" : [ "d2c861b2-38a5-4baf-8f54-9186b1eba2ec" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}

On the edge-postgres-server log, the following errors is being thrown

at com.apigee.analytics.serverbootstrap.pga.EventSourceProvider$1$1.run(EventSourceProvider.java:89) ~[server-bootstrap-impl-1.0.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /organizations/xxx/environments/prod/properties
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.getData(ZooKeeperServiceImpl.java:827) ~[zookeeper-1.0.0.jar:na]
... 7 common frames omitted
2

bytes out can be 0 when your postgres is down or qpid-server unable to talk to postgres.

Can you check the connectivity and also run curl -v http://localhost:8084/v1/servers/self and check if the UUID is matching with the one you see in the axgroup output?

3

@Sri - also check logs for edge-qpid-server which should be writing into the DB

I’d also check zookeeper for errors and that /organizations/xxx/environments/prod/properties is accessible in zookeeper (zkClis.sh then get /organizations/xxx/environments/prod/properties should return info about that node..)

4

@Andy Trickett The /properties path itself not exists under environment in ZK.

[zk: localhost:2181(CONNECTED) 4] get /organizations/xxx/environments/test/
metadata     servers      apiproxies