Hi everyone,
We recently upgraded to the On prem Private Cloud 4.15.04.03 Edge service pack, and we notice that there is no analytics data coming through to the management server.
We are running java 1.7 and i believe analytics was coming in just fine before the patch upgrade. Any idea why this might be the case? Thanks!
Hi @Chris Covney, I don’t have a direct answer for you but I remembered a related question had been asked - check out Analytics not showing up in newly created orgs, especially the point about the 15.04 specific script: apigee-postgres-register.sh that you need to run if you have not.
Let us know if you don’t get any success from the answers to that other question.
It looks like the ax status is all good:
{
"environments" : [ {
"components" : [ {
"hosts" : "[localhost, localhost]",
"message" : "success",
"name" : "qs",
"status" : "SUCCESS",
"uuid" : "[9be9a4e6-3d18-4d2b-8090-6f17de5daece, 7c251b96-243a-4de8-a76e-02e19b0a8add]"
}, {
"hosts" : "[localhost:localhost]",
"message" : "success at Mon Sep 28 15:19:23 EDT 2015",
"name" : "pg",
"status" : "SUCCESS",
"uuid" : "[ab448e45-5af2-48c8-badc-7b8dc166c3b4:494365b0-3efc-4620-a7bd-077ad99ab2ac]"
} ],
"message" : "",
"name" : "prod"
} ],
"organization" : "Intralinks",
"status" : "SUCCESS"
}
In the Qpid servers system.log we see this error:
2015-09-30 15:45:45,021 Dispatcher-0-Conn-27 WARN c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ.
com.apigee.analytics.persistence.api.PersistenceException: No Sink found for qualifier MsgQualifier{queueName='ax-q-axgroup001-consumer-group-001', groupName='axgroup001', cg=consumer-group-001}. Rejecting messages
I know that java needed to be upgraded on the Management Servers for the .03 patch. We upgraded java on ALL the Apigee nodes for consistency, do you think this is breaking analytics?
It says access denied when i try to access this community article, and judging by the title, it appears like it might be relevant to this investigation. Can you ascertain the contents of this article?
[https://community.apigee.com/articles/8512/qpid-re...](https://community.apigee.com/articles/8512/qpid-rejecting-messages-with-no-sink-found-for-qua.html)
also, this might be an issue. not sure if i’ve run the commands correctly though:
-bash-4.1$ sh /apps/apigee4/bin/psql.sh
psql (9.3.9)
Type "help" for help.
apigee=# \d analytics."accountmanagement.Common.agg_app"
Did not find any relation named "analytics."accountmanagement.Common.agg_app"".
apigee=# \d analytics."accountmanagement.Common.fact"
Did not find any relation named "analytics."accountmanagement.Common.fact"".
apigee=# \d analytics."accountmanagement.Common.agg_app" \d analytics."accountmanagement.Common.fact";
Did not find any relation named "analytics."accountmanagement.Common.agg_app"".
lol sorry for inundating you with messages, but i also am seeing errors in the PG sys logs when i bounce PG server:
2015-09-30 16:37:12,055 main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path
2015-09-30 16:37:12,056 main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [onboarding-module]
2015-09-30 16:37:12,057 main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: []
2015-09-30 16:37:12,057 main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway.
2015-09-30 16:37:14,041 main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts
2015-09-30 16:37:14,438 qtp39112485-30 - /v1/servers/self/uuid ERROR REST - ExceptionMapper.toResponse() : Error occurred : null
org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336)
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211)
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
2015-09-30 16:37:14,455 qtp39112485-32 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/
2015-09-30 16:37:14,456 qtp39112485-32 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null
org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336)
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211)
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
2015-09-30 16:37:15,254 main ERROR KERNEL - MicroKernel.deployAll() : Some services were not started due to unresolved dependencies:
2015-09-30 16:37:15,254 main ERROR KERNEL - MicroKernel.deployAll() : AccessControl was not started because of missing dependencies: [CacheService]
2015-09-30 16:37:15,254 main ERROR KERNEL - MicroKernel.deployAll() : SecurityService was not started because of missing dependencies: [UserService, ResourceService]
I do have access to that community article, you are correct it’s talking about the same qpid error “no sink found for…” here’s what it says as the solution:
-Run the following sript:
${APIGEE_ROOT}/apigee4/share/installer/apigee-postgres-register.sh
-Restart qpid services
It’s also worth noting the the OPDK Documentation mentions the following points about the procedure:
The restart is required to create new consumer group queues in existing consumer group QPIDs. On occasion, queue creation can take up to 15-20 minutes. Checks mentioned in #11 should be repeated a few times till the confirmation is obtained. (This is referring to running the qpid-stat -q command until you see the Consumer group in the QPid.)
Not sure though whether this extra insight into what happens after running the script and the below are related, or independent.
@Alex Toombs saw you had been helping with similar issues, are you able to help @Chris Covney with this one?
@mschreuder and @Chris Covney running the ${APIGEE_ROOT}/apigee4/share/installer/apigee-postgres-register.sh script should resolve this.
The only other suggestion I have on this is to restart all components (RRTs/MPs, not just AX components) so that everything picks up these changes. Even after doing this, are you still seeing the “No Sink found” errors?
Thanks guys! I will run this script and let you know if it fixes the issue. Just to be clear, after running this script, should do a rolling bounce of all components?
Thanks!
@Chris Covney yes, all AX components (PG / QPID), and the Routers and MPs. In the past, that’s been required for all the changes to be picked up by all needed components.
Please advise if you still get “No sink found” errors after that, and we can proceed from that point if so.
Thanks.
@Chris Covney can you give it another try to access the article?