Stuck with error in edge router - Checking for router on 127.0.0.1 port 8081

satmce · November 24, 2021, 6:15pm

Trying the 9 node setup. node1 to node3 setup is done( edge management server), cassandra, zookeeper.

trying this setup on node4. stuck here. Your help is really appreciated.

ENVIRONMENT:

OS: RHEL

Apigee Version: 4.51.00

[root@rhel-apigee9-set2-vm-04 apigee]# java -version
openjdk version “1.8.0_312”
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

[root@rhel-apigee9-set2-vm-04 apigee]# uname -a
Linux ydx-stage-msa-rhel-apigee9-set2-vm-04 4.18.0-305.17.1.el8_4.x86_64 #1 SMP Mon Aug 30 07:26:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

/opt/apigee/apigee-setup/bin/setup.sh -f /tmp/apigee/response-edge.txt -p rmp

apigee-service: disable confirmation prompts
0 * * * * nice -n 19 ionice -c3 /usr/sbin/logrotate -f /opt/apigee/edge-router/logrotate/logrotate.conf
removing logrotate cron entry
adding logrotate cron entry
logrotate reconfigure for edge-router
updated property conf_datastore_push_cron_duration in file /opt/apigee/token/application/router.properties
apigee-configutil: edge-router: # OK
apigee-service: edge-router: edge-router is not running
apigee-configutil: edge-router: # OK
apigee-service: edge-router: Not running (NO_LOCKFILE)
apigee-service: edge-router: status=2, continuing
apigee-service: edge-router: OK
apigee-service: edge-router: edge-router is running
Checking for router on 127.0.0.1 port 8081 …

============= port currently running

[root@rhel-apigee9-set2-vm-04 apigee]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 890/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1647/sshd
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 81976/nginx: master
tcp6 0 0 :::5355 :::* LISTEN 890/systemd-resolve
tcp6 0 0 :::22 :::* LISTEN 1647/sshd

[root@apigee9-set2-vm-04 sbin]# getenforce
Permissive

[root@apigee9-set2-vm-04 sbin]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

this is what the healthcheck of the

apigee-services on all the nodes:

host01 | CHANGED | rc=0 >>

apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
apigee-service apigee-openldap status
apigee-service: apigee-openldap: OK
apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK
apigee-service edge-management-server status
apigee-service: edge-management-server: OK
apigee-service edge-ui status
apigee-service: edge-ui: OK

host02 | CHANGED | rc=0 >>

apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK

host03 | CHANGED | rc=0 >>

apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK

host04 | FAILED | rc=1 >>

apigee-service edge-router status
apigee-service: edge-router: Not running (DEAD)
apigee-all: Error: status failed on [edge-router]non-zero return code

host05 | FAILED | rc=1 >>

apigee-service edge-router status
apigee-service: edge-router: Not running (DEAD)
apigee-all: Error: status failed on [edge-router]non-zero return code

Edge router error is:

2021-11-24 18:05:51,559 main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting
2021-11-24 18:06:01,611 main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Default schema
2021-11-24 18:06:01,612 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Successfully initialized the service : ZooKeeperService
2021-11-24 18:06:01,612 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : MetricsService has these dependencies [ [NameRegistration{name=CommunicationService}] ]
2021-11-24 18:06:01,613 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : OrganizationService has these dependencies [ [NameRegistration{name=CommunicationService}, NameRegistration{name=RepositoryService}] ]
2021-11-24 18:06:01,613 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : RepositoryService has these dependencies [ [NameRegistration{name=ZooKeeperService}, NameRegistration{name=DataStoreService}, NameRegistration{name=ServerMetricsService}] ]
2021-11-24 18:06:01,614 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Initializing service : RepositoryService that implements InitializeAwareService
2021-11-24 18:06:01,657 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=apprepo is using class=com.apigee.services.repository.nosql.CassandraRepository
2021-11-24 18:06:01,659 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=zookeeper is using class=com.apigee.services.repository.zookeeper.ZKRepository
2021-11-24 18:06:01,666 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=localrepo is using class=com.apigee.services.repository.filesystem.FileSystemRepository
2021-11-24 18:06:01,685 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Successfully initialized the service : RepositoryService
2021-11-24 18:06:01,686 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ServerRegistrationService has these dependencies [ [NameRegistration{name=RepositoryService}, NameRegistration{name=CommunicationService}, NameRegistration{name=ComponentConfigurationService}, NameRegistration{name=ServerMetricsService}] ]
2021-11-24 18:06:01,687 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Initializing service : ServerRegistrationService that implements InitializeAwareService
2021-11-24 18:06:21,635 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed
2021-11-24 18:06:21,746 main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED
2021-11-24 18:06:21,757 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:06:24,746 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 3000. Adjusted session timeout ms: 3000
2021-11-24 18:06:24,749 main-EventThread WARN o.a.c.ConnectionState - ConnectionState.handleExpiredSession() : Session expired event received
2021-11-24 18:06:27,747 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 3001. Adjusted session timeout ms: 3000
2021-11-24 18:06:41,868 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:06:41,871 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 14124. Adjusted session timeout ms: 3000
2021-11-24 18:06:41,871 main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: LOST
2021-11-24 18:06:41,872 main-EventThread WARN o.a.c.ConnectionState - ConnectionState.handleExpiredSession() : Session expired event received

2021-11-24 18:07:22,002 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed
2021-11-24 18:07:22,104 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:07:22,104 main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:07:22,115 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Got a life cycle exception while initializing service [ServerRegistrationService, Error while checking path existence for path : /types] : {}
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
… 19 common frames omitted
2021-11-24 18:07:22,116 main ERROR KERNEL - MicroKernel.initAll() : Error in Initializing the deployment : RepositoryService. Exception: {}
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
… 19 common frames omitted
2021-11-24 18:07:22,117 main ERROR KERNEL - MicroKernel.main() : Exception in starting up the kernel:
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
… 19 common frames omitted

cjking · November 25, 2021, 1:48am

2021-11-24 18:07:22,115 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Got a life cycle exception while initializing service [ServerRegistrationService, Error while checking path existence for path : /types] : {}

Your edge-routers are failing to connect to Zookeeper. You should check connectivity (routing, firewall rules) from your edge-router nodes to your Zookeeper nodes.

satmce · November 25, 2021, 2:38am

Thanks @cjking

I have done these things already.

sudo systemctl disable firewalld; sudo systemctl stop firewalld

In the cloud also, these are part of same subnet and all vms have private ip & only public ip associated to node01 alone. These is NO network barrier to connect between these nodes.

This is on RHEL 8.0

Any other suggestions please apart from below checks?

[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 apigee]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

I am able to do telnet to all the zookeeper nodes from my node4 & node5.

[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.197 2181
Trying 172.168.1.197…
Connected to 172.168.1.197.
Escape character is ‘^]’.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.15 2181
Trying 172.168.1.15…
Connected to 172.168.1.15.
Escape character is ‘^]’.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.142 2181
Trying 172.168.1.142…
Connected to 172.168.1.142.
Escape character is ‘^]’.
Connection closed by foreign host.
[

[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.197 2181
Trying 172.168.1.197…
Connected to 172.168.1.197.
Escape character is ‘^]’.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.15 2181
Trying 172.168.1.15…
Connected to 172.168.1.15.
Escape character is ‘^]’.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.142 2181
Trying 172.168.1.142…
Connected to 172.168.1.142.
Escape character is ‘^]’.
Connection closed by foreign host.
[

–

Hostlist:

host01 ansible_host=172.168.1.197
host02 ansible_host=172.168.1.15
host03 ansible_host=172.168.1.142
host04 ansible_host=172.168.1.193
host05 ansible_host=172.168.1.192
host06 ansible_host=172.168.1.195
host07 ansible_host=172.168.1.194
host08 ansible_host=172.168.1.191
host09 ansible_host=172.168.1.149

cjking · November 25, 2021, 2:46am

I’d suggest giving the ruok and stat commands from https://docs.apigee.com/api-platform/troubleshoot/diagnostic-logs/zookeeper-problems?hl=en a try, both locally on your ZK nodes and remotely using nc from your router nodes. It’s possible your ZK cluster isn’t healthy.

satmce · November 25, 2021, 3:11am

I have run it on all the vms as suggested.

[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo "ruok" | nc localhost 2181
imok

[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee ]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 15883
Sent: 15883
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208

[root@rhel-apigee9-set2-vm-01 apigee]# echo mntr | nc localhost 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  22
zk_min_latency  0
zk_packets_received     15886
zk_packets_sent 15886
zk_num_alive_connections        3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  36
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0

[root@rhel-apigee9-set2-vm-01 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /0:0:0:0:0:0:0:1:41614[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44810[1](queued=0,recved=2238,sent=2238)
 /172.168.1.197:44610[1](queued=0,recved=13636,sent=13637)

Latency min/avg/max: 0/0/22
Received: 15890
Sent: 15890
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo cons | nc localhost 2181
 /172.168.1.197:44810[1](queued=0,recved=2241,sent=2241,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=47969889,llat=0,minlat=0,avglat=0,maxlat=5)
 /0:0:0:0:0:0:0:1:41656[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44610[1](queued=0,recved=13657,sent=13658,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2d8a,lzxid=0xffffffffffffffff,lresp=47974178,llat=0,minlat=0,avglat=0,maxlat=22)

----------------------------------------------------------------------------------
[root@rhel-apigee9-set2-vm-02 apigee]# echo ruok | nc localhost 2181
imok

[root@rhel-apigee9-set2-vm-02 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208

[root@rhel-apigee9-set2-vm-02 apigee]# echo mntr | nc localhost 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     3
zk_packets_sent 2
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   34
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0

[root@rhel-apigee9-set2-vm-02 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /0:0:0:0:0:0:0:1:37784[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo cons | nc localhost 2181
 /0:0:0:0:0:0:0:1:37786[0](queued=0,recved=1,sent=0)

----------------------------------------------------------------------------------
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo ruok | nc localhost 2181
imok

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo mntr | nc localhost 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     3
zk_packets_sent 2
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
zk_followers    2
zk_synced_followers     2
zk_pending_syncs        0
zk_last_proposal_size   36
zk_max_proposal_size    1136
zk_min_proposal_size    32

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /0:0:0:0:0:0:0:1:37480[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo cons | nc localhost 2181
 /0:0:0:0:0:0:0:1:37482[0](queued=0,recved=1,sent=0)

[apigee@rhel-apigee9-set2-vm-03 ~]$

----------------------------------------------------------------------------------
FROM NODE4:(edge-router machine)
--------------------------------
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.197 2181
imok[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16224
Sent: 16224
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.197 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  22
zk_min_latency  0
zk_packets_received     16225
zk_packets_sent 16225
zk_num_alive_connections        3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  36
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.197:44810[1](queued=0,recved=2290,sent=2290)
 /172.168.1.193:36732[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44610[1](queued=0,recved=13910,sent=13911)

Latency min/avg/max: 0/0/22
Received: 16226
Sent: 16226
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.197 2181
 /172.168.1.197:44810[1](queued=0,recved=2290,sent=2290,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48623605,llat=0,minlat=0,avglat=0,maxlat=5)
 /172.168.1.193:36734[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44610[1](queued=0,recved=13910,sent=13911,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e5b,lzxid=0xffffffffffffffff,lresp=48634177,llat=1,minlat=0,avglat=0,maxlat=22)

[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.15 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     8
zk_packets_sent 7
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   34
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.193:57486[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.15 2181
 /172.168.1.193:57488[0](queued=0,recved=1,sent=0)

[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.142 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     8
zk_packets_sent 7
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
zk_followers    2
zk_synced_followers     2
zk_pending_syncs        0
zk_last_proposal_size   36
zk_max_proposal_size    1136
zk_min_proposal_size    32
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.193:42672[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.142 2181
 /172.168.1.193:42674[0](queued=0,recved=1,sent=0)

------------------
From Node5(edge-router):
========================

[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.197 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16198
Sent: 16198
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.197 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  22
zk_min_latency  0
zk_packets_received     16199
zk_packets_sent 16199
zk_num_alive_connections        3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  36
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.192:59548[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44810[1](queued=0,recved=2289,sent=2289)
 /172.168.1.197:44610[1](queued=0,recved=13890,sent=13891)

Latency min/avg/max: 0/0/22
Received: 16200
Sent: 16200
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.197 2181
 /172.168.1.197:44810[1](queued=0,recved=2289,sent=2289,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48610261,llat=0,minlat=0,avglat=0,maxlat=5)
 /172.168.1.192:59550[0](queued=0,recved=1,sent=0)
 /172.168.1.197:44610[1](queued=0,recved=13890,sent=13891,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e48,lzxid=0xffffffffffffffff,lresp=48614211,llat=0,minlat=0,avglat=0,maxlat=22)

[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.15 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     13
zk_packets_sent 12
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   34
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.192:50260[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.15 2181
 /172.168.1.192:50262[0](queued=0,recved=1,sent=0)

[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.142 2181
zk_version      3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     13
zk_packets_sent 12
zk_num_alive_connections        1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  208
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        16188
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    262144
zk_fsync_threshold_exceed_count 0
zk_followers    2
zk_synced_followers     2
zk_pending_syncs        0
zk_last_proposal_size   36
zk_max_proposal_size    1136
zk_min_proposal_size    32
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /172.168.1.192:58846[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.142 2181
 /172.168.1.192:58848[0](queued=0,recved=1,sent=0)

satmce · November 25, 2021, 3:27am

Hi @cjking

Thanks for helping me to narrow down to issue. Please find the below diagnostic information on same. please look and suggest anything to be done here.

=================================

[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo “ruok” | nc localhost 2181
imok

[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 15883
Sent: 15883
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208

[root@rhel-apigee9-set2-vm-01 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 15886
zk_packets_sent 15886
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0

[root@rhel-apigee9-set2-vm-01 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:416140
/172.168.1.197:448101
/172.168.1.197:446101

Latency min/avg/max: 0/0/22
Received: 15890
Sent: 15890
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo cons | nc localhost 2181
/172.168.1.197:448101
/0:0:0:0:0:0:0:1:416560
/172.168.1.197:446101

[root@rhel-apigee9-set2-vm-02 apigee]# echo ruok | nc localhost 2181
imok

[root@rhel-apigee9-set2-vm-02 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208

[root@rhel-apigee9-set2-vm-02 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0

[root@rhel-apigee9-set2-vm-02 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:377840

Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:377860

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo ruok | nc localhost 2181
imok

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32

[apigee@rhel-apigee9-set2-vm-03 ~]$ echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:374800

Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:374820

[apigee@rhel-apigee9-set2-vm-03 ~]$

FROM NODE4:(edge-router machine)

[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.197 2181
imok[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16224
Sent: 16224
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16225
zk_packets_sent 16225
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.197:448101
/172.168.1.193:367320
/172.168.1.197:446101

Latency min/avg/max: 0/0/22
Received: 16226
Sent: 16226
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:448101
/172.168.1.193:367340
/172.168.1.197:446101

[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:574860

Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.15 2181
/172.168.1.193:574880

[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:426720

Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.142 2181
/172.168.1.193:426740

From Node5(edge-router):

[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.197 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16198
Sent: 16198
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16199
zk_packets_sent 16199
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:595480
/172.168.1.197:448101
/172.168.1.197:446101

Latency min/avg/max: 0/0/22
Received: 16200
Sent: 16200
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:448101
/172.168.1.192:595500
/172.168.1.197:446101

[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:502600

Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.15 2181
/172.168.1.192:502620

[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:588460

Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.142 2181
/172.168.1.192:588480

================================

satmce · November 25, 2021, 4:15pm

I have also created a case 1508350

Following up with support team on this. Even I have updated this thread, but didn’t see my update.

I have also followed the procedure you have given and attached all the diagnostic logs in the case.

Denis_KALITVI · November 27, 2021, 9:58pm

Error while checking path existence for path : /types] : {}

This is pretty 100% problem connecting to ZK. Are you using Apigee mTLS?

satmce · November 28, 2021, 4:59pm

I haven’t used any mTLS for this setup. its failing during the installation itself.

satmce · November 29, 2021, 8:06pm

Thanks for your time and help in answering my question @Denis_KALITVI @cjking

This is the root cause of the issue:

I had to make below settings on all the apigee nodes for the installation wait for the port to be available and installation to be succeeded.
echo “PORT_TIMEOUT=300” > /opt/apigee/customer/defaults.sh; chown apigee:apigee /opt/apigee/customer/defaults.sh

Once you run the bootstrap and install command, update below timeout settings on respective service nodes according to your installation topology you setup, before deploying any apigee service. This will solve the issue.

This can be reproduced at RHEL setup(using RHEL 8.0 from azure cloud marketplace vm) . I didn’t face this issue in CentOS. I think some fix is needed in the Product installation script.

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/management-server.properties; chown apigee:apigee /opt/apigee/customer/application/management-server.properties

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/message-processor.properties;chown apigee:apigee /opt/apigee/customer/application/message-processor.properties

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/router.properties; chown apigee:apigee /opt/apigee/customer/application/router.properties

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/qpid-server.properties; chown apigee:apigee /opt/apigee/customer/application/qpid-server.properties

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/postgres-server.properties; chown apigee:apigee /opt/apigee/customer/application/postgres-server.properties

echo “conf_zookeeper_session.timeout=300000” > /opt/apigee/customer/application/postgresql.properties; chown apigee:apigee /opt/apigee/customer/application/postgresql.properties

Topic		Replies	Views
OnPrem Edge 4.16.01 - edge-router: Not running (DEAD) Apigee private-cloud-deployment	4	1	October 1, 2018
OPDK 4.16.05- Environment not coming up on port Apigee api-runtime	5	2	August 2, 2016
Install Router and Message Processor on node Apigee private-cloud-deployment	7	4	November 4, 2020

Stuck with error in edge router - Checking for router on 127.0.0.1 port 8081

FROM NODE4:(edge-router machine)

From Node5(edge-router):

AI Suggested topics