router not restarting after VHOST updates

Google Cloud Apigee
1

Hi All,

We are having issue with router not starting whenever a change is done to VHOST configs. I checked the logs for router and could find below errors/warning whenever router was restarted/started.

2017-06-14 11:26:31,401  WebServer INFO  c.a.g.f.v.EntityCRUDInterceptor - EntityCRUDInterceptor.<clinit>() : Failed to retrieve security properties with the exception {}
java.lang.NullPointerException: null
        at java.util.Properties$LineReader.readLine(Properties.java:434) ~[na:1.8.0_121]
        at java.util.Properties.load0(Properties.java:353) ~[na:1.8.0_121]
        at java.util.Properties.load(Properties.java:341) ~[na:1.8.0_121]
        at com.apigee.gateway.flow.validation.EntityCRUDInterceptor.<clinit>(EntityCRUDInterceptor.java:74) ~[validation-1.0.0.jar:na]
        at java.lang.Class.forName0(Native Method) [na:1.8.0_121]
        at java.lang.Class.forName(Class.java:264) [na:1.8.0_121]
        at com.apigee.rest.framework.CustomJAXRSInvoker.<init>(CustomJAXRSInvoker.java:78) [rest-1.0.0.jar:na]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_121]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_121]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_121]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_121]
        at java.lang.Class.newInstance(Class.java:442) [na:1.8.0_121]
        at com.apigee.rest.framework.container.RestServlet.init(RestServlet.java:51) [rest-1.0.0.jar:na]
        at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:456) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:276) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]

and

2017-06-14 11:26:31,441  main WARN  REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts
2017-06-14 11:26:31,845  main INFO  REGISTRATION - ServerRegistrationServiceImpl.storeServerRegistrationPath() : Registering the server uuid aa834eaf-8a34-4098-8f80-e66b406260f4 with region dc-1 and pod gateway information
2017-06-14 11:26:31,888  WebServer WARN  o.e.j.u.c.AbstractLifeCycle - AbstractLifeCycle.setFailed() : FAILED SelectChannelConnector@0.0.0.0:8081 FAILED: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_121]
        at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_121]
        at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_121]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_121]
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_121]
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:172) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:300) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:249) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.server.Server.doStart(Server.java:273) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
        at com.apigee.rest.framework.container.Container.start(Container.java:78) [rest-1.0.0.jar:na]
        at com.apigee.web.WebServiceImpl$1.run(WebServiceImpl.java:39) [webserver-1.0.0.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

For “Address already in use” issue I can guess that it is to do with some ports already listening on the ports which router is trying to listen. I checked netstat for all listening ports and could find the old port for VHOST listening. Shouldn’t the restart take care of unbinding old and binding new ports?

Could anyone please help me understand these two exception and the cause of it?

Thanks…

2

Looks like the routers stop is not happening completely.
Can you try apigee-service edge-router stop and check for ps -ef | grep router and start it when no process is running.

When the router starts properly, I think the first might go away. @Mohammed Zuber

1 Like
3

@Maruti Chand, you are correct apigee-service edge-router stop is not stopping the router.

bash:$ sudo /opt/apigee/apigee-service/bin/apigee-service edge-router stop
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Server is already stopped.
apigee-service: edge-router: Not running (DEAD)

But ps -ef | grep router is still showing that router is running.

bash:$ ps -ef | grep router
root     17435     1  1 Jun09 ?        01:56:31 /usr/java/latest/bin/java -classpath /opt/apigee/edge-router-4.16.09-0.0.891/conf::/opt/apigee/edge-router-4.16.09-0.0.891/lib//*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//actions/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//cps/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/steps/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//kernel/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//thirdparty/*:/opt/apigee/edge-router-4.16.09-0.0.891/webapps/api/:/opt/apigee/edge-router-4.16.09-0.0.891/utils/scripts/sampling/:/opt/apigee/edge-router-4.16.09-0.0.891/utils/scripts/utilities/ -Xms256m -Xmx512m -XX:MaxPermSize=128m -XX:MetaspaceSize=128m -Djava.security.auth.login.config=/opt/apigee/edge-router-4.16.09-0.0.891/conf/jaas.config -Dinstallation.dir=/opt/apigee/edge-router-4.16.09-0.0.891 -Dprofile=router -Dlib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services -Dpython.verbose=error -Ddebugsession.timeout=120 -Ddefault.response.format=json -Dauthenticated.user.header.name=X-Apigee-Current-User -Dbundle.validation.enabled=false -Dbundle.validation.schema.enabled=true -Dcasssandra.maxConnectTimeInMillis=-1 -Drbac.number.of.tries.for.email.search=1 -Dvirtual.host.restricted.ports=8080 -Djavacallout.policy.validate=false -Dastyanax.datastores=taurus-datastore,kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore,apimodel-datastore,auth-datastore,scheduler-datastore,edgenotification-datastore -Djson.request.schema.validation.enabled=true -Djavax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl -Dvalidation.entity.name.enabled=true -Dcheckmarx.scan.validator.enabled=true -Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl -Dvalidation.targetendpoint.connection.check.enabled=true -DshutdownServicesOnStop=true -Dsun.net.maxDatagramSockets=2048 -Dhas.monetization=${microkernel_hasMonetization} -Dmigration.mode.status=false -Djute.maxbuffer=10485750 -Dqpid.session.command_limit=524288 -Dqpid.session.byte_limit=8388608 -Djava.io.tmpdir=/var/tmp -Dlog.level=INFO -Dregion=dc-1 -Dpod=gateway -Djsse.enableSNIExtension=false -Dcom.warrenstrange.googleauth.rng.algorithm=SHA1PRNG -Dcom.warrenstrange.googleauth.rng.algorithmProvider=SUN -Djdk.tls.allowUnsafeServerCertChange=true -Djava.security.properties=../conf/jvmsecurity.properties -Djava.security.nssprovider.enabled=false -Dbean.retry.enabled=true -DuseStringDeduplication=false -Dlicense.file.location=/opt/apigee/customer/conf/license.txt -Dconf.dir=/opt/apigee/edge-router-4.16.09-0.0.891/conf -Ddata.dir=/opt/apigee/var/log/edge-router com.apigee.kernel.MicroKernel

How can I stop the router properly?

4

Looks like some issues with your installation. Your router is running as root user but it should be apigee. check this http://docs.apigee.com/private-cloud/latest/installation-requirements#creatingtheapigeeuser.
Can you go through that link and check if you took care of all the pre-requisites? @Mohammed Zuber

5

Thanks, @Maruti Chand, issue was with root user owning the processes related to apigee.

To fix this I had to forcefully kill the router process owned by root and restart it with the normal sudoer user under apigee user with apigee-service.

6

@Mohammed Zuber thank you for sharing the solution.

Please make sure /opt/apigee (all files and folders) are owned by apigee:apigee