Unable to connect to any of the target servers from our org

Google Cloud Apigee
1

All of a sudden, we are getting 503 Service Unavailable errors and we are unable to connect to any of the target servers in all the environments of our organization.

Below is the error:

{ 
  "fault": { 
       "faultstring": "The Service is temporarily unavailable", 
       "detail": { 
             "errorcode": "messaging.adaptors.http.flow.NoActiveTargets" 
          } 
     } 
}
2

  1. Checked and confirmed that there were no changes made from the Edge perspective to customer’s org MPs.

  2. The Message Processor logs showed that there were connection time outs to the target server

2017-02-27 06:49:21,047 org:<org-name> env:<env-name> api:<api-name> rev:<revision> messageid:<message-id> NIOThread@0 ERROR HTTP.CLIENT - HTTPClient$Context.onTimeout() : ClientChannel[C:]@64532 useCount=1 bytesRead=0 bytesWritten=0 age=3000ms lastIO=3000ms .onConnectTimeout connectAddress=<host-name>/<IPaddress>:443 resolvedAddress=<host-name>/<IPaddress>

2017-02-27 08:10:48,665 Apigee-Timer-2 WARN SERVICES.HEALTH_MONITOR - HTTPMonitor.monitor() : Connect Failed. scheme:http Host: <host-name> Port:80 
java.net.SocketTimeoutException: connect timed out 
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_101] 
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_101]
  1. Attempted to telnet to the target server from the Message Processor and got the “Connection timed out” error as shown below:
[adevegowda@MP1 ~]$ telnet <host-name> 80 
Trying <IP-address>... 
telnet: connect to address <IP-address>: Connection timed out 
[adevegowda@MP1 ~]$

  1. This indicated that possibly the IP whitelisting/firewall rules are somehow modified at the customer end. Informed the customer to check the same.

  2. Customer found that their team had removed the IP whitelisting rules while migrating to a newer software due to which the services were inaccessible from Message Processors.

We need to ensure that IP whilelisting rules are retained if we are migrating/upgrading the softwares or making any changes on the target servers so that we don’t cause these kind of errors.