Sporadic issues connecting to a target server, how can I debug?

Google Cloud Apigee
1

I’m using Edge to proxy a connection to one of our partner’s services.

The connection is very straightforward, just like this

	<HTTPTargetConnection>
	  <Properties>
	  <Property name="success.codes">1xx,2xx,3xx,4xx</Property>
	  <Property name="keepalive.timeout.millis">0</Property><!-- I added this to try to fix the issue! -->
	  </Properties>
	  <URL>http://partner.io/api</URL>
	  </HTTPTargetConnection>

The problem is that, when I’m connecting to the partner through the Apigee proxy, with a small sustained load of 8 calls an hour, I get sporadic error 503 responses from the target server - about 8% of the time, seemingly at random. Running a trace, I can see complete details of the HTTP call, and the response from the target server.

However, the partner says that these calls never arrive at their system, and so they haven’t been able to provide me any useful logs. When I try to reproduce this using Runscope Radar to target the partner’s API directly, sending HTTP calls that are as similar as possible to Apigee’s calls (same URL, same keep-alive headers, etc.), I can’t reproduce the issue, everything works great. I will see calls that pass through Apigee fail at the same time that virtually identical calls that go directly to our partner work correctly. Everything points to a problem on Edge.

We’ve tried a few things, like disabling keep-alive (another policy adds the Connection: close header), they’ve added and removed load-balancers, and the issue remains the same. Where can I learn more details about how Edge contacts the target servers? Is there any way to get more detailed debugging information than the proxy trace?

2

@tpearson , Are you on Apigee Cloud / on prem ?

3

Hi @tpearson,

Amadeus is our cloud customer and, I assume, this requirement is for them only. On Apigee cloud, you won’t have access to logs. You have to raise a support ticket to analyse the MP logs.

To troubleshoot this scenario, one option is to have pass-through proxy and connect with the backend. Analyse if target (/partner server) is getting requests or not. Later on, additional policies may be added.

4

Indeed, this is what I’ve just done. I implemented it as a NodeJS proxy on Apigee and with this I can reproduce the same issue. Running the same NodeJS script on local, I have not yet been able to reproduce the issue. Thanks for the tip!

5

I presume you already have a support ticket logged @tpearson . But on other thing that would be useful is to see your partner can enable tcpdumps at the backend that watch out for packets coming in and going out to the Apigee message-processor.

tcpdump -n -w capture.cap

The dump should carry some information on failed connections