java.io.IOException: Broken pipe issue when target sends large response

Google Cloud Apigee
1

We were getting Gateway Timeout issue when target backend sends large response.

Target back-end API is working fine without gateway.

So we updated below properties

Router

conf_router_ServerContainer.io.timeout.millis=500000

conf_router_Client.pool.iotimeout=450000

conf_router_Client.pool.connection.timeout=450000

MP

conf_http_HTTPTransport.io.timeout.millis=400000

conf_http_HTTPClient.connect.timeout.millis=20000

We are still getting the same issue but now we have below error in system.log on MP.

Also find the attached zip containing logs, tcpdump, client error screenshot and the trace of the proxy on gateway.

Also we have APIGEE setup with AIO profile.

Please take a look on priority and thanks in advance

2016-07-16 08:47:53,949 org:kronos env:test api:legacy_api-v1-BatchService rev:1 messageid:keng01-dev01-api72-gtw72.dev.mykronos.internal-22125-362-1  NIOThread@0 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.3.114.25:8998 Remote host:10.3.114.25:41252]@257 useCount=1 bytesRead=0 bytesWritten=1193 age=57410ms  lastIO=0ms ))
2016-07-16 08:47:53,950 org:kronos env:test api:legacy_api-v1-BatchService rev:1 messageid:keng01-dev01-api72-gtw72.dev.mykronos.internal-22125-362-1  NIOThread@0 INFO  HTTP.SERVICE - ExceptionHandler.handleException() : Exception trace: 
java.io.IOException: Broken pipe
at com.apigee.nio.channels.ClientOutputChannel.writePending(ClientOutputChannel.java:51) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.OutputChannel.onWrite(OutputChannel.java:116) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.OutputChannel.write(OutputChannel.java:81) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.ChunkedOutputChannel.doWritePending(ChunkedOutputChannel.java:43) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.FilterOutputChannel.writePending(FilterOutputChannel.java:37) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.OutputChannel.onWrite(OutputChannel.java:116) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.OutputChannel.write(OutputChannel.java:81) ~[nio-1.0.0.jar:na]
at com.apigee.protocol.http.io.BodyPipe.onWrite(BodyPipe.java:169) [http-1.0.0.jar:na]
at com.apigee.protocol.http.io.BodyPipe.start(BodyPipe.java:100) [http-1.0.0.jar:na]
at com.apigee.protocol.http.io.MessageWriter.onWrite(MessageWriter.java:165) [http-1.0.0.jar:na]
at com.apigee.protocol.http.io.MessageWriter.<init>(MessageWriter.java:110) [http-1.0.0.jar:na]
at com.apigee.protocol.http.HTTPServer$Context$2.<init>(HTTPServer.java:685) [http-1.0.0.jar:na]
at com.apigee.protocol.http.HTTPServer$Context.send(HTTPServer.java:685) [http-1.0.0.jar:na]
at com.apigee.messaging.adaptors.http.configuration.MessageProcessorHttpSkeletonFactory$FlowCompletionListener.onFinish(MessageProcessorHttpSkeletonFactory.java:512) [http-adaptor-1.0.0.jar:na]
at com.apigee.flow.MessageFlowImpl.onCompletion(MessageFlowImpl.java:213) [message-flow-1.0.0.jar:na]
at com.apigee.flow.MessageFlowImpl.resume(MessageFlowImpl.java:175) [message-flow-1.0.0.jar:na]
at com.apigee.flow.execution.ExecutionContextImpl$1.run(ExecutionContextImpl.java:93) [message-flow-1.0.0.jar:na]
at com.apigee.nio.NIOSelector.runTasks(NIOSelector.java:317) [nio-1.0.0.jar:na]
at com.apigee.nio.NIOSelector.access$000(NIOSelector.java:29) [nio-1.0.0.jar:na]
at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:265) [nio-1.0.0.jar:na]
at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:256) [nio-1.0.0.jar:na]
at com.apigee.nio.util.NonNullIterator.computeNext(NonNullIterator.java:16) [nio-1.0.0.jar:na]
at com.apigee.nio.util.AbstractIterator.hasNext(AbstractIterator.java:36) [nio-1.0.0.jar:na]
at com.apigee.nio.handlers.NIOThread.run(NIOThread.java:59) [nio-1.0.0.jar:na]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_79]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_79]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_79]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_79]
at com.apigee.nio.PlainTransport.write(PlainTransport.java:107) ~[nio-1.0.0.jar:na]
at com.apigee.nio.ClientChannel.write(ClientChannel.java:307) ~[nio-1.0.0.jar:na]
at com.apigee.nio.channels.ClientOutputChannel.writePending(ClientOutputChannel.java:47) ~[nio-1.0.0.jar:na]
... 23 common frames omitted
2

Hi,

The problem here is the router timing out. Should be fixed by setting this in /<inst_path>/customer/application/router.properties

conf_load_balancing_load.balancing.driver.proxy.read.timeout=

Default is 57 seconds, so increase to something higher. The value will be determined by how long you believe a target server should take to respond (+ a little time for the Message processor).

Russell

3

HI @GAURAV

Does this work? I am seeing same broken pipe issue in my MP logs and as a result, it returns HTTP 502. I am wondering if this solves your issue.

This is my router log.

2017-09-06T16:59:58+00:00evpla936.XXXX.comIP:36355IP:89980.700–499-28030POST /shopping/v1/products?brandOnly=true HTTP/1.1-Java/1.7.0apigateway.XXXX.comevpla936.XXXX.com-62682-5742263-5310.68.93.164-

And I have below configuration,

conf_load_balancing_load.balancing.driver.proxy.read.timeout=300 conf_load_balancing_load.balancing.driver.server.keepalive.timeout=300 conf_apigee-nginx-logrotate_pidfile=/opt/apigee/edge-router/nginx/logs/nginx.pid conf_http_HTTPTransport.io.timeout.millis=240000

Thanks,

Krish

4

I am having the exact same issue, I would love to know if you could finally find the Broken Pipe root cause.

thanks,

Claudio