Error parsing request body EOFException

Google Cloud Apigee
,
1

We use Apigee Cloud which sends the client request to our Cloud Run service which is a fastapi python server.

We are noticing that < 1% of our POST requests are ending up as Request Validation errors with body being empty. When clients retry the same request, it works.

Things I tried:

During the investigation, I added a new header in Apigee which would copy over the original content-length header into a new custom header and logged the request headers. I noticed that the new header had non-zero length indicating the size of the request. But the actual Content-Length header is 0 and body is empty
As a next step, I added a ExtractVariables policy in Apigee to check what is the actual request body, but I noticed the following:

Error parsing request payload. Cause: java.io.EOFException

I am unable to reproduce the issue outside reliably because it is happening for < 1% of requests and when the clients retry the request it succeeds. Can you please let me know how to debug this further?

2

error message.jpg
error message.jpg2402×662 85.9 KB

Pasting the error message from the ExtractVariables policy as a screenshot here. (Sorry, not a text post as I am getting an error posting in the discussion forum)

3

Upon further debugging, I noticed that the requests which are having problems might be timing out waiting for the body to arrive. The client might be sending the headers first, but is not sending the body?

As you can see in the screenshot below, all of these requests have high latencies and most of the time is spent in constructing the request to send to the target backend, isn’t it?

showing_422.jpg
showing_422.jpg2918×1588 307 KB

4

@venu84 - can you try increasing the timeout in the proxy config? And also see if there are any timeout configured in the CloudRun service as well

What if you hit the CloudRun service directly? Do you see any issues of 422s then? Can you run some tests pointing to it directly and see if it happens. If it does, then you need to tweak the service’s timeout

5

Hi @venu84 , we hope @ssvaidyanathan 's response was helpful!

If so, please consider marking it as the accepted solution. Thanks to Sai for the answer :blush:

For more insights, check out our latest articles here and join our weekly office hours, Thursdays at 4:00 PM CET, register here a :right_arrow: https://rsvp.withgoogle.com/events/apigee-emea-office-hours-2024/home. We cover in-depth topics on Apigee and Google Cloud Application Integration.

6

The root cause of the issue was that we enabled request streaming.

true
When this was happening, sometimes apigee sends an empty body to the target backend. https://cloud.google.com/apigee/docs/api-platform/antipatterns/payload-with-streaming
The issue was fixed after we turned off that property.

The errors I saw when I added an ExtractVariables policy to peek into the body was because when request streaming is enabled, the entire body is not available to the policy and causes the parsing error. Thanks to @PoornimaD for helping out.

1 Like
7

Thanks for the updating it here Venu!