How to support large JSON payloads (upto 25 MB size) in Apigee without streaming

Google Cloud Apigee
1

Hi,

We have a requirement to support large JSON request payload up to 25 MB. This JSON payload can contain base64 encoded string for five attachments with each attachment size upto max 5 MB.

When JSON contains one 5 MB attachment, JSON to XML policy takes long time to execute.

In a JavaScript policy, for each attachment content in the request payload, we are calling an outside service to scan attachments for virus.

When tested with 25 MB JSON payload containing five attachments, we get error “protocol.http.TooBigBody” with http status code 413.

‘http.prppertirs/HTTPResponse.body.buffer.limit’ buffer config is increased upto 10 MB for MessageProcessor components in this on-prem setup.

We know Apigee suggests streaming for large payloads. Our target bac kend system is Service-Now cloud. Service-Now cloud supports payloads upto 25 MB without streaming. We need to support same payload sizes in Apigee without streaming.

Please suggest what are the options available for us to support large payloads upto 25 MB. What configuration changes would be needed in MPs and Routers to be able to support 25 MB payload without streaming? Would this require any changes to JVM configuration in MP and/or Router?

Thanks,

Hemana

2

Hi @Hemana ,

As this is private cloud set-up, http.properties/HTTPResponse.body.buffer.limit can be increased to value as required. Heap memory of each MP instance need to be increased accordingly, it depends on the peak TPS requirement. Note that heap will be consumed for whole payload. As I understand, there is JSON to XML transformation, keep and same e.g. “request” to avoid consuming extra memory space.

Curious to know that how backend service handles when content size is > 25 MB? Most probably, they would be supporting streaming for payload > 25MB. Any specific reason why can’t streaming be applied to lower content size?

Best Regards, Rajesh Doda