We are currently facing an issue in Application Integration where the data payload size exceeds the allowed limit (10MB). Our integration processes a large dataset (around 30MB), and due to this, we are unable to complete data mapping or proceed with execution properly.
We are encountering the following error:
Warning: Database persistence is disabled as the total integration data size exceeds the allowed limit of 10 MB. This limits task execution logs and retry attempts.
Error:
“Data mapping failed. Failed to set the input value on an output variable. Reason: Integration variables exceeded maximum size allowed (approx. 30MB).”
Status: FAILED_PRECONDITION (400)
Has anyone faced a similar limitation in Application Integration when handling large payloads? What are the recommended approaches or best practices to process such large datasets?
Any guidance (e.g., design patterns, payload handling strategies, or architectural suggestions) would be highly appreciated.
Hi @Asjid_Tahir, thanks for sharing your use case and the error messages. While we wait for community feedback, you might find this resource from our knowledge hub helpful:
From what I understand, your payload is like ~30MB but the integration limit is 10MB, so mapping fails and even logs/retries get disabled. I faced something similar a while back when moving large JSON datasets, and it kept breaking exactly like this.
What worked for me (not perfect, but helped):
I split the payload into smaller chunks (like batching 5–8MB) and processed them in loops
Stored raw data in external storage (like DB / object storage) and passed only references in the flow
Reduced unnecessary fields before mapping (surprisingly saved a lot)
Also in one project (I think it was for something like my “Dooflix” data sync), we avoided passing full payloads inside variables completely.
Not sure if it helps but are you able to redesign it as chunk-based processing?
