Hey Tejpal, I think there is a limit on Apigee payload size to 10MB or less.
So probably, changing the execution time isn’t the right solution to your goal of handling payloads larger than 10mb. If you have large payloads commonly, then I would suggest a different architecture and flow path:
- client calls Apigee and sends a request-to-submit. Informing Apigee that it wants to upload a large SOAP/MTOM message.
- Apigee proxy generates a signed URL (via something like this), sends THAT to the client
- the client uploads that large SOAP/MTOM payload to a GCS bucket.
- a separate process or function gets kicked off to process that message. A Good option is Google Cloud Functions triggered via storage finalization. (doc here)
- The cloud function then processes the message and does whatever is necessary
This way, the Apigee API proxy never handles the 10+mb payload. It simply generates a signed URL and allows the Google Cloud storage to ingest the larger payloads.
But you cited a payload size of around 4mb, not 10mb If your Java callout is failing for some other reason on a payload of ~4mb, then you will want to diagnose that separately. What’s the failure? etc etc
@dchiesa1 : I debugged the issue and found the In XML payload, they are sending content-ID where special character (@) has been replaced with encoding (%40) in XOP object. Like below:
But in the Attachment section, they are passing content-ID without any encoding like below:
Content-ID - 2c785f26-9d7d-4388-a66e-c31d9fda1d8a@ACORD.org
So we are not able to fetch the attachment value as part 1 payload cid has encoded special character and part has special character not encoded.
So is it possible to accommodate changes in Callout where we can read encoded character or without encoded version and able to read the attachment?
Sure thing, Tejpal
Try the newest version of the JAR (20220706). It first tries to locate the element in the XML with the unencoded form of the content id (foobar@apache.org), and if that does not find an element, then it tries to find the element in the XML with the encoded form (foobar%40apache.org).
See if it works for you.
You will need to change your JavaCallout policies to reference the newer version of the jar (apigee-custom-xop-handler-20220706.jar)
@dchiesa1 Thanks you for your reply. I tried with new callout, but its still giving same error: java.lang.IllegalStateException: no matching xop:include object found in the document (href=‘cid:5d31ab1f59579aad895dc7a32d310@apache.org’)
I can see the logic for encoder, but still its not able to fetch compare the contentID. I tried with decoder as well, but same issue. Can you please help me to figure out the issue?
String urlEncodedContentId = URLEncoder.encode(contentId, StandardCharsets.UTF_8.name());
includeElement = findIncludeElement.apply(urlEncodedContentId);
if (includeElement == null) {
throw new IllegalStateException(
String.format(
“no matching xop:Include element in the XML document (href=‘cid:%s’)”,
contentId));
}
Content-Type: multipart/related; type=“application/xop+xml”; start=“<http://tempuri.org/0/>”; start-info=“text/xml”; boundary=“8997b0ab-852a-406f-bc43-a8b48321efa5+id=20”
(Please ignore the soap validation)
–8997b0ab-852a-406f-bc43-a8b48321efa5+id=20
Content-Type: application/xop+xml; charset=utf-8; type=“text/xml”
Content-Transfer-Encoding: binary
Content-ID: <http://tempuri.org/0/>
soapenv:Body
<ns3:process
xmlns:ns3=“http://abc.com/”
xmlns:ns2=“http://ACORD.org/Standards/Life/2”>
ns2:withMessg
ns2:Life
<ns2:LifeRequest PrimaryObjectID=“An7485”>
<ns2:SupportMultipleResponsesInd tc=“1”>True</ns2:SupportMultipleResponsesInd>
ns2:OLifE
<ns2:FormInstance id=“FormInstance_477364”>
ns2:FormNameRequired Forms For Client Signature</ns2:FormName>
ns2:ProviderFormNumberPDFmanifest</ns2:ProviderFormNumber>
<ns2:Attachment id=“xx4704”>
ns2:DateCreated2022-02-10</ns2:DateCreated>
<ns2:AttachmentBasicType tc=“3”>File</ns2:AttachmentBasicType>
ns2:AttachmentData64Binary
<xop:Include href=cid:5d31ab1f59579aad895dc7a32d310%40apache.org
xmlns:xop=http://www.w3.org/2004/08/xop/include/>
</ns2:AttachmentData64Binary>
<ns2:AttachmentType tc=“1”>Document</ns2:AttachmentType>
<ns2:MimeTypeTC tc=“17”>File/PDF</ns2:MimeTypeTC>
<ns2:TransferEncodingTypeTC tc=“4”>Base 64</ns2:TransferEncodingTypeTC>
<ns2:AttachmentLocation tc=“5”>XOP Include</ns2:AttachmentLocation>
ns2:AttachmentHashValueabc</ns2:AttachmentHashValue>
<ns2:AttachmentHashType tc=“2”>2</ns2:AttachmentHashType>
</ns2:Attachment>
<ns2:Attachment id=“xx4705”>
ns2:DateCreated2022-02-10</ns2:DateCreated>
<ns2:AttachmentBasicType tc=“3”>File</ns2:AttachmentBasicType>
ns2:AttachmentData64Binary
<xop:Include href=cid:5d314343570%40apache.org
xmlns:xop=http://www.w3.org/2004/08/xop/include/>
</ns2:AttachmentData64Binary>
<ns2:AttachmentType tc=“1”>Document</ns2:AttachmentType>
<ns2:MimeTypeTC tc=“17”>File/PDF</ns2:MimeTypeTC>
<ns2:TransferEncodingTypeTC tc=“4”>Base 64</ns2:TransferEncodingTypeTC>
<ns2:AttachmentLocation tc=“5”>XOP Include</ns2:AttachmentLocation>
ns2:AttachmentHashValueabc</ns2:AttachmentHashValue>
<ns2:AttachmentHashType tc=“2”>2</ns2:AttachmentHashType>
</ns2:Attachment>
</ns2:FormInstance>
</ns2:OLifE>
</ns2:LifeRequest>
</ns2:Life>
</ns2:withMessg>
</ns3:process>
</soapenv:Body>
–8997b0ab-852a-406f-bc43-a8b48321efa5+id=20
Content-Type: application/pdf
Content-Transfer-Encoding: binary
Content-ID:
5d31ab1f59579aad895dc7a32d310@apache.org
%PDF-1.7 %���� 1 0 obj <
</Type/Catalog/Pages 2 0 R/Lang(en-US) /StructTreeRoot 10 0 R/MarkInfo<
</Marked true>>/Metadata 20 0 R/ViewerPreferences 21 0 R>> endobj 2 0 obj <
</Type/Pages/Count 1/Kids[ 3 0 R] >> endobj 3 0 obj <undefined</Type/Page/Parent 2 0 R/Resources<undefined</Font<undefined</F1 5 0 R>>/ExtGState<undefined</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] yt;��~|�7����Z�����_9}6D�q�n�������o�mm��n�}p��m.�kx��A��@��O8�I����Y��|7;q�{˥V��ݯ��5C����? �B�:�����KYL^M]�3վ>������pyD�Ũ���|Vs1i��5帨I�I5Eo� �&N?�+� ڇ�O���"��_�XB%fr S}4��Ц�� ��C���[�{���ϒ�a�a- �� endstream endobj 20 0 obj <undefined</Type/Metadata/Subtype/XML/Length 3088>> stream undefined<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>undefined<x:xmpmetaundefinedxmlns:x=“adobe:ns:meta/” x:xmptk=“3.1-701”>undefinedrdf:RDFundefinedxmlns:rdf="[http://www.w3.org/1999/02/22-rdf-syntax-ns#](http://www.w3.org/1999/02/22-rdf-syntax-ns#)"undefined<rdf:Description rdf:about=““undefinedxmlns:pdf=“http://ns.adobe.com/pdf/1.3/”>undefinedpdf:ProducerMicrosoft® Word for Microsoft 365</pdf:Producer>undefined</rdf:Description>undefined<rdf:Description rdf:about=”” x %%EOF
–8997b0ab-852a-406f-bc43-a8b48321efa5+id=20
Content-Type: application/pdf
Content-Transfer-Encoding: binary
Content-ID: undefined5d314343570@apache.org
%PDF-1.7 %���� 1 0 obj <undefined</Type/Catalog/Pages 2 0 R/Lang(en-US) /StructTreeRoot 10 0 R/MarkInfo<undefined</Marked true>>/Metadata 20 0 R/ViewerPreferences 21 0 R>> endobj 2 0 obj <undefined</Type/Pages/Count 1/Kids[ 3 0 R] >> endobj 3 0 obj <undefined</Type/Page/Parent 2 0 R/Resources<undefined</Font<undefined</F1 5 0 R>>/ExtGState<undefined</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] yt;��~|�7����Z�����_9}6D�q�n�������o�mm��n�}p��m.�kx��A��@��O8�I����Y��|7;q�{˥V��ݯ��5C����? �B�:�����KYL^M]�3վ>������pyD�Ũ���|Vs1i��5帨I�I5Eo� �&N?�+� ڇ�O���"��_�XB%fr S}4��Ц�� ��C���[�{���ϒ�a�a- �� endstream endobj 20 0 obj <undefined</Type/Metadata/Subtype/XML/Length 3088>> stream undefined<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>undefined<x:xmpmetaundefinedxmlns:x=“adobe:ns:meta/” x:xmptk=“3.1-701”>undefinedrdf:RDFundefinedxmlns:rdf="[http://www.w3.org/1999/02/22-rdf-syntax-ns#](http://www.w3.org/1999/02/22-rdf-syntax-ns#)"undefined<rdf:Description rdf:about=““undefinedxmlns:pdf=“http://ns.adobe.com/pdf/1.3/”>undefinedpdf:ProducerMicrosoft® Word for Microsoft 365</pdf:Producer>undefined</rdf:Description>undefined<rdf:Description rdf:about=”” x %%EOF
–8997b0ab-852a-406f-bc43-a8b48321efa5+id=20–
I understand what you wrote and what you’re observing. There is a test for exactly this case in the callout. The existing test passes with the new version. If I copy the data you provided above, and use that as input to the test, the test still passes. It works for me.
There’s something that confuses me about the example data you are sharing. All of the XML looks wellformed, EXCEPT for the xop:Include element. Here’s an excerpt from your sample data:
<ns2:Attachment id="xx4704">
<ns2:DateCreated>2022-02-10</ns2:DateCreated>
<ns2:AttachmentBasicType tc="3">File</ns2:AttachmentBasicType>
<ns2:AttachmentData64Binary>
<xop:Include href=cid:5d31ab1f59579aad895dc7a32d310%40apache.org
xmlns:xop=http://www.w3.org/2004/08/xop/include/>
</ns2:AttachmentData64Binary>
<ns2:AttachmentType tc="1">Document</ns2:AttachmentType>
<ns2:MimeTypeTC tc="17">File/PDF</ns2:MimeTypeTC>
<ns2:TransferEncodingTypeTC tc="4">Base 64</ns2:TransferEncodingTypeTC>
<ns2:AttachmentLocation tc="5">XOP Include</ns2:AttachmentLocation>
<ns2:AttachmentHashValue>abc</ns2:AttachmentHashValue>
<ns2:AttachmentHashType tc="2">2</ns2:AttachmentHashType>
</ns2:Attachment>
Every attribute in that snippet is quoted, EXCEPT for the attributes on the xop:Include element. The ns1:Attachment element has an attribute named id , and the value is xx4704, in double quotes. The ns2:AttachmentBasicType has a tc attribute which has a quoted value. Every attribute has a quoted value. Except for the href and the xmlns:xop on the xop:Include element. Do you have any idea why that would be?
I’m sure the data is not exactly as I see it above in this forum, because attributes without quotes are not well-formed XML, and the callout would have returned a different error if it encountered this (really without quotes):
<xop:Include href=cid:5d31ab1f59579aad895dc7a32d310%40apache.org
xmlns:xop=http://www.w3.org/2004/08/xop/include />
It needs to be something like this:
<xop:Include href='cid:5d31ab1f59579aad895dc7a32d310%40apache.org'
xmlns:xop='http://www.w3.org/2004/08/xop/include' />
Either single or double quotes are ok.
I’m sure your document has quotes, somehow. But I’m wondering if the encoded @ sign (%40) is related to the lack of quotes. Are we really clear on what the document holds? Is it possible for you to ZIP the document and attach the zip file here in a response? If you can do that we can be assured that there is no issue with fidelity associated to the copy/paste into the message body here. If you do that, I could run the test on exactly the data you have on your side.
@dchiesa1 : Sorry I missed the double quotes. I trying to upload the file, but not getting any option here to upload. Also If I put the complete payload, its not taking. Below will be the snippet for xop include. If I try without encoded content-Id, its working but If I put encoded content-Id in xml, its not working. please check:
<xop:Include
xmlns:xop=“http://www.w3.org/2004/08/xop/include”
href=“cid:5d31ab1f59579aad895dc7a32d310%40apache.org”>
</xop:Include>
** @dchiesa1 : I tried everything, but I am niether able to paste complete payload- nor attach it. I just tried to put the xml snippet. Not sure how much it can help, but Its working where url encode is not there (both the places its use @) but with encode its not working**
--8997b0ab-852a-406f-bc43-a8b48321efa5+id=20
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
Content-Transfer-Encoding: binary
Content-ID: <http://tempuri.org/0/>
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header>
<ch:MessageHeader
xmlns:ch="http://abc.com/">
<ch:MessageName>Withdrawal</ch:MessageName>
<ch:RouteInfo>
<ch:FromParticipant>0057</ch:FromParticipant>
<ch:ToParticipant>4597</ch:ToParticipant>
<ch:Sender>EDWARD_JONES</ch:Sender>
<ch:TransmissionDate>2022-04-07</ch:TransmissionDate>
<ch:TransmissionTime>11:32:03.0000000-04:00</ch:TransmissionTime>
</ch:RouteInfo>
</ch:MessageHeader>
</soapenv:Header>
<soapenv:Body>
<ns3:processWithdrawal
xmlns:ns3="http://abc.com/"
xmlns:ns2="http://ACORD.org/Standards/Life/2">
<ns2:Withdrawal_Msg>
<ns2:TXLife>
<ns2:TXLifeRequest PrimaryObjectID="AnnuityOrder757238">
<ns2:TransRefGUID>0369e970-e4c2-452c-894e-aa34aba3c428</ns2:TransRefGUID>
<ns2:TransType tc="105">Withdrawal</ns2:TransType>
<ns2:TransExeDate>2022-04-07</ns2:TransExeDate>
<ns2:TransExeTime>11:32:04-04:00</ns2:TransExeTime>
<ns2:TransMode tc="2">Original request. Sent by sender.</ns2:TransMode>
<ns2:PendingResponseOK tc="1">True</ns2:PendingResponseOK>
<ns2:SupportMultipleResponsesInd tc="1">True</ns2:SupportMultipleResponsesInd>
<ns2:OLifE>
<ns2:Holding id="AnnuityOrder757238">
<ns2:HoldingTypeCode tc="2">Policy</ns2:HoldingTypeCode>
<ns2:DistributorClientAcctNum>561651</ns2:DistributorClientAcctNum>
<ns2:Policy CarrierPartyID="Party_Carrier">
<ns2:PolNumber>E1885415</ns2:PolNumber>
<ns2:CarrierCode>79227</ns2:CarrierCode>
<ns2:CusipNum>74429E164</ns2:CusipNum>
</ns2:Policy>
</ns2:Holding>
<ns2:Party id="Party_Distributor">
<ns2:PartyTypeCode tc="2">Organization</ns2:PartyTypeCode>
</ns2:Party>
<ns2:Party id="Party_Carrier">
<ns2:PartyTypeCode tc="2">Organization</ns2:PartyTypeCode>
<ns2:Carrier>
<ns2:CarrierCode>79227</ns2:CarrierCode>
</ns2:Carrier>
</ns2:Party>
<ns2:Party id="Party_Agent">
<ns2:PartyTypeCode tc="1">Person</ns2:PartyTypeCode>
<ns2:Person>
<ns2:FirstName>HOPAN</ns2:FirstName>
<ns2:LastName>DANDRIDGE</ns2:LastName>
</ns2:Person>
<ns2:Producer>
<ns2:NIPRNumber>34567891</ns2:NIPRNumber>
</ns2:Producer>
<ns2:PartialIdentification>
<ns2:IdentificationPart>8241</ns2:IdentificationPart>
<ns2:PartialIDType tc="1">SSN</ns2:PartialIDType>
</ns2:PartialIdentification>
</ns2:Party>
<ns2:Party id="Party_PrimaryOwner">
<ns2:PartyTypeCode tc="1">Person</ns2:PartyTypeCode>
<ns2:Person>
<ns2:FirstName>Test</ns2:FirstName>
<ns2:LastName>Owner</ns2:LastName>
</ns2:Person>
</ns2:Party>
<ns2:FormInstance id="FormInstance_486844">
<ns2:FormName>Required Forms For Client Signature</ns2:FormName>
<ns2:ProviderFormNumber>PDFmanifest</ns2:ProviderFormNumber>
<ns2:Attachment id="xx486844">
<ns2:DateCreated>2022-04-07</ns2:DateCreated>
<ns2:AttachmentBasicType tc="3">File</ns2:AttachmentBasicType>
<ns2:AttachmentData64Binary>
<xop:Include
xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:5d31ab1f59579aad895dc7a32d31d3c859ddf4343570fc10%40apache.org">
</xop:Include>
</ns2:AttachmentData64Binary>
<ns2:AttachmentType tc="1">Document</ns2:AttachmentType>
<ns2:MimeTypeTC tc="17">File/PDF</ns2:MimeTypeTC>
<ns2:TransferEncodingTypeTC tc="4">Base 64</ns2:TransferEncodingTypeTC>
<ns2:AttachmentLocation tc="5">XOP Include</ns2:AttachmentLocation>
<ns2:AttachmentHashValue>2352FB48E31460C789C6822926F6B728C85A70B5</ns2:AttachmentHashValue>
<ns2:AttachmentHashType tc="2">2</ns2:AttachmentHashType>
</ns2:Attachment>
</ns2:FormInstance>
</ns2:OLifE>
</ns2:TXLifeRequest>
</ns2:TXLife>
</ns2:Withdrawal_Msg>
</ns3:processWithdrawal>
</soapenv:Body>
</soapenv:Envelope>
--8997b0ab-852a-406f-bc43-a8b48321efa5+id=20
Content-Type: application/pdf
Content-Transfer-Encoding: binary
Content-ID: <5d31ab1f59579aad895dc7a32d31d3c859ddf4343570fc10@apache.org>
%PDF-1.7 %%EOF
--8997b0ab-852a-406f-bc43-a8b48321efa5+id=20--
Hi thanks for that ^^
I just ran more tests with that example, and still the tests pass.
Are you certain you have the latest version of the callout? apigee-custom-xop-handler-20220706.jar | 20182 bytes
Hi @dchiesa1 : Thanks for you reply. I think I was doing something wrong. I replace all over again new callout as it is and its working fine now.
Thanks you soo much for your help as always. Really Appreciate.