How to extract file from multipart/form-data to upload to S3?

Google Cloud Apigee
1

Hi,

I need to upload a file to an S3 bucket and the file is sent in a multipart/form-data.

As far as I could understand, the EV policy does not work with Content-Type: multipart/form-data.

Can someone give me some advice on how I can get the file provided in the request so it can be uploaded to S3?

Thanks in advance

2

Instead of form data, I would suggest to use binary option of postman. We have customers who use that option to upload pdf files to there application.

Hope this information will help.

1 Like
3

OK

I’m not quote clear on what you’re trying to do.

You say you want to upload a file to S3… and also … that you want to extract a file from a payload. So I’m not sure how those two things are related. AFAIK, the S3 API does not require a multipart form payload. (I may be wrong, I am not an expert on S3).

You are correct that ExtractVariables does not extract items from multipart forms payloads. Can you check this answer, regarding parsing multipart form data? There is a Java callout that allows you to extract things.

As far as submitting to S3, that’s a separate thing, And I think you’ll need to use a Java callout to sign that request, too. Here’s an option. and here’s another.

4

Hi Dino-at-Google thanks for your help.

Yes the S3 API does not require a multipart form payload but I’ll receive the file and a JSON payload (that is used to do something else), that’s why I want to extract the file so it can be uploaded.

So, if I understood correctly, after the Java callout to parse the multipart form request I’ll have the file as a byte array in a context variable that then I can use in the flow. Having this context variable, I can build a message, sign it using one of the options you provided and then send it to S3. Did I miss anything?

5

Yes, I think you’ve got it.

As for sending that unencoded byte array in the message to S3… I haven’t done something like that.

Apigee often assumes that the content for a request or response is a UTF-8 text payload. If you have a byte array and want to send it to a target you may need to do something like

<AssignMessage name='AM-ByteArrayAsPayload'>
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
  <AssignVariable>
    <Name>request.content</Name>
    <Ref>extracted-byte-array</Ref>
  </AssignVariable>
</AssignMessage>

That might do it.

You may need to attach that policy into the target request flow.

6

Hi Dino-at-Google,

I’ve followed your suggestion and built a simple proxy with no target, imported the jar to parse the multipart form and I can see the variables in the trace.

In the response flow I attached a simple AM policy that looks like this:

<AssignMessage name="AM-Response">
    <DisplayName>AM-Response</DisplayName>
    <AssignVariable>
        <Name>response.content</Name>
        <Ref>mpf_item_content_1</Ref>
    </AssignVariable>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="false" transport="http" type="response"/>
</AssignMessage>

From Postman, I sent a .png file and I was expecting to get the file back as a response from the proxy but what I’m getting is something like this: [B@1cf582a1.

From your response here, I was already expecting to see the [B@1cf582a1 value in trace. But shouldn’t I get the image back as a response?

If I set the content-type of the response to image/png, I’m also not getting back the image. What am I missing?

7

I don’t think you’re missing anything. The problem is the AssignMessage policy treats every variable as a string. Your mpf_item_content_1 contains a byte array with the actual image contents, but when AssignMessage coerces it to a string, it gets the result of Java’s .toString() method on the byte array, which renders a string starting with [B@ followed by a unique identifier. That is a representation of the byte array that is internal to the Java VM. Obviously not what you want.

This is why above I said “that might do it.” I wasn’t sure if AssignMessage would be smart enough to look at the type of the object stored in the referred variable, and then do the right thing. Apparently it is not smart enough to do that. I think it’s a reasonable change request to ask that AssignMessage do the right thing, but … even if that request is accepted, it will take a while for it to become available.

The workaround I can think of is to use an additional, as-yet-unwritten Java callout, to set the response.content to the right thing. It’s a really simple thing, a few lines of Java, but it does not yet exist. I could work on producing that. It seems like a reasonable complement to the multi-part form parser, which always returns byte arrays.

EDIT - Thursday, 1 April 2021, 09:05

I just updated the multipart form callout to add a ContentSetter callout that does this. Check the repo (git pull), and you should see the new callout.

Usage would be like this:

<JavaCallout name='Java-SetContent>
  <Properties>
    <Property name="destination">message</Property>
    <Property name="contentVar">mpf_item_content_1</Property>
    <Property name="contentType">{mpf_item_content-type_1}</Property>
  </Properties>
  <ClassName>com.google.apigee.callouts.ContentSetter</ClassName>
  <ResourceURL>java://apigee-multipart-form-20210401.jar</ResourceURL>
</JavaCallout>

Use this in lieu of the AssignMessage I suggested above.

But I’m still not clear - why set response.content? I thought you wanted to send this thing to S3? Maybe you’re using response.content just as a step while debugging the thing, eh?

8

Yes, in fact, I won’t need to set the response.content. I was just trying to figure out how I could use the variables provided by the multi-part form parser…

I’ve seen your update. Thanks a lot