Sending File Content in API Call

[ManishRaw]

Hi Everyone,

We have a requirement to send files over to a system using API calls.

The typical postman call looks like this:

I am using HTTP -VBO to replicate the same and the process works fine till we have files < ~4MB of size but when we try to encode a bigger file to base64, it gives us system out of memory exception and we can't move ahead.

As per our requirements, the endpoint can handle up-to 150MB of file size, but I am unable to encode the same and use it via HTTP-VBO, is there any other way in BP that we can achieve this?

------------------------------
Manish Rawat
Project Manager
Mercer
New Delhi
------------------------------

[ewilson]

Hi @Manish Rawat

Which action are you using on the VBO? Is it HTTP Request or HTTP Request Multipart, and are you base64 encoding the file contents yourself or just passing the raw file into the VBO?

Cheers,

------------------------------
Eric Wilson
Director, Integrations and Enablement
Blue Prism Digital Exchange
------------------------------

[ManishRaw]

Hi @ewilson 

I am using HTTP Request with method type "POST" and base64 encoding the file contents using base64Encoder Utility and then updating it in the "base64content" in the body in a variable. 🙂

------------------------------
Manish Rawat
Project Manager
Mercer
New Delhi
------------------------------

[ewilson]

@Manish Rawat

Ok, and the error occurs on the HTTP VBO when you try to send the file as opposed to the Base64Encoder VBO when you try to encode the file?

Cheers,

------------------------------
Eric Wilson
Director, Integrations and Enablement
Blue Prism Digital Exchange
------------------------------

[ManishRaw]

@ewilson 

It actually occurs at the very next calculation stage, where I am trying to update the Body variable of the call to include the base64Contents just encoded for the file:

------------------------------
Manish Rawat
Project Manager
Mercer
New Delhi
------------------------------

[ewilson]

@Manish Rawat

Ok, so this is an issue with a Calculation stage that's trying to build the body contents for the HTTP request. System.OutOfMemoryException is basically a problem of addressable memory on 32-bit operating systems. In this case, Windows allocates up to 2GB of addressable memory per process, but that's used by everything going on in your process. So you have quite a bit less per individual data item. Further, Base64 encoding data effectively increases the size of the data your encoding, so the result of the encoding ends up being about roughly 133% of the size of the original source data. 

If you need to upload large files in this fashion then there will probably need to be a custom action written to handle this use case. Typically, when you're uploading files to a service over the web you would do this in a multipart/form fashion. In that case, you could have a section of JSON that describes what's being sent, but the file(s) would then be added to the request via a stream writer so you're not loading the entire contents into memory at one time.

Do you know if this API you're working with supports multipart/form requests or is including the file as base64 encoded data in a JSON body the only option?

Cheers,

------------------------------
Eric Wilson
Director, Integrations and Enablement
Blue Prism Digital Exchange
------------------------------

[ManishRaw]

@ewilson 

Thanks for the update, I thought so that loading all the contents at a time would be having those issues.

At the moment the API only include the base64 encoded data for the call.

I will get in touch with the teams to include multipart/form request as that would be a suitable solution.

Just a question:

I haven't used HTTP Request Multipart action yet but as shown in the request below:

Is there a way in which by using HTTP Request Multipart action, we can define key value pair for whole request (The ones highlighted in green block) and others present at file level (The ones highlighted in yellow block)

To replicate the above?

I can see there are FileMetadata but that can only contain the four default fields below:

and FieldMetadata can work for green block I suppose

------------------------------
Manish Rawat
Project Manager
Mercer
New Delhi
------------------------------

[shashank.kumar280]

Hi Manish,

The HTTP Request Multipart action wouldn't be used here as the API call isn't multipart. 

If the calc stage is giving issue to update a large content in Data Item, I would try composing the Body Json within the HTTP Request code block of the method. So essentially you could try modifying the code stage to add a step to encode and build the json before the actual HTTP Request in the code stage.

------------------------------
Shashank Kumar
DX Integrations Partner Consultant
Blue Prism
Singapore
+6581326707
------------------------------