Need Advice: Handling Large Base64 Files in Blue Prism Queues

[Christer_Tellefsen]

Hi everyone,

I'm contemplating an approach and could use some advice.

My plan is to download an important document, encode it in base64, add it to my collection, and then "Add to Queue." This way, any of my robots from different servers can download this file and use it as needed.

However, I'm concerned about potential issues. Base64 encoding results in large data files. If we start adding every email, file, etc., to the Queue, I'm worried it could impact performance. I recall reading that the recommended Blue Prism database size should not exceed 10 GB.

Has anyone tried something similar? If I put a base64 encoded file in the Queue, does it get stored in the Blue Prism database?

Any insights or experiences would be greatly appreciated.

Kind regards,
Christer Tellefsen
RPA Developer

[devneetmohanty07 · Accepted solution]

Hi @Christer_Tellefsen,

My advice here would be to go with a hybrid approach of using both File System and Database. Ideally, once you download the document, you can first try to compress it using one of the compression algorithms (GZip, Brotli, Zstandard), convert the same to Base64 format and then basically save the file in some File System such as Shared Network Drive or any Cloud Storage Solutions like AWS S3, Microsoft SharePoint, Microsoft OneDrive etc. Whatever solution you use, use any unique identifier for that file such as FileId, File Name or Full File Path to use it as a reference in your queue item. This way, you can optimally use queues to their best capacity without impacting the performance of the database eventually.

However, if the above approach is not feasible, you can first download the file, compress the file using one of the compressional algorithms. Once you have compressed the file, then you can encode the same to Base64 format and store it in the queue item itself. However, I would not recommend the same as it will eventually kill your database at some point even if compression will save you some bandwidth.

Do note, encoding a file to Base64 format does impact the file size as Base64 encoding increases the size of the data by approximately 33%. Hence, I recommended to use a compression logic so that your final file size is less than or near to the original file size itself.

For performing the compression, you will need to create a custom business object using one of the VB .NET compression classes (GZipStream, BrotliStream, Zstandard) or you can also try to zip a file using the Utility - Windows Compressed File business object which will provide the file in ZIP format. If you use the Business Object approach, then while you try to read this file, you need to decode Base64 format, de-compress\extract the Zip file contents and then read the actual content.