cancel
Showing results for 
Search instead for 
Did you mean: 

Merge 2 big CSV Files

LaurentAxa
Level 3

Hello,

 

My robot need to download from a website a large CSV File and add all the rows from this file to an existing CSV file. 

Any idea if I can find an object or a code able to do that ?

I tried with a C# code I found online but when I execute on Blue Prism I have compilation error.

1 BEST ANSWER

Helpful Answers

Hello LaurentAxa

One of the methods I would try is as follows:

When the CSV can fit in memory and you don’t mind a couple of seconds of processing:

-> Download

● If the URL is publicly reachable (or uses your Windows auth):
Use Utility – HTTP > HTTP Request (GET) → save response to a temp file (e.g., C:\Temp\new.csv).

○ Set Use Default Credentials = True if you need NTLM/AD.
○ Add headers/cookies if needed (Authorization, cookie, etc.).

-> Append (skip header)
● Use Utility – File Management:

○ Read Lines From File → C:\Temp\new.csv → collection NewLines
○ If you want to skip header in new.csv: loop from row 2, else from row 1.
○ In loop, call Append Line To Text File targeting C:\Data\existing.csv.

Note: this reads the full file into a collection first—fine for small/medium CSVs, not great for multi-GB.

Regards

Hossein Azimi (APJ Team)

View answer in original post

6 REPLIES 6

Hi @LaurentAxa 

 

We can help you.

Could you just tell me if you want to read the CSV file inside a collection, or merge it directly without going through a collection?

Regards

Hello LaurentAxa

One of the methods I would try is as follows:

When the CSV can fit in memory and you don’t mind a couple of seconds of processing:

-> Download

● If the URL is publicly reachable (or uses your Windows auth):
Use Utility – HTTP > HTTP Request (GET) → save response to a temp file (e.g., C:\Temp\new.csv).

○ Set Use Default Credentials = True if you need NTLM/AD.
○ Add headers/cookies if needed (Authorization, cookie, etc.).

-> Append (skip header)
● Use Utility – File Management:

○ Read Lines From File → C:\Temp\new.csv → collection NewLines
○ If you want to skip header in new.csv: loop from row 2, else from row 1.
○ In loop, call Append Line To Text File targeting C:\Data\existing.csv.

Note: this reads the full file into a collection first—fine for small/medium CSVs, not great for multi-GB.

Regards

Hossein Azimi (APJ Team)

SayeedBinAbdullah
Verified Partner

Hi @LaurentAxa 

Please also have a look at the below thread as well,

https://community.blueprism.com/t5/Product-Forum/Handling-large-CSV-file/td-p/121856 

Best Regards,
Sayeed Bin Abdullah

Senior Consultant
Wonderbotz

Hi @Mohamad_Dakkouri 

As I have to merge large file with around 1  million lines and 50 columns, going through a Blue Prism collection will not work.

I found a solution with executing a PowerShell script.

LaurentAxa_1-1756312823238.png

LaurentAxa_2-1756312840871.png

LaurentAxa_3-1756312885629.pngLaurentAxa_4-1756312930224.png

 

LaurentAxa_5-1756312964461.png

 

 

 

Thank you

You’re welcome.
I’m always happy to help.