Most efficient way of adding to work queue

[TomBlackburn1]

Does anyone have a suggestion for the best way to add big amounts of data into a Blue Prism work queue? The amount of data can vary day to day, but on average I estimate a daily amount of around 5000 records that need adding to a work queue and it's taken several minutes to load on. I'm also intermittently getting a Internal : Exception of type 'System.OutOfMemoryException' was thrown error. The Data collection has around 40 fields. I need a solution that takes the least amount of time (ideally). Of course if the best solution is not the quickest solution, I may have to allocate more time into loading the data. I have though about the following which have both pros and cons: - Create several work queues and load split the amount of data equally across these queues. - Cons - More work queues to maintain - Several processes will need amending to handle multiple work queues - Extra processes/objects may need to be created to allocate work - Pros - I assume the OutOfMemoryException should be resolved - Quicker? - Loop through the collection and add 1 record from the collection at a time to the work queue. - Cons - Time consuming - Quick build time - Pros - Again I assume the OutOfMemoryException will go - More flexibility on querying data prior to adding to a work queue Does anyone have a suggestion? Can I allocate more memory to Blue Prism? or can I increase the Java memory on the robots?

[John__Carter]

Hi Tom Try reading your data source in parts so you have fewer rows in your collection at any one time. I think the out of memory errors could be due to the PC struggling to 'lift' a 5000x40 collection. Doing a series of smaller reads rather than a single big one might be more efficient. Also consider adding new queue items one at a time instead of 'all in one go'. And think about the effect of having logging switched on - collections in particular can add big pieces of network traffic. John

[JustinBishop]

Hi Tom, I have a process that works 75,000 plus records each day in one queue, I initially ran into the same issues as you. The key for my process was to reduce the amount of columns in the data capture before the data makes it to the collection. The collection is where your memory errors generally occur. Collections cannot handle working with large amounts of data because they are not SQL driven like the work queue. My process also captures the 75,000 records in increments of 5000. I perform some initial data cleansing (several loop stages) as well before sending it on to the queue. Reducing the columns was the first step. We also recently discovered that the log file and work queues on the BP server needed to be cleaned and reduced and put on a maintenance schedule. We also added more RAM to our BP server to help with all of the data ""crunching"". However, adding all 5000 to the queue at once has never been a problem. That problem may be an issue with resource limitations on your workstation.

[TomBlackburn1]

Thanks all for your replies. To summarise the answers to all the above: Memory Allocation: - The SQL Server is allocated 60GB - The development machine is 8GB - The BP Server is 4GB - The BP Clients are 4GB Logging: - Admittedly I have full logging turned on. I may look to reduce the amount of logging. - I did want to keep full logging on ideally as we have essentially ""unlimited"" disk space on the SQL Server. Actions I may take: - Increase the memory allocation on the BP Server (Definitely - didn't realise this was only 4GB) and Clients - Look to reduce the amount of logging. - Select 500 records a time from the SQL table and add them to the work queue, then repeat until all records have been extracted and added to the work queue. Perhaps look to increase this number once the memory allocations have been increased and logging reduced.