Out of memory exceptions and collection clean up

[NicholasZejdlik]

I have been running into an increasing number of OutOfMemory exceptions when launching processes. The issue is definitely not memory availability, nor is Blue Prism using too much memory. I came across a support article that goes into more depth on it, which says that not emptying collections after using them could contribute to the problem.

Not emptying Collections when the Process or Object no longer requires them to exist (e.g. Remove Null Rows, Remove Empty Rows or Remove All Rows)

Once the process completes, will the collections used by the process/objects be freed automatically? I always assumed this was the case, that once a process completes any internal collections would be freed automatically by Blue Prism. The specific robot running into the out of memory issues is dealing with collections - small ones - but it runs many processes with many small collections many times. If for some reason the collections are not freed automatically when a process completes, then it would likely be the source of my issue.

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

[devneetmohanty07]

Hi,

As per the concept of Business Objects, whenever you call Business Objects from a process, the memory is consumed till the parent process is active. If your process is interacting with a large set of data such as collections with huge number of rows as well, the memory consumed could exceed the memory that gets freed periodically by your Operating System. In past, we have faced such issues while interacting with excel files having 50000+ rows and in order to resolve this we created a custom business object and used a code stage to forcefully invoke the Garbage Collector via VB .NET. You can create an action and add a code stage, there just include the function: GC.Collect()

You should be able to reclaim some of the memory in order to have enough space to carry out your process. Let us know if it helps

------------------------------
Regards,
Devneet Mohanty
Intelligent Automation Consultant
Blueprism 6x Certified Professional
Website: https://devneet.github.io/
Email: devneetmohanty07@gmail.com
------------------------------

[NicholasZejdlik]

The odd thing with that is that I'm not using any large collections in the processes that this bot runs. It uses single-row collections that contain a moderate amount of data (the content and some metadata on a small email).

The reason I am wondering if there is something being persisted beyond the process level is because the problem only occurs after a couple of days of running processes (it runs upwards of 700 sessions across five or six different processes per day). That is why I suspect there may be something being persisted beyond the process level. Restarting Blue Prism resolves the out of memory exception when it occurs.

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

[zdenek.kabatek]

Hi, Nicholas,

did you try to turn on Log Memory Usage? That could give you some insight why it is happening, however, bear in mind that it increases the space consumption for the logging in your database.

Regards

Zdenek

------------------------------
Zdeněk Kabátek
Head of Professional Services
NEOOPS
http://www.neoops.com/
Europe/Prague
------------------------------

[NicholasZejdlik]

Thankfully our IT greatly over-estimated disk space usage, so I got plenty for logging :). I turned on Log Memory Usage, but I am uncertain as to where that information will show up; I'm not seeing anything different on the session logs. Where does that log to exactly?

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

[devneetmohanty07]

I am not sure exactly if the memory usage logs enable these but in past to check the usage of my Blue Prism processes I have enabled the Blue Prism memory column in the session log viewer. Also, in [BPASessionLog_NonUnicode] table (if non-unicode logging is enabled) or in [BPASessionLog_Unicode] table you can check the [automateworkingsetcolumn]. Both of them will indicate the Blue Prism memory that got consumed.




------------------------------
Regards,
Devneet Mohanty
Intelligent Automation Consultant
Blueprism 6x Certified Professional
Website: https://devneet.github.io/
Email: devneetmohanty07@gmail.com
------------------------------

[John__Carter]

Hi Nicholas - memory consumed during a session will be released by BP at the end of a session. But, because you are running many sessions, I'm wondering if the .Net garbage collector is struggling to recycle the memory?  Could you experiment with this theory by increasing the time between the end of one session and the start of the next?

------------------------------
John Carter
Professional Services
Blue Prism
------------------------------

[NicholasZejdlik]

It wasn't feasible to increase the time between sessions, instead I tried adding a call to GC.Collect() on the more frequently run processes. I figured the end result should be similar to what you suggested, since one way is allowing the garbage collection time to take place naturally, and the other is just forcing it. So far, this seems to be working. I feel yucky for doing something like this, but if it works...

I am still curious why this happens, particularly because there is always free memory available and automate.exe is usually taking a mere 300 MB when the exceptions start to occur. In my experience in C# and .NET applications, I usually hit a couple of GB before I run into out of memory exceptions (that makes my coding sound pretty bad, lol). Unfortunately, I think that's too intricate of a question to determine easily.

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

[AndreyKudinov]

@Nicholas Zejdlik

When you pass huge collection in/out of VBO action, it stays within that action consuming memory until process ends by design (or until you call that action again with another input). What I did to remedy this problem in collection manipulation for example:
1) Moved all inputs to Init page and made them public.
2) Created ​an action to clean up and run GC
Then it works like this:
1) Call Filter collection action, get data back - collection data also stays in VBO at that point
2) Call Clean up action that does clean up inside VBO - memory is released.
 
The only problem that approach is that you better always run that clean up, or you might end up with stale data sometimes as public data items wont clean on every action call anymore.

In extreme cases, you might just as well just merge everything into a single VBO, avoiding any data copy altogether, but this is a last resort and won't work that well for Excel VBO for example that has to keep it's state.

------------------------------
Andrey Kudinov
Project Manager
MobileTelesystems PJSC
Europe/Moscow
------------------------------

[NicholasZejdlik]

I've run into similar issues with big collections, it makes me wish Blue Prism had a concept of references; it is rather difficult to have pass-by-value semantics for large chunks of memory. Not to mention the VBOs end up creating multiple copies of a collection - once when passing it into the VBO, and again when passing it into a code stage within the VBO, and then if you have an output variable you store it in... and again when outputting it from the object to the process...

In my particular case, the collections are relatively small - they're all single row - but there is a text field within them that can range from a few characters to ten thousand. It at most gets passed a single time into an underlying object.

As far as the way you perform cleanup, are you calling something that runs GC.Collect(), or do you do anything further like empty the collections first?

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------