Performance issues after Blueprism 6.6 upgrade

[UdayB]

We have recently upgraded our Test Environment(PAT) to Blueprism 6.6.
After the upgrade, during the initial validation, we have noticed that the processes take an unusually long time to load inside the process studio.
A single process takes around 4.5 minutes to load, which will definitely have an impact on our developer productivity.

The same process when opened in another digital worker, on BP v 6.5 takes around 90 seconds, which is still too long, but something that was known. 
Further investigation shows the processor utilization peaking into 90% and the service becoming unresponsive for a period of time before coming back to normal. 

I wanted to check-in and see if any of the teams here has upgraded their environments to Blueprism v6.6 and faced any issues. Any pointers to the possible root cause for this performance degradation will also be very helpful. 

Thank you!


------------------------------
Uday B
Solution Designer

------------------------------

[AmiBarrett]

Just curious, what does your setup look like, going between the bots, the controller and the DB?

Previously when I've seen long load times like these, each layer was segregated into its own VLAN. The routing between each layer wound up costing upwards of multiple minutes in load time. (IE: Importing a VBO would take around three minutes instead of a few seconds, and opening a process would take around a minute).

Though it's listed as not being a recommended approach in the published documentation, I've also seen where connecting the resources directly to the DB instead of the application server has sped up environments.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

[VivekGoel]

Few recommendations:

1- Did you restart every server (Including App and DB) after the upgrade?
2- Since you mentioned it to be non-prod environment, May I know if regular purging/truncating /cleaning of logs are done on your database?
3- If you upgraded from v5.x to v6.6 directly, you may be experiencing this issue because all XML's are being stored differently in v6. Hence Once you open the object/process , it may speed up . 

Hope it helps.

------------------------------
Vivek Goel
RPA Architect
Asia/Singapore
"If you like this post, please press the "Recommend" Button.
------------------------------

[PhilipTrovato]

Can you confirm what process is eating the CPU? automate? We have seen the DataGateway process (Java) chew CPU if you installed that with 6.6.

------------------------------
Philip Trovato
------------------------------

[UdayB]

Thank you for the input Vivek!
To answer your questions .
1- Did you restart every server (Including App and DB) after the upgrade?  - Yes. Every component was restarted after the upgrade.
2- Since you mentioned it to be non-prod environment, May I know if regular purging/truncating /cleaning of logs are done on your database?
  -Yes, there are regular archiving and cleanup jobs in place. We have tested by removing all logs older than 6 months reducing the overall database size significantly.
3- If you upgraded from v5.x to v6.6 directly, you may be experiencing this issue because all XML's are being stored differently in v6. Hence Once you open the object/process , it may speed up . 
 - Our upgrade was from 6.5 to 6.6. The process that takes 90s to load in 6.5 is taking close to 4.5 minutes, which is a major cause for concern in our case.

If it helps, we have a total of around 3000 processes/objects combined. When a brand new database is created and just a couple of processes are imported, we observed that the performance is much much better. But in a production scenario, there will definitely be a large number of processes running, and this level of slowness after an upgrade is puzzling.

------------------------------
Uday B Solution Designer
------------------------------

[UdayB]

Hello Philip.,
It was Automate.exe that was eating up the CPU. We do not have data gateway process configured, but were planning to do that for offloading session logs once the upgrade was completed. But based on your input, it looks like that will be the next challenge to tackle.

------------------------------
Uday B Solution Designer
------------------------------

[UdayB]

Hello Ami,
Our setup, has all the components (DWs, interactive clients and Appservers) in the same VLAN. The issue is that the same setup with BP v6.5 was at least usable (90s) load times vs upwards of 4 minutes after BP v6.6 upgrade.
Connecting the DWs to the database directly would not be feasible as we have more than 350 DWs in our primary site alone and almost as many in our DR site.
We are working with the product vendor to investigate further, but I was hoping to get some insights from the community for possible root causes that would help in our analysis.

------------------------------
Uday B Solution Designer
------------------------------

[DaveMorris]

I haven't heard of the XML change causing such an issue, but just to mention, the XML change did occur with 6.6. I cannot imagine that causing such incredible load times though. If you open a process, save it, and then open it again, does it still take 4.5 minutes to open the second time?

------------------------------
Dave Morris
3Ci @ Southern Company
Atlanta, GA
------------------------------

[UdayB]

HI Dave,
Unfortunately yes. We have tried opening the same process multiple times and it did not improve the load times at all. The issue seems to be with the internal queries that are executed while loading the process. But we are trying to understand what aspect of this loading process has changed between v6.5 and v6.6 that results in such drastic degradation 


------------------------------
Uday B Solution Designer
------------------------------

[AmiBarrett]

Per Dave's suggestion, did you also save it before reopening? The load time might be attributed to converting the XML to the new format. If you don't save it, the conversion will theoretically have to be done again.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------