topic RE: Performance issues after Blueprism 6.6 upgrade in Product Forum

Performance issues after Blueprism 6.6 upgrade

UdayB — Mon, 03 Feb 2020 16:16:00 GMT

We have recently upgraded our Test Environment(PAT) to Blueprism 6.6.
After the upgrade, during the initial validation, we have noticed that the processes take an unusually long time to load inside the process studio.
A single process takes around 4.5 minutes to load, which will definitely have an impact on our developer productivity.

The same process when opened in another digital worker, on BP v 6.5 takes around 90 seconds, which is still too long, but something that was known.
Further investigation shows the processor utilization peaking into 90% and the service becoming unresponsive for a period of time before coming back to normal.

I wanted to check-in and see if any of the teams here has upgraded their environments to Blueprism v6.6 and faced any issues. Any pointers to the possible root cause for this performance degradation will also be very helpful.

Thank you!

------------------------------
Uday B
Solution Designer

------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

AmiBarrett — Mon, 03 Feb 2020 18:17:00 GMT

Just curious, what does your setup look like, going between the bots, the controller and the DB?

Previously when I've seen long load times like these, each layer was segregated into its own VLAN. The routing between each layer wound up costing upwards of multiple minutes in load time. (IE: Importing a VBO would take around three minutes instead of a few seconds, and opening a process would take around a minute).

Though it's listed as not being a recommended approach in the published documentation, I've also seen where connecting the resources directly to the DB instead of the application server has sped up environments.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

vivek.kumar.goel — Tue, 04 Feb 2020 03:00:00 GMT

Few recommendations:

1- Did you restart every server (Including App and DB) after the upgrade?
2- Since you mentioned it to be non-prod environment, May I know if regular purging/truncating /cleaning of logs are done on your database?
3- If you upgraded from v5.x to v6.6 directly, you may be experiencing this issue because all XML's are being stored differently in v6. Hence Once you open the object/process , it may speed up .

Hope it helps.

------------------------------
Vivek Goel
RPA Architect
Asia/Singapore
"If you like this post, please press the "Recommend" Button.
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

PhilipTrovato — Tue, 04 Feb 2020 11:05:00 GMT

Can you confirm what process is eating the CPU? automate? We have seen the DataGateway process (Java) chew CPU if you installed that with 6.6.

------------------------------
Philip Trovato
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Tue, 04 Feb 2020 13:36:00 GMT

Thank you for the input Vivek!
To answer your questions .
1- Did you restart every server (Including App and DB) after the upgrade? - Yes. Every component was restarted after the upgrade.
2- Since you mentioned it to be non-prod environment, May I know if regular purging/truncating /cleaning of logs are done on your database?
-Yes, there are regular archiving and cleanup jobs in place. We have tested by removing all logs older than 6 months reducing the overall database size significantly.
3- If you upgraded from v5.x to v6.6 directly, you may be experiencing this issue because all XML's are being stored differently in v6. Hence Once you open the object/process , it may speed up .
- Our upgrade was from 6.5 to 6.6. The process that takes 90s to load in 6.5 is taking close to 4.5 minutes, which is a major cause for concern in our case.

If it helps, we have a total of around 3000 processes/objects combined. When a brand new database is created and just a couple of processes are imported, we observed that the performance is much much better. But in a production scenario, there will definitely be a large number of processes running, and this level of slowness after an upgrade is puzzling.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Tue, 04 Feb 2020 13:39:00 GMT

Hello Philip.,
It was Automate.exe that was eating up the CPU. We do not have data gateway process configured, but were planning to do that for offloading session logs once the upgrade was completed. But based on your input, it looks like that will be the next challenge to tackle.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Tue, 04 Feb 2020 13:43:00 GMT

Hello Ami,
Our setup, has all the components (DWs, interactive clients and Appservers) in the same VLAN. The issue is that the same setup with BP v6.5 was at least usable (90s) load times vs upwards of 4 minutes after BP v6.6 upgrade.
Connecting the DWs to the database directly would not be feasible as we have more than 350 DWs in our primary site alone and almost as many in our DR site.
We are working with the product vendor to investigate further, but I was hoping to get some insights from the community for possible root causes that would help in our analysis.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

david.l.morris — Tue, 04 Feb 2020 13:44:00 GMT

I haven't heard of the XML change causing such an issue, but just to mention, the XML change did occur with 6.6. I cannot imagine that causing such incredible load times though. If you open a process, save it, and then open it again, does it still take 4.5 minutes to open the second time?

------------------------------
Dave Morris
3Ci @ Southern Company
Atlanta, GA
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Tue, 04 Feb 2020 15:04:00 GMT

HI Dave,
Unfortunately yes. We have tried opening the same process multiple times and it did not improve the load times at all. The issue seems to be with the internal queries that are executed while loading the process. But we are trying to understand what aspect of this loading process has changed between v6.5 and v6.6 that results in such drastic degradation

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

AmiBarrett — Tue, 04 Feb 2020 16:08:00 GMT

Per Dave's suggestion, did you also save it before reopening? The load time might be attributed to converting the XML to the new format. If you don't save it, the conversion will theoretically have to be done again.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Tue, 04 Feb 2020 16:34:00 GMT

We have tested the process load time after saving the process after opening it in BP v6.6
That does not seem to resolve the issue. We suspect that the root cause is related to some issue in the way, processes are loaded into the process studio when its started. But the drastic degradation of performance after the upgrade looks like a bug that has been introduced in this version which manifests only in scenarios where there are a large number of processes/objects.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

david.l.morris — Wed, 05 Feb 2020 12:28:00 GMT

When you say 'large number of processes'objects', how many would that be? This is something Blue Prism often mentions, but I've never heard a certain number. 100, 1000, 10000? I would also think that you'd see long load times with fewer processes and objects as well, but just maybe not quite as long as you all are seeing.

Have you been able to test this in more than one BP environment? We've encountered issues (not loading times though) that only affect a single environment because the database schema itself was broken by a silently-erroneous BP upgrade/installation. Unfortunately the answer we were given at that point was to reinstall the database.

What connection mode are you on? WCF, .NET remoting, ...?

------------------------------
Dave Morris
3Ci @ Southern Company
Atlanta, GA
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Wed, 05 Feb 2020 14:06:00 GMT

In our case, we have about 3200 processes/objects. We have tested this in lower environments (DEV,SIT) and UAT. The same issue is observed in all the environments. The same processes when opened in a DW with BP v6.5 takes about 90 secs. This is still very slow, but something that we were aware of.
The issue exists in databases that have been upgrades, a new database with just a few processes, however does not have this issue of long process load times.
We are currently using .NET remoting. We have tried WCF connection mode in BP v 6.5 but observed very slow response time for the application itself, and a decision was made to stick with .NET remoting until more testing is done.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

AmiBarrett — Wed, 05 Feb 2020 15:37:00 GMT

If it really is related to volume, I'm starting to wonder if your BPAProcess table needs to be forcibly re-indexed.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

JuniorNyanga — Wed, 05 Feb 2020 15:40:00 GMT

Do you experience this slowness when interacting with other parts of the Product? For example, when clicking on "Control Room", how long does it take to load the Session Management screen?

------------------------------
Junior Nyanga
Software Support Engineer
Blueprism
Europe/London
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

HakanEngman — Thu, 06 Feb 2020 11:52:00 GMT

Do you utilize TEAMS functionality to logically separate your processes and objects?

Some activites done in the UI seem to be very affected by the number of in the user context. The same activty take 5sek for a developer with a limited number of proc/object count, for me (admin) the exact same activity take 35-40 seconds. If I perform this activity on a server( A lot of power) or a VDI ( limited resources ) ; it doesn't reflect on the time. Utilisation is down as expected though.

What about your Design Control, to what extent do you check? (Open,Resetting,Close)

------------------------------
Hakan Engman
Automation engineer
Teliacompany.com
Europe/Stockholm
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

NicholasZejdlik — Thu, 06 Feb 2020 14:13:00 GMT

It may be worthwhile to run SQL profiler while opening a process in Blue Prism to see if any behavior issues exist on that end. I don't think a database call would be behind the issue since automate.exe is spiking on CPU usage - typically the CPU usage would be minimal if it were waiting for a response from the database. Certainly wouldn't hurt to take a look though.

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Thu, 06 Feb 2020 14:33:00 GMT

No. We do not use TEAMS functionality. The testing was done using a developer account, and they have access to all processes/objects.
Design control validations have been disabled, but that did not improve performance.
We are currently working with the Blueprism support team to investigate further.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

UdayB — Thu, 06 Feb 2020 14:36:00 GMT

We did run the SQL profiler and are in the process of analyzing the trace. It does look like an issue with the loading of the processes when opening the process studio.
We are also looking at infrastructure requirements, space issues and complexity of the processes themselves.

So far the issue seems to be with BP v6.6 and the mechanism for loading processes into the process studio. The high volume of processes/objects(close to 3200) in our case is also amplifying the problems.

------------------------------
Uday B Solution Designer
------------------------------

RE: Performance issues after Blueprism 6.6 upgrade

vivek.kumar.goel — Wed, 12 Feb 2020 09:34:00 GMT

Hi Ami,

I may be going a bit off topic, But this thread does bring forward the need for BP to come up with database maintenance related documents.
I understand that normally, each org's IT department does the database maintenance, but when it comes to small scale organizations with limited IT resources, the IT team sometimes is reluctant to touch the BP provided tables. They do the generic maintenance and backup though.

A document from BP about the tables/procedures for maintenance/re-indexing etc would be of great help to everyone.

------------------------------
Vivek Goel
"If you like this post, please press the "Recommend" Button.
------------------------------