Performance issues after Blueprism 6.6 upgrade

[UdayB]

We have recently upgraded our Test Environment(PAT) to Blueprism 6.6.
After the upgrade, during the initial validation, we have noticed that the processes take an unusually long time to load inside the process studio.
A single process takes around 4.5 minutes to load, which will definitely have an impact on our developer productivity.

The same process when opened in another digital worker, on BP v 6.5 takes around 90 seconds, which is still too long, but something that was known. 
Further investigation shows the processor utilization peaking into 90% and the service becoming unresponsive for a period of time before coming back to normal. 

I wanted to check-in and see if any of the teams here has upgraded their environments to Blueprism v6.6 and faced any issues. Any pointers to the possible root cause for this performance degradation will also be very helpful. 

Thank you!


------------------------------
Uday B
Solution Designer

------------------------------

[UdayB]

We have tested the process load time after saving the process after opening it in BP v6.6
That does not seem to resolve the issue. We suspect that the root cause is related to some issue in the way, processes are loaded into the process studio when its started. But the drastic degradation of performance after the upgrade looks like a bug that has been introduced in this version which manifests only in scenarios where there are a large number of processes/objects.

------------------------------
Uday B Solution Designer
------------------------------

[david.l.morris]

When you say 'large number of processes'objects', how many would that be? This is something Blue Prism often mentions, but I've never heard a certain number. 100, 1000, 10000? I would also think that you'd see long load times with fewer processes and objects as well, but just maybe not quite as long as you all are seeing.

Have you been able to test this in more than one BP environment? We've encountered issues (not loading times though) that only affect a single environment because the database schema itself was broken by a silently-erroneous BP upgrade/installation. Unfortunately the answer we were given at that point was to reinstall the database.

What connection mode are you on? WCF, .NET remoting, ...?

------------------------------
Dave Morris
3Ci @ Southern Company
Atlanta, GA
------------------------------

[UdayB]

In our case, we have about 3200 processes/objects. We have tested this in lower environments (DEV,SIT) and UAT. The same issue is observed in all the environments. The same processes when opened in a DW with BP v6.5 takes about 90 secs. This is still very slow, but something that we were aware of.
The issue exists in databases that have been upgrades, a new database with just a few processes, however does not have this issue of long process load times.
We are currently using .NET remoting. We have tried WCF connection mode in BP v 6.5 but observed very slow response time for the application itself, and a decision was made to stick with .NET remoting until more testing is done.


------------------------------
Uday B Solution Designer
------------------------------

[AmiBarrett]

If it really is related to volume, I'm starting to wonder if your BPAProcess table needs to be forcibly re-indexed.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

[JuniorNyanga]

Do you experience this slowness when interacting with other parts of the Product? For example, when clicking on "Control Room", how long does it take to load the Session Management screen?


------------------------------
Junior Nyanga
Software Support Engineer
Blueprism
Europe/London
------------------------------

[HakanEngman]

Do you utilize TEAMS functionality to logically separate your processes and objects? 

Some activites done in the UI seem to be very affected by the number of in the user context. The same activty take 5sek for a developer with a limited number of proc/object count, for me (admin) the exact same activity take 35-40 seconds. If I perform this activity on a server( A lot of power) or a VDI ( limited resources ) ; it doesn't reflect on the time. Utilisation is down as expected though. 

What about your Design Control, to what extent do you check? (Open,Resetting,Close)

------------------------------
Hakan Engman
Automation engineer
Teliacompany.com
Europe/Stockholm
------------------------------

[NicholasZejdlik]

It may be worthwhile to run SQL profiler while opening a process in Blue Prism to see if any behavior issues exist on that end. I don't think a database call would be behind the issue since automate.exe is spiking on CPU usage - typically the CPU usage would be minimal if it were waiting for a response from the database. Certainly wouldn't hurt to take a look though.

------------------------------
Nicholas Zejdlik
RPA Developer
------------------------------

[UdayB]

No. We do not use TEAMS functionality. The testing was done using a developer account, and they have access to all processes/objects. 
Design control validations have been disabled, but that did not improve performance.
We are currently working with the Blueprism support team to investigate further.


------------------------------
Uday B Solution Designer
------------------------------

[UdayB]

We did run the SQL profiler and are in the process of analyzing the trace. It does look like an issue with the loading of the processes when opening the process studio.
We are also looking at infrastructure requirements, space issues and complexity of the processes themselves.

So far the issue seems to be with BP v6.6 and the mechanism for loading processes into the process studio. The high volume of processes/objects(close to 3200) in our case is also amplifying the problems.

------------------------------
Uday B Solution Designer
------------------------------

[vivek.kumar.goel]

Hi Ami,

I may be going a bit off topic, But this thread does bring forward the need for BP to come up with database maintenance related documents.
I understand that normally, each org's IT department does the database maintenance, but when it comes to small scale organizations with limited IT resources, the IT team sometimes is reluctant to touch the BP provided tables. They do the generic maintenance and backup though.

A document from BP about the tables/procedures for maintenance/re-indexing etc would be of great help to everyone.


------------------------------
Vivek Goel
"If you like this post, please press the "Recommend" Button.
------------------------------