Robotic Operating Model (ROM)

Expand all | Collapse all

Blue Prism Optimization

  • 1.  Blue Prism Optimization

    Posted 07-28-2020 09:44
    Edited by Norah Alsudani 07-29-2020 12:35
    Large Blue Prism environments with more than 300 Digital Workers are becoming more and more common.    As these environments grow optimizing becomes critical.  Optimizing solution designs, data retention, archiving, database maintenance...  All of these things need to be running well. 

    Until recently the biggest issues we would see were mainly due to the technical architecture.  Now as environments with more than 100 Digital Workers become far more common we are seeing very solid technical environments where the performance problems lie with the solution designs.  One of the biggest issues is using Item Tags for data instead of classification. Combine that with another issue, solutions that cause items to be retained for extended periods in Work Queues, and you have a recipe for slowness because of the queries needed to run Get Next Item. 

    Why should I care when I'm only running 10 Digital Workers?   Following design best practices may seem like a bit of overkill at the beginning but as things start to scale and grow due to your automation genius you will be glad you did. Create a Design Authority to insure design best practices from Day 1.  Blue Prism environments can grow and become very critical in the blink of an eye.  Going back and updating automations months or years from now will not be trivial, particularly when there is a large pipeline for new automations. Organizations will not want to allocate development resources to rebuild existing automations.   
      
    Forgive me if I am preaching to the choir but I see this everyday.  I don't want to see relatively small RPA programs become overwhelmed.

    ------------------------------
    Steve Waters
    Platform Consultant
    Blue Prism Professional Services
    America/Chicago
    ------------------------------


    #BPtechtips click the hashtag to explore more helpful content​ like this​


  • 2.  RE: Blue Prism Optimization

    Posted 07-29-2020 11:11
    Edited by Nicholas Zejdlik 07-29-2020 11:22

    I think this all very practical advice. Speaking from a small operation, I can attest to the value of having proper designs in place as early as possible. When we first started running Blue Prism three years ago, the first couple of processes did not follow a proper design model and we have certainly paid for that in terms of maintenance. Thankfully we adjusted very quickly, but had we not, maintenance would have been a nightmare. No matter the size and scope of the operation, proper designs are worth the time.

    I am curious though about work queue items being retained for extended periods. We have not purged any old work queue items from our database, and we currently have about 3 million items across 52 work queues. I have not encountered any performance issues as a result. In fact, I have found the BPAWorkQueueItem table to be very well designed and indexed. My question is, what kind of database performance issues have happened as a result of retaining work queue items? We are in a position to delete old items as we have customer reports that archive the statistics that we need, but I haven’t seen a reason to purge anything yet.

    I’m also wondering what kind of database issues you see when supporting Blue Prism clients? I have a hard time fathoming what could possibly go wrong, since the Blue Prism database seems to be pretty well designed. We did have issues once with a sub-optimal query plan generated by the SQL optimizer which caused usp_getnextcase to run slow (by about four orders of magnitude), but it was resolved with an index rebuild, and never happened again.

    ------------------------------

    Nicholas Zejdlik
    RPA Developer
    ------------------------------



  • 3.  RE: Blue Prism Optimization

    Posted 07-29-2020 14:40
    Edited by Steve Waters 07-29-2020 15:42
    Nicholas;

    Thanks for the response.  It is important to clear out completed items from the Work Queue and archive them as well.  In small environments this is not a big issue it's when you're running over 100 RR's and a hundred or so Processes that it can become an issue.  3 Million is starting to get pretty high.  The challenge is the Get Next queries start to take longer and longer.  

    In terms of the databases the things that go wrong are not doing the maintenance, using BP for reporting, having large numbers of people connected to the console will cause problems.  The other one is using tags for data.  Add that to the Get Next Item query and even the best systems will slow down. 

    The other is running the database on regular VM infrastructure.  The customers that have the right storage infrastructure and NUMA tend to do very well.  The key is to be intentional about the design.  It's not rocket science.  What I am starting to see now is environments that are actually very good with the automation designs being the source of performance issues.

    ------------------------------
    Steve Waters
    Platform Consultant
    Blue Prism Professional Services
    America/Chicago
    ------------------------------



  • 4.  RE: Blue Prism Optimization

    Posted 07-29-2020 16:05
    On the Get Next Item slowness, it seems to only be a problem depending on the number of pending items. The index makes it really quick to pull pending items, but if you have thousands of them, it can start to take a while for the DB to look through them. I think one of the reasons I haven't had issues in my environment is because we generally have a mere handful of work queue items pending at any given time.

    I appreciate the insight; I would love to see more posts like this in the future.

    ------------------------------
    Nicholas Zejdlik
    RPA Developer
    ------------------------------



  • 5.  RE: Blue Prism Optimization

    Posted 07-29-2020 15:23
    These are great points Steve. 

    Couple of challenges I face (we have an 120+RRs growing at a rapid rate)
    1. Business users doing the development. When the developer is from a non-IT background following the standards becomes a challenge, we do ensure they are certified still when it comes to subject of tags it persists. The learning curve impacts the whole environment in longer term. Especially when we have a federated model with centralized infrastructure, each federated unit is going to have its own learning curve and period to achieve maturity even with a proactive design authority.
    2. Reviewing and ensuring the best practices are followed. When we have a highly complex process which has calls to multiple sub processes and VBOs (at times ending up as a spaghetti code) it is humanly impossible to review the processes to check if best practices are followed.

    ------------------------------
    Vinodh Kannan Krishnan
    AVP
    State Street Corporation
    America/New_York
    ------------------------------



  • 6.  RE: Blue Prism Optimization

    Posted 07-29-2020 16:25
    Vinodh

    Thanks for the response.  I had hoped to open a  dialog.  Last year the issue was mainly technical architecture now that doesn't seem to be the problem as much anymore.  Now it seems to be more about how the solutions are designed.  To your point you can just about get away with anything under 100 Digital Workers.  Above that optimizing becomes more and more critical. 




    ------------------------------
    Steve Waters
    Platform Consultant
    Blue Prism Professional Services
    America/Chicago
    ------------------------------



  • 7.  RE: Blue Prism Optimization

    Posted 07-30-2020 12:03
    @Steve Waters - Thanks for the discussion, please suggest if get next item is slow, clearing old completed items is the only solution. Does this depend on only pending items in queue being too high or all items in queue in any status.
    What about if we have nested collection in WQ due to data related that way ex. one key ----> multiple rows (row1, row2 etc.) where each row has different fields. Each key is one line item in WQ. So we create collection inside collection to store row1, row2 within a single line item in WQ. Is this related to performance anyways or doesn't matter?

    ------------------------------
    Mayank Goyal
    ------------------------------



  • 8.  RE: Blue Prism Optimization

    Posted 07-31-2020 09:32
    As with anything technical there are often many working solutions.  Indexing helps.  It's like any query, the more data it needs to go through the longer it takes.  Clearing out items is a good start. 

    I'm just an architect  :)   I'd have to ask one of our DB guys what they think.  Ultimately the point of this is for a Blue Prism RPA environment to scale to hundreds of Digital Workers it takes a bit of optimization in many areas.  

    There are a lot of really smart people using this technology so I am confident there are solutions out there we don't even know about.   I will cheerfully learn and steal ideas when I can.    I'd like to follow up on your idea and share them with our DBAs.

    ------------------------------
    Steve Waters
    Platform Consultant
    Blue Prism Professional Services
    America/Chicago
    ------------------------------



  • 9.  RE: Blue Prism Optimization

    Posted 07-31-2020 14:25
    @Steve Waters - Thanks for your response, let me know once you are able to check with DB team for nested collection in WQ. There are many situations where we have to do that and if there is a better way to address it in case nested collection is not a good practice in WQ.

    ------------------------------
    Mayank Goyal
    ------------------------------



  • 10.  RE: Blue Prism Optimization

    Posted 07-31-2020 16:39
    Not a DBA myself but I have worked with SQL for a while. Storing large chunks of data in work queue items should not cause any performance issues over time (aside from increased disk usage). When Blue Prism queries for the next work queue item, it does not search the data itself, so there would be no effect on that side. At some point it does need to read that data so your processes can use it which will incur some cost, though it should not compound over time. If you're not currently noticing a performance issue with the amount of data you're storing, you shouldn't experience one in the future.


    ------------------------------
    Nicholas Zejdlik
    RPA Developer
    ------------------------------



  • 11.  RE: Blue Prism Optimization

    Posted 08-04-2020 10:40
    As luck would have it my colleague is on holiday until the 10th.  I will get with him and share his thoughts if he doesn't respond directly. 

    I didn't forget.  :)

    ------------------------------
    Steve Waters
    Platform Consultant
    Blue Prism Professional Services
    America/Chicago
    ------------------------------



  • 12.  RE: Blue Prism Optimization

    Posted 08-14-2020 05:22
    If you have not done so already,  it is worth looking at the release notes for the latest version of Blue Prism and scan through all the notes for versions as far back as version 6.4.  There has been a lot of work in these recent versions (and still ongoing) to improve the performance of our product,  and that includes the addition of new database indexes.   If you are still on v5 or an earlier version of v6 it may be that a product upgrade could improve some of the work queue performance issues you are having (as well as making sure you keep WQ tables lean).


    ------------------------------
    Denis Dennehy
    Head of Customer Success, EMEA
    Blue Prism Ltd
    Europe/London
    ------------------------------