Work Queues - To use or not to use - Best Practice

[RickBurrows]

My team has been having a discussion about work queues….to use or not to use?

 There are many benefits to using these queues when working through several items of data that are processed in the same manner.  However there are use cases out there that do not require the processing of repetitive input.  I'm working on a process like this at the moment.  This process simply goes to an application, clicks a few buttons to download a couple of reports, massages the data in these to reports to come up with another report which is saved to an accessible location for later use by the end users.

To me, using queue processing adds to the complexity of the process.  Using the queue for a single work item (not even sure what that item would be in my use case) would seemingly consume BP resources unnecessarily.  We are wondering if there are any technical implications around queue usage (or not) around logging, database, future BP releases or whatever that should be considered?  There doesn't seem to be much online regarding best practice in terms of using a queue or not. 

Again, using work queues is great when the use case calls for it, but what if it does not.   Your thoughts?

------------------------------
Rick
Principal Software Developer
------------------------------
​

[Anonymous]

Hi Rick,

I'm very interested in this discussion as well. 

The developers I work with don't always agree on best practices, but "always use a queue" was one of the few they all immediately endorsed. Our team uses work queues for every process for a number of reasons:
- Reporting: we have custom reporting that tracks volumes and processing time for every process as part of a value realization model and routine monitoring strategy. Having queues by default for all processes makes this more consistent in terms of source.
- Within process failures: we had some processes that looped through collections that were challenging to recover if there was an unexpected error partway. This is easier to manage with a queue-based design.

I can certainly appreciate why you might not see value in a queue for this use case. In terms of performance implications and the other best practices, I'm hoping that others will weigh in. I haven't seen any functionality that not using a queue would prevent in the current version of Blue Prism, though I have seen other third party solutions that I believe would need a queue to use.

If you haven't seen this yet, I find the Solution Design Overview helpful in thinking through this kind of challenge: https://portal.blueprism.com/system/files/documents/Blue%20Prism%20-%20Solution%20Design%20Overview.pdf 

Thanks for starting this discussion. Curious to hear other thoughts on the matter.

Heather
Solution Architect, ATB Financial

------------------------------
Heather Ruhl
Sr Manager, RPA
ATB Financial
America/Edmonton
------------------------------

[RickBurrows]

Thanks for your response and insights Heather. 

Agreed though, it would be interesting to hear from others on this topic!  I do see the value in your reporting and metrics side of things, but curious about how the use of a queue for the sake of using it would impact that, say if it was a single work item that would "initiate" a process.

Thanks again,
Rick​

------------------------------
Rick
Principal Software Developer
------------------------------

[LucaValente]

HI Rick,

the real question is: is that process a good candidate for an RPA automation? What is the value of using Blue Prims to automate a single task? What are the benefits? Blue Prism is an RPA platform. Its Digital Workers run unattended and process business cases that can generate cost savings or additional revenue. You can see those benefits only if you automate business processes that maximize return-on-effort. The Process Discovery tool (available on the Blue Prism portal under the ROM section) explains how to select processes that are good candidates for an RPA automation. All those processes require a Work Queue.
Of course nobody stops you from creating script-like (not scalable) automations with Blue Prism but I'm not sure that the best way to make use of the license.
So long story short: yes, a Blue Prism RPA process should always make use of Work Queues. 
Hope this helps.
Regards,
Luca

------------------------------
Luca Valente
Senior Product Consultant
Blue Prism
America/New_York
------------------------------

[DaveMorris]

I agree with the comments before mine that while some automations do not require a queue, it is best to go ahead and do it anyway. For automations where you're just performing a single set of linear steps once per day for like 2 minutes and stuff like that, I'd just create and start a queue item at the beginning and then mark it complete or exception at the end of the linear set of steps.I know this isn't necessary, but as others have stated Blue Prism is intended to work based off of queues in order to perform back office tasks in a repetitive nature.

The automations that seemingly don't require a queue (though perhaps they may be frequent) are non-standard as far as how the software is intended to be used. I think it is best for non-standard automations to bend to the will of the normal ones by conforming to queue usage and such, if nothing else but to ensure your view of work done can be tracked in the same way across all automations.

In my opinion, this is the same reason that most of us use that same basic format for the Main Page of processes that Blue Prism suggests. It is a design pattern and conformity to a standard makes things run more smoothly. It's less jarring to new people if they see a similar structure as they are used to.

All that said, if you have a solid way to include non-queue automations and queue automations into the same view in a dashboard or whatever by combining session runs and queue items, I don't know that it really makes a difference whether a queue is used or not.

------------------------------
Dave Morris
3Ci @ Southern Company
Atlanta, GA
------------------------------

[DeborahYarosh]

I believe that work queues can be used for most automations however there are times when a work queue may not make sense.  Here are 2 examples:
1. Automating a task where the purpose is to look for work in an application that simply needs to be cleared.  We have an application that when quotes are turned into policies, under certain situations the policy doesn't bind because a referral gets created.  The referral is no longer needed and so a rep simply goes in and submits the referral so the policy will bind.  In this situation, the process was automated to check for these referrals every hour and if present submits it.  A work queue isn't needed here because where you look for the data is the same place that you submit the referral.  You can argue that this is a process that should not have been automated but in our case the automation prevents calls into our call center because changes can not be made on the policy until the referral is submitted.
2. Another case where I would argue that work queues may not be warranted is when the process to be automated gets its work from a system that controls the work.  For example, if users get their work from a work queue within an application, the application controls access so multiple users can work the tasks without collision.  The bot can do the work once it receives it from the application's work queue and if exceptions occur, the task can be placed back in the application's work queue.  If the required reporting cannot be generated from the application's work queue, the completed tasks and exceptions can be added to a BP queue that is used strictly for reporting.

I am interested in the thoughts others may have on these 2 scenarios and whether BP work queues should be used.​

------------------------------
Deborah Yarosh
Application Developer II
Farmers Insurance
America/New_York
------------------------------

[Anonymous]

Hi Deborah, 

Thanks for sharing your thoughts!

In the first scenario, I'd agree with your design recommendation and see how a queue wouldn't add any value.

In the second, I've seen this design struggle a bit in one scenario where the bot continually tried to pick up a cases that had been deemed a business exception until a human user finally picked it up, or where both a human and bot tried to pick up the same case at the same time. Changing the status of an item solved both of these issues, but in our system that change made it important for us to queue the items in the unlikely scenario that the change to the status isn't correctly completed.

Happy to hear how others feel about queue design. It's excellent food for thought.

------------------------------
Heather Ruhl
Solution Architect
ATB Financial
America/Edmonton
------------------------------

[GopalBhaire]

I think using Queue is always ideal, I as a developer use Queues for almost all the processes, had a issue with Queue related to use of tags a while ago (fixed) apart from that had no issues even adding large volume of data shouldn't take a more that few seconds. On logging part yes the database fills up quickly if you're saving lots of data in queue which is required by some business and yes that can impact bot performance but there Blue Prism Database Maintenance document that will help you with this, also small amount of data I don't think will have a large impact on performance. In short Queue is the way to go, It helps in designing scalable, multi-bot solutions and almost always help keep track of data.

@Rick Burrows

In your scenario you can create a folder with random name (or with timestamp) add it to Queue as key, then save the reports you download to ​this folder update the item that files have been downloaded, if in case the process fails while generating the second report then you don't have to download the files again, also it will help give you proof of work that the bot performed the right download and generated the right report (some businesses want this). You can separate out downloading and report generation if you'll be downloading different files again and again, using one bot to download and multiple bots to create reports.

@Deborah Yarosh

1. Ok - in this scenario not using a Queue can work (although you can still use it for reporting what referral id was submitted), but then why use Blue Prism in first place, there are free to use tools (MIT licensed) which obviously are not as feature rich as Blue Prism but will help you in this scenario without any hiccups and also helps save you in licensing.

2. I agree with @Heather Ruhl you can have a scenario where bot keeps picking up the same request (a Business exception) again and again if​ there are no more item to process 

​

------------------------------
Gopal Bhaire
Analyst
Accenture
------------------------------

[RobinToll]

I really love the insights in this thread and the discussion around work queues and their uses. I want to propose a slightly different way of looking at the question, though - not "should work queues always be used" but "what is the purpose of a work queue inside of a BP Automation"? And while it's a discussion that you could drag on extensively, the short version of the answer to me is "organizing/prioritizing work". And from that perspective, the question is less "does a work queue always need to be used" and more "has the purpose of a work queue been fulfilled, either using a work queue or not"?

To @Deborah Yarosh's first scenario, which doesn't seem to naturally include a use for a work queue, what's actually triggering this work? The assumption proposed in the scenario was that it was scheduled - which brings up the point that functionally, there's sometimes very little operational difference between a schedule-driven process of a certain type and a *data*-driven process of a similar type. Don't forget here that you can create a work queue item in Blue Prism with a future date. You could re-implement the process - instead of as a stand-alone process scheduled every half hour to run this single thread - a meta-process monitoring a bank of 3, 5, 10, however many work queues... And one of those work queues holds the object "referral submission trigger". Whenever it finishes whatever it's working on, if it comes back and finds that work item has matured (hit its execution time), it then runs the process and - as the last step - inserts a new one 30 minutes into the future. Not only does this kind of model allow you to share the work over a larger pool of workers (giving you some more resilience against machine failure) but lets you have more efficient use of the resource that is tasked to this job; instead of being required to have a clear window in its schedule every 30 minutes for this, it implicitly takes care of it at approximately the right time with no gap in its work.

My point is... Work Queues are designed to organize the work that needs to be done. In Deborah's first example, it's not that that prioritization doesn't need to happen; it's that it's being done implicitly by the scheduler acting in the place a work queue otherwise would.

As another thought, don't forget that work queues and work items can act in a second manner - they can provide natural process segmentation allowing for more flexible/dynamic process creation, and assisting with things like disaster recovery.

What I mean by process segmentation is this: let's say you have a business process you're automating that involves reading in an e-mail from a mailbox, moving it to a new mail folder, parsing some data out of it, creating a new account in an internal web site, and submitting a ticket on a 3rd-party web site to link the two accounts together. Because of the nature of the process, the only natural place to utilize a work queue from a simpler point of view is "pooling" those e-mails; but there's more you can look at here. What if, for example, you actually had 10 different business processes that all had to create accounts in this internal web site? That segment of the business process we're discussing could be extracted with a work queue. Similarly, the portion of work for creating tickets in the 3rd-party system can be extracted using a second work queue and separate worker pool. Then, those little process fragments can be used by multiple business processes by the simple expedient of sharing an inbound work queue; and if those processes need to be sequential instead of parallel, that can still be accomplished with just a little bit of metadata indicating the parent process it was submitted from to allow the follow-on work queue items to be created in line.

Obviously, for smaller and simpler situations, this type of complexity is totally unnecessary; but for more complex situations where you have numerous business processes overlapping activities, it can help create great efficiencies and save duplicate effort.

As  to the thought of Work Queues assisting with data recovery... The primary nature of a work queue item is to save the work's state. They can be updated - using tags or status or even meta-data stored inside the work queue - at each stage of a process, or if the process is segmented then passed from work queue to work queue. Then if there's a system failure, the work queue item itself is telling the story of what's already been done and what needs to be done next; in many cases, no human intervention is necessary, and a well-constructed process can see from the queue item exactly what needs to be done next to pick up where it was left off.

So I apologize for the big wall of text... But Work Queues are actually a fascinating and deep topic, which I really enjoy exploring. Hope this is interesting to you folks.

​

------------------------------
Robin Toll
Senior Technical Consultant
Blue Prism
Professional Services Americas
Infrastructure & Architecture Specialist
------------------------------