Bulk add to queue unless it already exists

[michaeloneil]

In the add to queue action where you place a collection into the work queue there could be a flag option to write only those that dont already exist in the queue. At the moment we need to check each row if it exists in the queue already and if so then remove it and with a few thousand rows this takes a while. If the option existing in the Add to queue action it could add and check simultaneously removing the need for a time consuming loop.

[chris.strong]

Thank you for raising @Michael ONeil

I understand what you are asking for, so I’ve set the status to Reviewed, so that you and the community can continue to add thoughts and opinions.

Follow-up questions:

1. What would you expect to be returned in the OUT parameters if some rows were successfully added and some were not? (a set of Items IDs for added and a separate set of IDs for the not added?)
2. For context, could you share an example for where you have a collection of 1,000s of work queue items where duplicates could already exist in the Queue?

Related idea:

• Work Queue Action - "Is Item in Queue" enable case-sensitive

Kind regards

Chris Strong

Senior Product Manager

SS&C | Blue Prism

[michaeloneil]

Hi @chris.strong 

Thanks for looking into this the additional info you requested is below.

1. What would you expect to be returned in the OUT parameters if some rows were successfully added and some were not? (a set of Items IDs for added and a separate set of IDs for the not added?)I think generally we would only need to know which ones are added and which arent so as you suggested 2 collections output. It might also be worth have the KEY field included in the output, taken from the original collection, for those not added as this could possibly be used to write out to a report or email advising which ones have been worked before and werent added on this run.

For context, could you share an example for where you have a collection of 1,000s of work queue items where duplicates could already exist in the Queue?

1. I recently built a bot for a client that takes records from multiple files writes, writes these into the queue, for each queue item creates a new record on a system through an api then reports back to the original files. 

In the files received the data is taken from a database by a business team and an excel file created for the bot. The database has a backlog of cases around 80K which the bot is working through in batches. Ideally we would be taking records from the database and updating the database to say which ones were worked to prevent getting these again later but as this wasnt an option it means there is nothing filtering out ones previously worked by the bot. Each file had around 3-5k records on each file with around 6 files received each run. The requirement is that if, for each file there are duplicates of a record we remove the duplicates or we risk creating duplicate records through the api, a record received on a file today might have been received in a file a few days ago and to prevent duplication we need to check if this was worked previously. 

Hopefully the brief explanation of the process makes sense lol

[chris.strong]

Thank you @Michael ONeil

That all makes sense, thank you so much for sharing.  I’ve leave the status as Reviewed, so that you and the community can continue to add thoughts and opinions.

Related idea: 

• Work Queue Action - "Is Item in Queue" enable case-sensitive

Kind regards

Chris Strong

Senior Product Manager

SS&C | Blue Prism