Query batches ready for export from Decipher

[zdenek.kabatek]

Hi,

as Get Next Completed Batch does not work reliably I would like to build my own query for getting batches ready for export from Decipher. Studying this https://digitalexchange.blueprism.com/dx/entry/3439/solution/decipher-get-batch-status it shed some light on how this can be achieved, however, it is not possible to differentiate among batches which have been already exported and the ones which are still waiting for export. As the batch statuses are following (copied from the process).
But status 6 is the same for batch which has been exported and the batch waiting for export. Furthermore, there is a column Flag which seems as the solution but that is the same for exported/waiting for export batches. How the BP action Get Next Completed Batch knows which one to pick?

Status field values: • 0 – Invalid
• 1 – UserCreating – The batch is being created by user
• 2 – UserSending – The batch is being sent by user
• 3 – UserReceiving – The batch is being received by user
• 4 – UserProcessing – The batch is being processed by user
• 5 – ReadyForProcessing – The batch is ready to be processed by next role
• 6 – Finished – The batch has finished the processing by all defined roles and is exported
• 7 – Closed – The batch has been manually closed. It stops appearing in the list of active batches, but all data about its documents and pages is stored, so it can be opened again in the future.
• 8 – AutoClosed – The batch has been automatically closed. The system automatically closes a batch if all its documents have been closed manually (for now only in Admin station). If one of the documents is opened by an admin, its batch will be automatically opened.
• 9 – Deleted – The batch has been explicitly deleted – it cannot be recovered anymore because the deletion removes its image data. The object metadata is still in the system and can be seen during Reporting (information about which batches are deleted, why, when etc.)
• 10 – UserUploading – The batch is currently being uploaded (directly using files) by the user application
• 11 – WaitingForImageProcessing – The batch created from file(s) upload is waiting for image processing
• 12 – InImageProcessing – The batch created from file(s) upload is in image processing. The input files of the batch are sent simultaneously to the available Image Processing clients
• 100 – Exception – The batch has been classified as an exception

[IsmoLehtiniemi]

Hi,

I believe that RabbitMQ is used for creating a queue of finished items from the server side.

RabbitMQ is also used when new batches are added to the server from the client.

As you said the queueing is not reliable, and the root cause is somewhere in the RabbitMQ queue logic. For example if an invalid item gets stuck on top of the queue, rest of items can't be retrieved.

I got frustrated with this already a year ago, so we built a custom solution which combines SQL queries straight from the Decipher DB with a Blue Prism work queue of added batches. After the item is retrieved once from Decipher, it is marked as completed in the "added batches" work queue.

[Ben.Lyons1]

Hi Zdenek,

Just to add to Ismo's great response.

When a batch has completed verification, it goes to the export client. This client sends a message to RabbitMQ which will just contain the batch ID. The action in Blue Prism will pick these up in the order they are added. As a result Decipher will see this batch as complete. The action to get the next batch uses the next batch ID found in RabbitMQ to get all the respective document ID's via the Decipher API.

You can help differentiate batch types by creating different queues in RabbitMQ. This can be done easily by changing the queue suffix in the upload action.

We are working on alternative methods for check the batch stage and retrieving the document data. What are your thoughts on how you would like to see this done?

[zdenek.kabatek]

Hi, Ben,

as I am not familiar with RabbitMQ and how to query/work with it I believe that for many other BP users (who are in most cases familiar with SQL language in my opinion) would be the best option to have another status in the Status field which would be readyforexport or exported to RabbitMQ and another one ExportedFromDecipher so that I can write my own query and run it and get all the information which I need to avoid RabbitMQ step (and learning how to work with it).

Regards,

Zdenek

[Ben.Lyons1]

Thanks Zdenek,

I think what we're working on is on the right lines with what you're after, so hopefully we'll have a suitable solution for this in a future release.

[bbobo]

Hi Ben, 

If we implement logic to skip the RabbitMQ steps for getting the next batch will that have a negative impact on RabbitMQ long-term? Is Decipher doing any cleaning/deleting of messages within RabbitMQ?

[Ben.Lyons1]

Hi Ben B,

You will end up with a large volume of messages waiting in RabbitMQ. When Blue Prism gets the next available batch, it takes that message out of the queue.

Thanks

Ben L