topic RE: Decipher one field's header "takes" value of another field in Product Forum

Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Tue, 02 Jan 2024 22:38:00 GMT

Hi,

I am struggling with one more thing with DFD in parallel and due to urgency of the problem I am not able to create new ticket and wait for response.

I have to extract 2 things - Invoice number and title of the Invoice (word "Invoice")

I created 2 fields:

InvNo with sample headers like 'Invoice, Invoice number' etc.

and

InvTitle as a selectable field with option 'Invoice' and 'Other' as default.

My problem is that the InvNo field "takes" the "Invoice" title as its header to find the invoice number and then the "Invoice" value is not assigned to the "InvTitle" field.

How to make it works?

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Ben.Lyons1 — Wed, 03 Jan 2024 07:56:00 GMT

Hi Wojciech,

Is this based on the same DFD and sample document shared with customer support?

Thanks

------------------------------
Ben Lyons
Senior Product Specialist - Decipher
SS&C Blue Prism
UK based
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Wed, 03 Jan 2024 10:54:00 GMT

Hi, I created new ticket for that 276451.

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Wed, 03 Jan 2024 10:59:00 GMT

sample document is any invoice with only one occurence of word 'invoice', e.g like 'Invoice no 123/123'.

I have to extract value 'Invoice' to one fields, and '123/123' to another.

Problem is that sample header of invoice number is list of 'Invoice','Invoice no' and similar words and this header "takes" 'Invoice' word for itself and it cannot be assign to invoice title field.

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Ben.Lyons1 — Wed, 03 Jan 2024 13:27:00 GMT

Hi Wojciech,

I would avoid using overly common keywords, especially where they may appear in multiple places on a document and reusing them in multiple fields is highly likely to cause an issue. I would remove Invoice from the Invoice Number field, and all simple variations.

You don't need to specify all keywords for a field, Decipher will learn the position on the document. This would just mean Decipher won't automatically recognise the field on an untrained document.

Alternatively you could use Specific Versions if this suggestion is problematic, though it's not recommended to use this for a high number of variations and may still only work as needed after some training.

Thanks

------------------------------
Ben Lyons
Senior Product Specialist - Decipher
SS&C Blue Prism
UK based
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Wed, 03 Jan 2024 16:18:00 GMT

Hi,

In our case there is more than 2000 invoice templates with different placing of invoice number. Using sample header with keywords like 'Invoice' ensures satisfactory effectiveness. Without those keywords effectiveness drops dramatically.

So do I understand it correctly that sample header of invoice number field "steals" the keyword 'Invoice' from Invoice title field?

Is it possible to set same exact value for sample header for one field and for extracted value for other?

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Ben.Lyons1 — Wed, 03 Jan 2024 16:27:00 GMT

Hi Wojciech,

Removing the keywords shouldn't prevent Decipher from learning the layout of each invoice template. Without keywords Decipher will learn the layout through training in the data verification stage.

Having keywords is great because it gives Decipher something to start with when it hasn't seen a particular layout before, but without them it will use it's rules-based ML training. With each trained document, Decipher will learn and improve. I've done many use cases without any keywords, though you may need to reset your training data. Check out our best practice guide.

If you're unsure of how best to approach this I recommend seeing if you can get help with a knowledge support session. A member of our professional services team will be able to help get you up and running.

Thanks

------------------------------
Ben Lyons
Senior Product Specialist - Decipher
SS&C Blue Prism
UK based
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Wed, 03 Jan 2024 16:41:00 GMT

Ben, I am familiar with best practice guide, but our case isn't that simple.

First of all we have like 6000 invoices to process with at least 2000-3000 templates, every month there is a big bunch of new invoices with new templates. So even with machine learning "on" providing good results without keywords in sample headers will be not possible.

Second thing is that there is no option on production to correct Decipher choices after processing and we cannot set manual data capture verification for production process. Or maybe I am not familiar with "correction after processing" option? So how decipher would learn based on wrong choices? I was thinking about collecting incorrectly processed invoices, then uploading it to test environment to manually confirm data capturing with ML turned on and then to upload training data to prod env, but it is again- not effective and not prone to production processing error.

So... the only way to get correct invoice number in my point of view is to use those sample headers but than I cannot use this value, which was assigned to sample header in other field value. Is it normal decipher behavior? Those described "stealing"?

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Ben.Lyons1 — Thu, 04 Jan 2024 08:16:00 GMT

Hi Wojciech,

I appreciate the complexity of your use case (as best I can without the full detail). Where we've had customers with similar volumes of new invoice layouts, it was generally found that they worked better with minimal generic keywords. I would also start with the additional ML in the Document Type disabled, this would only improve an already well functioning set of training data.

If a new layout arrives in Decipher, it will only be able to auto-process if it's highly confident, else it will be held for manual verification. The training data (rules-based model) could only do this with keywords and previous training of this layout. If the additional ML is enabled it will use the training of unrelated invoices and could give you a false positive, incorrectly marking it as high confidence.

It's not a case of "stealing" the field, if you tell a person to look for the word "Invoice" for two fields, how will they know which is the correct field without previous training of the layout? Again it's recommended not to use words which are likely to be repeated in a document or could have different use across your different vendors/layouts.

All IDP tools need training and many do not use pre-defined keywords, so it is possible to process this volume without them, you will just need to verify 2/3 documents per layout (some of this can be done in production, once the DFD is complete). Additionally from Decipher 2.2 (current release), the training data will build its own keywords for each field per layout (this is stored in the training data and is not user accessible). These keywords can be completely different from layout to layout and will help Decipher if the field moves within the layout.

I really do recommend looking at reaching out to our professional services for support with such a complex use case, you may be entitled to some time with them included in your support agreement.

Thanks

------------------------------
Ben Lyons
Senior Product Specialist - Decipher
SS&C Blue Prism
UK based
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Thu, 04 Jan 2024 13:46:00 GMT

Regarding to "It's not a case of "stealing" the field, if you tell a person to look for the word "Invoice" for two fields, how will they know which is the correct field without previous training of the layout? Again it's recommended not to use words which are likely to be repeated in a document or could have different use across your different vendors/layouts." - I totally disagree. Case is not to assign same value to two different fields or even decide to which it should be assign (btw misc. params should work - another ticket for that already created), but case is that word "Invoice" is Sample header for one field and value for other and Decipher not letting to do that.

Due to "If a new layout arrives in Decipher, it will only be able to auto-process if it's highly confident, else it will be held for manual verification. The training data (rules-based model) could only do this with keywords and previous training of this layout. If the additional ML is enabled it will use the training of unrelated invoices and could give you a false positive, incorrectly marking it as high confidence." - I am not sure how to set it. Is it QC option? Can you elaborate more or link the guide/documentation about that?

I already booked help session on Monday. Thanks for your assistance but still I cannot understand is this named by me "stealing case" normal situation.

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Athiban_Mahamathi — Thu, 04 Jan 2024 22:09:00 GMT

Hi Wojciech,

I had faced the similar issue in my current process. Here is how I resolved the issue.

Added a specific DFD version for this supplier using the supplier name as key field.
Removed all the sample headers for InvNo and added the exact sample header as per the supplier template. Alternatively, you can let the Decipher to learn the based on the InvNo position without any sample headers.

------------------------------
Athiban Mahamathi - https://www.linkedin.com/in/athiban-mahamathi-544a008b/
Technical Consultant,
SimplifyNext PTE LTD,
Singapore
------------------------------

RE: Decipher one field's header "takes" value of another field

WojciechRogmanStuffAutomation — Thu, 04 Jan 2024 22:26:00 GMT

Hi Athiban,

thanks for your input. In my case I cannot define specific version trigger as vendor because I have too many invoice vendors to proceed and every month new templates appears.
I am wondering if I can create Specific Version of DFD based on rule 'when some field value is '' (empty)' but I dont think it would work because problem is that I have to use same word on invoice for two differenct purposes - value of field A and sample header of field B.

I was thinking to create 2 batch types, with 2 different DFDs, one for extracting this word "INVOICE" as a value of one field and another DFD and batch type for using word "INVOICE" as a sample header for invoice number. But I think that it is far away from good practices and it multiplies time complexity.

------------------------------
Wojciech Rogman
------------------------------

RE: Decipher one field's header "takes" value of another field

Ben.Lyons1 — Mon, 08 Jan 2024 08:12:00 GMT

Hi Wojciech,

I understand your point, though I would still advise against using a keyword that appears multiple times in a document. It's best to allow Decipher to use it's training data ML to learn and capture these fields. Hopefully this can be demonstrated in your session today.

Using 2 DFDs for the same document doesn't tend to work as they will use the same training template, these are not currently locked to a specific DFD. Training data is a 'pool' of document training templates shared by all DFDs.

Any low confidence document or document that fails validation will be held for manual verification, even after auto-skip is enabled. To ensure all low confidence fields 'trigger' this behaviour, add the "Required" flag. Any low confidence/blank fields with this flag will not be automatically processed.

Thanks

------------------------------
Ben Lyons
Senior Product Specialist - Decipher
SS&C Blue Prism
UK based
------------------------------

RE: Decipher one field's header "takes" value of another field

stuart.mar — Thu, 01 Feb 2024 17:22:00 GMT

Hi Wojciech,

Your urgent issue is about a month old but thought I would reply anyway as a matter of general interest. If your field invNo is only populated when there is an invoice number then the field invTitle could be set to "Invoice" (instead of "Other") using a formula instead of trying to read the word "Invoice" from the document.

In the invTitle field add a formula to determine if invNo is blank:

IF(EQ(invNo, ""), "Other", "Invoice")

Don't forget to add "AutoCalculate" to the flags of invTitle.

------------------------------
Stuart Mar
------------------------------