topic RE: Error in Microsoft Azure Computer Vision v3.0 in Digital Exchange

Error in Microsoft Azure Computer Vision v3.0

himanshu__sharma — Tue, 22 Sep 2020 11:59:00 GMT

Hi All,

While working with asset - Microsoft Azure Computer Vision v3.0
I am facing one issue in functionality of - Get Read Results.

I am able to run API to get results in JSON format (Response Content Data variable)
But the output collection ( Read Results Response ) is coming as blank.
That means this utility by itself is not converting response to collection and giving a blank collection in results.

I tried to use BP another asset to convert text to collection , but it is also giving me error

Any help would be much appreciated!!!

------------------------------
Himanshu Sharma
Senior Software Engg
Mercer
Asia/Delhi
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Fri, 25 Sep 2020 13:53:00 GMT

Hi Himanshu,

I just took a look at the definition of the JSON path query on that action. Right now it's set to:
$.readResults

Based on Microsoft's API documentation (Computer Vision 3.0 Reference) I believe it should be:

$.analyzeResult.readResults

I'll have someone take a look at the skill, verify the correct value, and get the skill updated. In the meantime, you can manually change the value in the Web API definition by going to System -> Objects -> Web API Services and editing the action within the Microsoft Computer Vision 3.0 object.

Cheers,

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Tue, 29 Sep 2020 16:09:00 GMT

@ewilson - Hi, I have question related to Azure Json response.
Suppose below is my image. After I get a Json response back from Azure (which has everything like Page, text, words, coordinates etc.), I want to get the text from it as displayed in page - left to right. top to bottom
which will be -

Text should be extracted from JSON in same way -
This is my Sample Image
Key: Testing
Extract values

What is the best way to extract the text from JSON response. We probably just need 'Page' (for multipage pdf) and 'text' key from JSON response with Read analyze.
Will this require a custom C# code and do we have something available in BP for this?

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Tue, 29 Sep 2020 16:39:00 GMT

Hi @Mayank Goyal,

There are a couple ways you could go about extracting specific information.

There's a utility VBO available on the DX (it may be included in the product install as well in the VBO folder) called Utility - JSON. With this VBO there's an action that will take a JSON blob as input and return it as a parsed Collection. Then you just move through the Collection and pull out what you want.
Another option would be to use a Code stage and the Newtonsoft JSON library to pull the specific information out via a LINQ or JSONP query. This is similar to what the skill is doing to create the Read Results Response collection. Newtonsoft's documentation is pretty good with a decent number of examples. I think the problem you'd face with doing it this way is that if you have multiple Page entries you'll also have to deal with counting them as they're returned as a JSON array.

Let me know if this helps.

Cheers,

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Tue, 29 Sep 2020 20:39:00 GMT

@ewilson - Can you help getting a custom C# action created as reusable functionality on DX portal, I am looking for a action were I can provide Json from read/analyze as input and I get as output something -

Page Text
1 All Text from Page 1 (in sequence - left to right and top to bottom)
2 All Text from Page 2 (in sequence - left to right and top to bottom)
and so on..........

This might be useful to lot of Blue prism developers.

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Tue, 29 Sep 2020 22:03:00 GMT

@Mayank Goyal,

Attached is a quick-and-dirty VBO that can be used to parse the Response Content output of the Get Read Results action. It will return a Collection that contains the page number and complete text found on that page for every page.

I say this is "quick-and-dirty" because I just threw it together. There is no exception handling in it. So, if you want to use this in production I would encourage you to review the VBO and added any additional exception handling, logging, etc where you see fit.

Cheers,

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Tue, 06 Oct 2020 04:44:00 GMT

@ewilson - Thanks a lot, this was very helpful, will get in touch if stuck with anything else on Azure.

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Mon, 12 Oct 2020 16:52:00 GMT

@ewilson - I am getting an error - "Internal : Could not execute code stage because exception thrown by code stage: Data Type mismatch in array" with blue prism VBO Utility Json when converting Json to collection. Please help on the same.
Its seems related to data type, may be if we can have something that considers everything as String might be useful, however not sure how the code works internally in this VBO.

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Mon, 12 Oct 2020 17:21:00 GMT

@Mayank Goyal,

I just went over this with a colleague who experienced the same behavior with another Microsoft service. The issue is that Microsoft are returning mixed numeric types, in some cases, in their JSON arrays. There is a method in the global code of the JSON VBO that checks to determine if the types in an array don't match because that's essentially a bad programming practice.

Below is an example of how you can change that method in the global code so that an array that has numeric types in it (even if they're different ex. Double vs Int64) will be allowed to pass the check. Just cut-and-paste this code in place of the existing DeserializeArray() action in the Global Code section of the VBO.

Private Function DeserialiseArray(ByVal o As JArray, ByVal populate As Boolean) As DataTable
        Dim dt As New DataTable
        Dim isNumArray As Boolean

        Dim first As Type = Nothing
        For Each e As Object In o
            If first Is Nothing Then
                first = GetTypeOf(DeserialiseGeneric(e, False))
                If (IsNumeric(e)) Then
                    isNumArray = True
                End If
            End If

            If (GetTypeOf(DeserialiseGeneric(e, False)) IsNot first) Then
                ' Check the array type. If it's numeric we want to make sure the subsequent values are not simply differing number types (ex Int64 vs Double).
                If (Not isNumArray) Or ((isNumArray) And (Not IsNumeric(e))) Then
                    Throw New Exception("Data Type mismatch in array")
                End If
            End If
        Next
        If first IsNot Nothing Then
            dt.Columns.Add(JSON.Array, first)
        End If

        If populate Then
            For Each e As Object In o
                Dim dr As DataRow = dt.NewRow()
                dr(JSON.Array) = DeserialiseGeneric(e, True)
                dt.Rows.Add(dr)
            Next
        End If

        Return dt
    End Function

We made add this logic to the published VBO at a later date.

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Mon, 12 Oct 2020 18:54:00 GMT

@ewilson - Before I create a copy of VBO and update this code, i want to understand from you what if we read everything as text (String) from Json and when putting it in collection, with that we will never need to worry about data type?
Or the fix you have provided already takes care of this? I am not 100% into the code of this VBO at this point of time and only understand it on high level, however want to ensure a permanent fix before I put anything on my environment?

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Mon, 12 Oct 2020 19:25:00 GMT

@Mayank Goyal,

As I mentioned before, the point of the function is to flag any cases where the developer may face an array with mixed data types. Could JSON arrays be treated as string only? Yes, but then in cases where a developer needs numeric values (ex. for calculations) they would then have to cast those values back to numeric data types. That's why we're discussing whether to make a change to that VBO.

The revised code I posted checks to see what the data type is of the very first item in the array. It then assumes every subsequent item in the array will also be of that same data type. If the first item is a numeric data type, it will verify that every other item is of a numeric data type.

So, if the first item is a number and the second item is text it will still throw the "Data type mismatch" exception. If the first item is a number (ex. Int64) and the second item is another number, but of a different type (ex. Double), it will not throw the exception.

Cheers,

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Mon, 12 Oct 2020 19:57:00 GMT

@ewilson - Thanks for your input, so if you consider with OCR, the results depend on quality of images and hand writing, so if in a array first item is coming as True and for 2nd item Azure OCR read it as Lroe, we will get error as it wont be a boolean anymore.
This is why i was thinking with OCR technology, we can have everything as String and developer can handle the logic later to map string to any required datatype, please let me know your thoughts?

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Mon, 12 Oct 2020 20:06:00 GMT

@Mayank Goyal,

I'm not sure that case would apply to OCR because everything returned by Azure Vision, as far as what it's read, is treated as text I believe. The area that we run in to the data type mismatch exception is with their bounding rectangle array definitions in the JSON response, but I understand your point. We'll take it under advisement.

Cheers,

------------------------------
Eric Wilson
Director, Partner Integrations for Digital Exchange
Blue Prism
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

MayankGoyal2 — Mon, 12 Oct 2020 21:18:00 GMT

@ewilson - If you think everything is already read as text, let me go ahead and create a copy of this VBO with updated code

------------------------------
Mayank Goyal
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

Alexander.Duff — Tue, 05 Oct 2021 16:07:00 GMT

Have you published this VBO on the DX as I can't find anything

However, the idea of producing a collection of:

Page Text
1 All Text from Page 1 (in sequence - left to right and top to bottom)
2 All Text from Page 2 (in sequence - left to right and top to bottom)

appeals to a use-case I currently have.

------------------------------
Alex Duff
RPA Developer
Lowell Financial
Europe/London
------------------------------

RE: Error in Microsoft Azure Computer Vision v3.0

ewilson — Tue, 05 Oct 2021 16:33:00 GMT

Hi @Alex Duff,

Which VBO are you referring to?

Cheers,

------------------------------
Eric Wilson
Director, Integrations and Enablement
Blue Prism Digital Exchange
------------------------------