Issue with UIA Spy Mode Not Extracting Element Display Values in Windows Application

[Akshay_MMC]

Hi Team, @devneetmohanty07 

We are currently working on a Windows-based application. During spying, we noticed that the Windows spy mode only works on the main window and not on any other elements. Although we are correctly attaching to the running application, UIA spy mode successfully retrieves the UIA Name for some elements like label texts. However, for certain elements on the same window, the UIA Name is blank despite the elements having visible values (please refer to the attached screenshot).

Could you please guide us on what might be causing the spy mode to fail in extracting the display values of
these elements during spying?

[david.l.morris]

Hi, this looks pretty standard to me. In my opinion, the root cause of the difficulty of automating desktop applications in Blue Prism comes down to that it does not have the concept of selectors that span multiple elements and/or the concept of anchors such as being able to target an ancestor/parent and then click on a child of it. Browser mode, surface automation, and Smart Vision don't have this issue as they do have that capability. I imagine Blue Prism intends to improve this in the future, but I've never come across a real solution to this. There are some workarounds that you could try.

Here are some things that I and other developers have tried in scenarios like this:

• Verify that the parent attributes also don't have values in the UIA element. I assume you already checked this so I expect this won't help.
• Try AA (Active Accessibility) as this quite frequently for me does have attributes filled in when UIA does not.
• Find a given element by tabbing from a known element that Blue Prism can uniquely find. For example, if you can have BP find the label of the combo box, then you could have Blue Prism click/select the label and then you send TAB which would select the combo box and then you send keys to select the right value.
• Use Match Index to uniquely identify the target element. This is generally not good practice, but sometimes you just don't have any other choice. The way I usually handle this while being safe is to have extra logic before interacting to make sure the rest of the screen is in the standard state. For example, I might check to see that the expected label is still the nth index or check how many children there are of the parent element (the window) or anything else that seems possible just to sanity check that the window is still designed like it was when you spied it originally, and then just move forward with Match Index.
• Use Surface Automation. This is actually quite good in some apps. Everyone will rightly tell you to stay away from using it unless you absolutely have to. As I said, they are right to do so. However, I have had an automation running for like 6 years that depends solely on Surface Automation for automating a desktop app that is inside an RDP session, and it hasn't needed a change in years. If the app is stable, this could work perfectly for you.
• Use Smart Vision. This is like an upgrade version of Surface Automation that is genuinely kind of amazing. I think it requires a special license so you may have to pay extra for it. I have absolutely no idea, and I only say this because I think it requires installing a license.

[Akshay_MMC]

Hi David,

Thank you for your reply.

1. I have already checked the parent attribute, and it does not have a value.
2. We tried using AA, but the results are the same as with UIA Mode.
3. We attempted this approach, but it is not working since the labels are not being selected when clicked, and the Tab key does not move to the next field.
4. The match index helps us navigate from one field to the next, but we are facing an issue with reading the value (UIA Name="").
5. This was our last option, and we tried it using region mode, which works for some of the elements.
6. Obtaining a Smart Vision license is not an option for us at the moment.

[Omar-Abu-Snineh]

Adding to what David has already mentioned, based on your update it seems that the issue is no longer only about identifying or navigating to the element, but specifically about the fact that the application is not exposing the displayed value through UIA or AA.

Since the parent attributes are also empty, AA gives the same result, and Match Index only helps with navigation but not with reading the value, this suggests that the value may be visually rendered on the screen without being published as an accessible property. In that case, Blue Prism can locate the control, but it cannot reliably read what the application itself does not expose through the automation layer.

At this point, I would separate the problem into two parts:

1. Interaction with the field
If Match Index can reliably move from one field to another, it may still be useful for interacting with the screen, provided that additional validation is added before the action. For example, checking that the expected labels, window title, number of child elements, or screen layout are still consistent before proceeding.

2. Reading the displayed value
If the value is not available in UIA, AA, parent attributes, or child attributes, then reading it through standard spying may not be feasible. In that case, the alternatives would be to retrieve the value from another source such as an export, clipboard behavior, application report, database, API, log file, or to use Region Mode/OCR where it is stable enough.

If Region Mode works for some elements, it may be worth standardizing the screen as much as possible before reading: fixed resolution, fixed zoom level, maximized window, consistent theme, and controlled RDP/session settings. This can make image-based extraction more reliable, although it should still be treated as a fallback rather than the preferred approach.

The most robust long-term solution would be to involve the application team and ask whether these controls can expose proper accessibility properties, such as Name, Value, Automation ID, or accessible text. Without that, Blue Prism may not have a dependable technical handle to read those values directly.

So in short, I agree with David’s points, but based on your findings, this looks less like a Blue Prism spying configuration issue and more like an application accessibility/exposure limitation. Blue Prism can only consume the properties made available by the target application; if the display value is only visually painted and not exposed through UIA/AA, then a workaround or an application-side change would likely be required.

[Akshay_MMC]

Thank you, Omar-Abu-Snineh, for understanding the issue and for sharing your suggestions. We are currently following the same approach you outlined in part 1. For part 2, we are using the Clipboard alongside send keys, and we are also working on implementing region mode wherever possible by standardizing the screen.

We are coordinating with the application team on this issue, as there are several automation opportunities lined up for this application. Thanks again.