Recommended Approach to Dynamic Website Spying

[theBelovedKing]

Good morning all,

We are currently working on a project that requires us to spy a website and download files from it. Everything worked fine until the automation began to fail. On close inspection, we observed that the web interface had changed (i.e., the website has been redesigned by the owner), and the location where the BOT used to click and download files was moved to a different section. 

In addition, Google Chrome (the browser with which this automation was built) has recently been updated (about twice now), changing how downloads are handled, eventually breaking our automation.

These two incidents made us re-spy the new interface to get our automation up and running again.

Now, my questions are:
1. What is/are the best approach(es) to automating a website that constantly changes due to UI and browser improvements/updates?

2. Is there an alternative to creating an end-to-end automation of a webpage?

What we want to achieve is a solution that does not easily break, and is robust to the dynamics of webpage UIs and browser upgrades. Or should we reconsider the aspects currently being automated?

We are open to suggestions. Your kind feedback is deeply appreciated.

Thanks and best regards,

------------------------------
Kingsley David
------------------------------

[Daniel_Sanhueza]

Hello Kingsley

This is definitely one of the most annoying scenarios that can occur when automating web platforms that constantly change. Therefore, I will respond to these topics based solely on my experience, and I'm eager to see other perspectives on this post.

1. The best approach when automating a website that undergoes frequent changes due to UI and browser improvements is, according to best practices, to set the attributes of the elements (Application Modeler) to the minimum necessary. In my experience, when dealing with such platforms, the only attribute that remains unchanged is the "WebID" (Browser Automation). Combining this with "Match Index=1" has proven to be the most reliable approach in these situations. Rarely does this fail, but if it does, I experiment with different combinations of attributes to find the most reliable one while keeping it to a minimum.

2. If you have the opportunity to request the IT Department to create a web service that allows you to download these files, I think that would be excellent because you can use blue prism to consume this ws. However, this option is only feasible the organization has the "availabiltiy" to help you in this situation.

Hope its help and I'm looking to see other perspectives

Regards

------------------------------
Daniel Sanhueza
RPA Professional Developer
Deloitte
America/Santiago
------------------------------

[theBelovedKing]

Thanks a lot Daniel for these suggestion, they are well appreciated.

At the moment, the organisation has not exposed any APIs for our consumption; I really wish we could tell them not to update their UI anymore (funny though).

I will stilt wait for further suggestions from others and apply them after which I will provide the outcome of the various tests on this platform.

Warm regards,

------------------------------
Kingsley David
------------------------------

[John__Carter]

To add to what Daniel said, the trick is to try to anticipate which attributes might change. The most obvious one is Path, which assumes the location of the element within the HTML structure is permanent, but often with a dynamic web page, this assumption will be incorrect. The effect is that your VBO might seem fine at design-time, but later at run-time it has problems. In simple terms the techniques to deal with this challenge are:

1. Avoid using the changeable attributes
2. Use wildcard matching (although this won't work for Path)
3. Use dynamic matching, where rather than use a hardcoded attribute value, you calculate a value in the diagram and pass it into App Modeller
4. Use an Xpath expression. This is a special (non-Blue Prism) syntax that allows you to specify multiple attributes in a single expression. Although it requires an understanding of HTML, Xpath can do things that regular spying simply cannot. There is plenty of educational material online, and this short course is available at BPU  https://blue-prism.docebosaas.com/learn/course/16924/blue-prismr-spying-using-css-selector-and-xpath
5. Use CSS selectors. Similar to Xpath, in that you deselect all attributes except Path, where you use an expression to identify your element.

------------------------------
John Carter
Blue Prism
------------------------------

[harish.m]

As others mentioned  - Its always better to look other options like Web API and if it is internal application you can check  whether you can access data from the database and see any view or Stored procedures  exist similar to the downloadable file.

If  you are able to use UI mode I prefer UI mode  with the options like UIA name, UIA control Type 
And if you are going to browser mode As john carter mentioned couple of good options like CSS selector, Xpath.

Other Approach i follow if some navigation  or button is potential to  change i used to spy in different modes  and open logs on these spots( Analyze  the logs  and keep the  one has more success rate  )and if one of the regular path is filing in that particular button I used to retry with different mode. but   its not good idea to implement every where but only certain spots having issues.

------------------------------
-----------------------
If I answered your query. Please mark it as the "Best Answer"

Harish Mogulluri
Lead developer
America/New_York TX
------------------------------

[theBelovedKing]

Thanks Harish and John for your valuable suggestions.

We will explore these possibilities, and feedback will be provided on the most effective approach we eventually adopted.

Thanks again.

------------------------------
Kingsley David
------------------------------

[MichealCharron]

@theBelovedKing

I wholeheartedly agree with @John Carter on the use of XPath and CSS Selectors for dynamic websites. We use XPath for almost all our chromium web elements these days. 

I'd just like to point out that one thing the BPU course does not mention is that DevTools has a little known feature that you can use by pressing Ctrl-F when on the "Elements" tab. This brings up a search bar which you can type XPath or CSS Selectors directly into the search box that gives you immediate feedback on whether the element is found or not. 

The other advantage of using this method over the "Console" tab method is that you can also see the number of elements that your XPath or CSS Selector catches because it will give you a count (most times you want to get that count to 1). The best advantage though is that you can copy what works directly from that search box into the appropriate attribute in BP.

(yes I know I should have used the "id" in both of the above cases but where is the fun in that)

------------------------------
Micheal Charron
RBC
Toronto, Ontario
Canada
------------------------------

[theBelovedKing]

Thanks @Michael Cohen,

I have taken some time to explore the XPath and CSS alternatives, and I have had improved success using the XPath.
I used the course title suggested by @John Carter in Blue Prism University to understand the basics of matching by XPath/CSS.

Also, I used the Match ID = 1 suggested by @Daniel_Sanhueza for some other elements I was spying and recorded an improved percentage of success.

I will keep exploring both options on a use case basis to the point where we instinctively know which approach will be best suited for each scenario in question.

I will provide further updates on this thread with recent discoveries along our learning journey.

Thank you all for your support and guidance.

------------------------------
Kingsley David
------------------------------

[faheemsd]

Dear @Kingsley David

For your post, I would like to share my experience.

1. Make sure you're using the Chrome browser to develop automation solutions in development machines by using the Dev, Test, and UAT URLs for that specific application. Then, deploy the same code into the production environment.
2. About the changes made to the Application UI Each time there is a UI change to the application, I would request your application team to notify the RPA team know about it so they can work on it or check what needs to be updated from the code perspective in the Development Environment. I would also suggest informing your application team when the new application goes live. Using this method, you can automate in a way that is organized and won't cause any disruption with the business team's daily operations.
3. Please advise your IT Team not to upgrade the Chrome browser on RPA machines without notifying the RPA team. This is because the automation flow will be affected when the IT team upgrades the Chrome browser without informing the RPA team, which will destroy the automation in both the development and production environments. The RPA team must first install the most recent version of the Chrome browser on a development machine and thoroughly test the automation procedure. In the event that the latest version of the Chrome browser undergoes any modifications, the RPA team must ensure that the updated code remains compatible with both the earlier and newer versions of the browser.
4. Upon the completion of all testing on the Developme computer using the latest version of Chrome, the RPA Team will proceed with the production deployment, utilizing any recent modifications made to the latest version of Chrome.
5. Ask the IT department to update the RPA production machines to a more recent version of Chrome after that. With the help of the Chrome updates, we can perform our Automation more effortlessly in this way.
   
   In the unfortunate circumstance that we lack the APIs to work on it, I have offered my ideas for the UI Application Automation based on the changes to the Chrome update and the application UI. We do not need to worry about application UI updates or Chrome upgrades if we have ready-to-use APIs.

------------------------------
SYED FAHEEM
RPA Developer
Arab financial Services
Manama
------------------------------

[Daniel_Sanhueza]

I think this thread has a good source of information that can be useful in the near future for anyone who's looking for this kind of automation and should be on #BPTechTips 

Regards!

------------------------------
If this was helpful, please mark it as the best answer 🙂
Daniel Sanhueza
RPA Professional Developer
Deloitte
America/Santiago
------------------------------