How to scrape website for broken hyperlinks

[JelleKoning]

Dear reader,

With BluePrism, is it possible to scan a website for broken hyperlinks within the page/site? Furthermore, we would like to export the results (e.g. via CSV) for processing by the website maintenance team. We know Automation Anywhere has such an Action within their product (please see link below). Does BluePrism have a similar (ready-built) solution? If not, what would be the best way to tackle this problem within the BluePrism Enterprise Suite?

https://docs.automationanywhere.com/bundle/enterprise-v2019/page/enterprise-cloud/topics/aae-client/bot-creator/commands/cloud-using-find-broken-links.html

[PvD_SE]

Hi Jelle,

I would expect this in its simplest form to be a let-the-process-click-each-link-and-scan-for-404-errors kind of solution, which is what the AutomationAnywhere seems to be doing. This as I would assume you cannot see on the link itself if it still is functional or not. 

That said, all of this would be greatly easier if the website maintenance team could deliver a list of links to be inspected, rather than have the robot look for all links visible or hidden on a site.

Happy coding!
---------------
Paul
Sweden