DEFEATED! Community Boss Battle #2: Robot Healthcare

[Michael_S]

🔥 Attention, mighty intelligent automation professionals! 🔥

Once again, there's trouble in bot-land, where Bob the Bad Bot has been causing chaos by causing all of your automations to fail and stop unexpectedly!

In this Community Boss Battle, you'll need to suggest possible solutions to the problem below in order to destroy the boss and earn the Defeated Bob! badge for your community profile. Let's see what you're up against this time...

---

👾 The Boss: Robot Healthcare (100 HP) 👾

Your mission, should you choose to accept it, is to tackle this daunting question:

"What steps can you take to prevent your automations from failing, stalling, stopping or causing unexpected chaos?"

PS. Thanks to @Jignesh Khakhriya for inspiring this Boss Battle with Dr. Bot!

---

🛡️ The Rules of Engagement 🛡️

• Post your innovative, well-thought-out answers to slay the boss!
  Each viable answer will reduce the boss's hit points (HP) based on the quality of your response. The boss starts with 100 HP!
  
  
• Join forces with your fellow community members!
  Recommend another member's answer by clicking the "recommend" button, or reply to their answer and cheer them on. This teamwork will also deal damage to the boss!
  
  
• Keep the conversation positive and respectful.
  We're all in this battle together, so let's support each other and learn from one another. When posting, you don't need to give all the answers - one tip will do. Leave room for your fellow community members to contribute too!
  
  
• You'll receive a notification each time your answer or recommendation deals damage to the boss.
  Please note that we're calculating damage manually, so there may be some delay between posting a reply and receiving a notification.

Now, let's gather our creative powers and expertise to slay this boss together! The future of intelligent automation awaits your innovative ideas and solutions. Good luck, and let the battle commence!

The Boss' HP is currently: 0HP
🎉 Congratulations - you defeated Bob once more! 🎉

------------------------------
💙 Michael
(I'm part of the SS&C Blue Prism Community Team)
------------------------------

[Neel1]

Short answer from my side - Follow the template and Best practice in developing the bot. Retry mechanism and good alerting can make the bot sustainable.

------------------------------
Neeraj Kumar
------------------------------

[EVIPUTI]

Let me try answering it the Bob the Bad Bot language this time (haha!). Have used them in my previous exp. to achieve fail-proof automation.

Unleash Your Automation Superpowers by simply following below easy steps:

1. Testing Sorcerer (Selenium): Let the Testing Sorcerer cast spells of thorough testing using Selenium, uncovering any hidden bugs or issues before they can cause trouble.

2. Exception Ninja (Python, Java): Become an Exception Ninja with Python, skillfully handling errors and unexpected events, ensuring your automations never stumble.

3. Version Control Wizard (Git): Enlist the Version Control Wizard, powered by Git, to effortlessly manage changes and roll back to previous working versions when needed.

4. Alerting Enchanter (Datadog): Empower the Alerting Enchanter from Datadog to keep a watchful eye on your automations, alerting you to any anomalies in real-time.

5. Cybersecurity Guardian (Azure Security Center/AWS): Summon the Cybersecurity Guardian from Azure Security Center to safeguard your automation infrastructure and data from any threats.

7. Scroll of Wisdom (Confluence, Jira): Record your wisdom in the sacred Scroll of Wisdom, using Confluence, to create a knowledge repository for future automation journeys.

------------------------------
------------------------------
Vipul Tiwari
Senior Process Simplification Developer
Amazon
------------------------------
------------------------------

[KodiSrihari]

 To Maintain the Robot Health and smooth running of Automations, we should reboot the Robots regularly at the specified intervals which are interacting with so many Applications like JAVA, Mainframe and Desktop Applications.

------------------------------
Srihari
RPA Developer
------------------------------

[Digital_WorkerB]

@KodiSrihari hit the boss for 10hp!

@Neeraj Kumar hit the boss for 10hp!

@Vipul Tiwari hit the boss for 15hp!

It's looking mildly inconvenienced....

@Michael Shillingford applied 5hp bonus damage by recommending posts!

------------------------------
Hi 👋 I am a Digital Worker. Please check out my profile to learn more about what I do!
------------------------------

[Mukeshh_k]

Well it seems "Bob" the bad bot and the boss aren't done yet, suggesting few things from my side - 

Use the standard Blue prism advance development template and customise additional safeguarding to make it more advanced: (https://portal.blueprism.com/products/developer-jumpstart)

• Do clean up before and after - put additional logging and reporting i.e. using sub page named as heart beat (gets the machine name, screen resolution, session ID, date time & etc) and execute this page at the start and end of the process additional to process logs, set the screen resolution as mandatory for coordinates, surface use cases and other checks.
• Ping the IPs, Web addresses to check the health, get the status through CMD/Powershell batch file before actually jumping onto target applications. (One use case where we discussed about Browser automatic updates and webdrivers which needs manual updating and replacing - Powershell could be used to get the latest version of the browser and subsequently through decisions a sub process could be called to ensure the webdrivers are replaced/updated as per the current browser version to avoid automation failures.
• Use Tags and Status as references to ensure what's to be worked upon again on failures use these for self healing

Follow development standards, set up the right governance, do design authority for solution designing, do the development peer reviews to ensure coding standards has been followed through out before making it live.

• Example 1 : Developers discretion advised to incorporate Retry Logic in Process Flow, If retrying, sections should include the use of the "Navigate to Home Page" to restart the system to the correct place, all retry loops should have a fail safe to avoid infinite loops etc.
• Example 2 : When doing exception bubbling from Object- include the object name, page name and action details - it can simply and ease debugging time if required or could be used to feed the decision making what to be retried or not, from the process.

Periodic maintenance and monitoring of the environments/run time resources -

• Maintenance processes could include generating a report by fetching out the credentials expiry details and everything - Use the credentials object wisely for listed actions such as Generate and Set, Generate Password etc .
• Do a periodic restarts on maintenance windows, use communications from business or application teams effectively to notify support team in advance of update rolls, shutdowns etc.
• Archive what's redundant and only for audits, use proper segregation. 

Document Failures/Learning, retrospectives -

• Use document repository to log learning, failures, SOP solution for issues/challenges faced - do consider these learning or recommend people to go through those learning before attempting to automate similar use cases n etc.

There could be many more self checks, system checks based on use case to use case basis but learning is one constant part of your automation journey.

------------------------------
Kindly up vote this as "Best Answer" if it adds value or resolves your query in anyway possible, happy to help.

Regards,

Mukesh Kumar - Senior Automation Developer

NHS, England, United Kingdom, GB
------------------------------

[johnwild]

Totally agree with what has bee put previously, but I also have some of my own techniques which aren't often listed:

• Design to fail - always assume within the Solution Design and Build that systems are slow and unstable. This will force you to always consider the most efficient way to use a given system. I've seen so many processes built where failure hasn't been assumed, which has resulted in a slow and inefficient process (best case), or a constantly failing process (worse case)
• Garbage Collection - if your process uses any large collections or sub processes, regular "Garbage Collection" between tasks will help keep your runtime memory healthy
• Schedule daily restarts - We all know the typical response from an IT help-desk - "have you turned it off and on again?". Robots are no different. Use a windows scheduler service to restart your machines regularly to give your operating system and other software a chance to fully reset. Particularly with operating systems, logging out and in again isn't always enough.
• Tell your bots to clean up after themselves - the applications your bots use may end up taking a lot of temporary file storage which can impact performance over time. If you can, work out where these are and get your bot to delete them regularly, or after each session.
• Make friends with other areas - Maintain regular contact with IT support teams and wider business areas. Not only will they be able to help you if something goes wrong, but they will also likely found out about future change before you will. This means you can stay ahead of any changes, correcting potential problems before they start causing you any headaches.

Nothing will ever compare to using an appropriate template when you build a process, and aligning to best practice. The points above will help within most other things.

------------------------------
John Wilding
Robotic Architect
ReAssure Ltd
Europe/London
------------------------------

[Tejaskumar_Darji]

Here is my small take on this as per my experience,

1) Solution Design: Lay out your solution design in the best possible manner. I believe that design is an art and the key to having fail-safe automation. The better the design, the lower the possibility of issues in the implementation and execution of the solution.

2) Peer Code Review: Encourage other team members to review your logic. By actively seeking feedback during code reviews, you will notice significant improvements in your work. Keeping an open mind and being receptive to feedback can make a remarkable difference in the quality and effectiveness of your code.

3) Exception Handling: Assign the most skilled individuals to handle the exception handling part during the development process. Even a minor oversight in this area can result in many failures. Also, have a Design Authority team in place to review the automation.

4) Email-based alerts: Email is one thing that we always keep an eye out for hence use proper email IDs in exception handling sub-routines with details about the failures.

5) Fail Scenarios: Identify as many potential failure scenarios as you can during the testing phase. Ensuring extensive coverage of failure cases in Testing/UAT reduces the chance of encountering unexpected issues in Production later. Missing test cases during the initial phases can lead to problems down the line, which I'm sure everyone must have experienced. Hence it is crucial to identify failure scenarios early on to minimize the chances of encountering issues and reduce the time required for fixes.

6) Contingency plan & Infra support: Make sure to keep a plan ready for such failures be it a simple process failure or the outage of the server. Talk openly about this with all stakeholders, and pre-plan it. This is like the fire drill that we generally do at office premises.

"Design is an art and the key to having fail-safe automation" - Tejas

------------------------------
If I was of assistance, please vote for it to be the "Best Answer".

Thanks & Regards,
Tejaskumar Darji - https://www.linkedin.com/in/tejaskumardarji/
Technical Lead
------------------------------

[Babjee24]

Great suggestions from the peers on the community which made me learn new points that are to be noted down. I would like to add few more things here

• If you're using browser automations try terminating the sessions properly this will free up the ports where the browser is being launched by Blue prism and this would reduce the attach issues. Also we can make use of scripts that clears the cache memory of the browser resulting in closing down the browser completely.
• Though you've multiple applications in your automation if possible try keeping one application open at once with which Blue prism interacts. Close all other applications which frees up the RAM which would avoid the bot going into warning status. (This approach needs multiple launches but yet reduces the chances of launch failures)
• Try avoiding patch updates happening during the Bot timings (though it'll be a common patch release un-necessary popups like system restart needed creates chaos) though the Bot design is stable infra failures results in terminations and any reason for termination is an impact on the performance.
• I've seen change of machines at times fetches results. Some applications won't work without any specific reason on a particular virtual machine in this case try deploying it on another server which might give you results and then try mirroring that server.
• Adding to @Mukesh Kumar Have a process built in Blueprism which auto updates the passwords for the applications that are being used both on applications and in credential manager. This could be possible only if there is no cognitive decision making like handling captcha, multi factor authentication in updating/changing password so that maintenance could be made easy to a certain extent.
• As suggested by @johnwild make your bot fail during unit, integration and UAT testing phases the more it fails the more robust you can make it by handling maximum of the scenarios and route it to all the logic flows that are being used in your automation during testing though the flow/logic is from a template.
• We need to make sure a value is always present in the flags and counter variables(a hawk eye has to be present on these which we'll be unaware of the values being missed at times) when used in decisions/choice(s) which would result in internal errors.

------------------------------
Babjee Vangipurapu
Senior RPA Developer
Wonderbotz
India
------------------------------

[mwulff]

The simple solution: Log in as a BP administrator and delete The Boss. Problem solved, move on to a cold drink.

If this is not an option we can try:

1. Summon the well designed business object. It maintains state, application state and it handles errors and navigation thus seperating business logic for the application logic. Design around the "application as an api" thought and ensure that your exceptions are consistent and meaningful. 
2. Ensure that exceptions are for the exceptional states and not in your "happy flow", the boss loves it when you manage flow with exceptions.
3. Move your operations outside Blue Prism. The Blue Prism scheduler is too weak to fight this boss. Write your own for a +10 attack against the boss, or buy a solution for a +2 attack buff.
4. If your lifestyle allows it maintain a Ballmer-peak for +4 to code-skills 😉

I didn't really have more time to come up with some fun stuff

------------------------------
Michael Wulff Nielsen
Team-lead
Odense Kommune
Europe/Copenhagen
------------------------------