Control Room suddenly useless - Can start any process on any machine whether it's logged in or not - obvious error and swiftly terminates

[BenKirimlidis]

Hi, In BP Control room, whole system became suddenly useless today. We have 10 resources, have multiple schedules that run and occaisionally kick off process manually when required. Suddenly today, control room stopped workign normally.  Now all process permanently show as logged out. -Using LoginAgent I can login to any resource however, while the session marks itself as a success and completes.  The machine is NOT logged into. -Can do this over and over, always a success but the machines status never changes. -Likewise I can manually kick off processes on any machine.  Normally, if the machine is not logged into, the process will always terminate immeadiately as it's not on the windows home screen.  However now, all process will allow the process to "start".  This morning I could not run a process on a machine that wasn't logged into.  Then process fails because obviously the machine is not logged into. Have tried - Rebooting all the machines, problem persists - With IT, remoting into resources.  This allows the BP listern to start workign normally so i can run processes.  But as soon as the machine logs out.  The same behavior outlined abvoe resumed. - With the machine remoted into, terminating the BP Listener service and restartign it, then logging out.  Does not work. Only way we can currently run our BAU is to have a PC remote int othe Run Time Resource so that the Listener starts working, run processes manually while moving the mouse. Naturally this is completely unacceptable and cannot continue. Been trying to solve the problem all day and have gotten nowhere. Open to any and all suggestions. Thanks in advance.

[John__Carter]

In reality all the login process does is pass the credentials to Windows, it does not wait for Windows to complete the user log in. So it would be worth double checking that 1) the Windows user/password are valid, 2) the user/password in the BP credential store are OK, 3) after logging into Windows the resource PC starts automatically, 4) the event log on the resource machine.

[AmiBarrett]

I'm curious about your usage of wording when you say ""the machine logs out"". Do you actually mean that the user account is logged out, or that the workstation simply goes to the lock screen? If the former, try setting up to just lock instead of logging the account out entirely. (Though if these are VMs, it seems silly to require it to lock to begin with.) Theoretically, you should only need to use Login Agent to enter the credentials while the VM is locked, rather than entirely logged out. This should save time and create additional stability. Just make sure that the VM is configured through Windows to automatically login the proper account on reboot (hint: check out netplwiz ). Likewise, how are you remoting into the system? If it's RDP, that's known to break Login Agent. Try using something like TeamViewer.  

[AmiBarrett]

If you absolutely have to use RDP to connect, I'm attaching a batch script as a txt file that'll unlock the system from an RDP disconnect (Assuming the forum will play nicely - if not, I'll have to make another post). Set this up as a scheduled task to kick off on remote disconnect from the resource, and make sure it runs with elevated privileges.   I know it may seem excessive to have this many tscon lines - but sometimes there are more sessions than you may be aware of. We had a resource reach 15 once. (Not at all a pretty sight.) Please note that there is a known bug with this particular scheduled task trigger that may cause the script to kick off on RDP connection, rather than just disconnection. The attached file has some logic included to check for this. In case of failure, you should be able to RDP in as any other account and disable the task.

[BenKirimlidis]

Thanks @John & @AmiBarrett for your input. Issue was resolved yesterday afternoon. It was a perfect storm of several small human errors which were not caught in time.  - Two environments were not in sync when a developer edited a VBO in our QA environment  (all edits should have been done in Dev) (mistake #1) - The edit broke the object - The QA object was imported back to our Dev environment which is against our protocols (mistake #2) - A package was created which included the overwritten object which was then imported into PROD, it should not have included the object (mistake #3) - The object is used in a few custom VBOs including a modified LoginAgent. - Package was imported into Production on Friday - The modified VBO causes LoginAgent to essentially do nothing and always report a success. - No changes were made at a process level so it took us longer to diagnose the issue (i ll call this mistake #4, i should noticed this sooner) It was a face palm moment when we figured it out. @Amibarett When I said ""the machine logs out"", I meant that when we remoted into the resource, it reported to the control that it was available to run processes so long as the remote desktop session is dispalyed on a monitor. So everything works normally from then.  But when we ran a 'LogOut' or a 'MachineRestart' process.  We could still not use the LoginAgent's 'Login' function. The thing that was throwing me off was that when we ran the Login, the it reported successful completion and the Log showed all steps completed successfully. But once we synced up our environments with the correct version everything returned to normal.  

[AmiBarrett]

Had a request earlier to re-attach the script. (Also, fun fact: Edit won't allow you to attach anything after the fact. 😛 )

------------------------------
Ami Barrett
Lead RPA Software Developer
Solai & Cameron
America/Chicago
------------------------------

[DavidVizcaíno]

Hi @AmiBarrett

Will this .bat allow me to close RDP and have my processes still running? Sadly RDP is the only tool we have to connect to server and resource PCs. 

​I've configured 2 VMs, one as server and the other as client. I've scheduled 2 different process at different schedules. One uses Login Agent to login, and the other does 'Global Mouse Clic Center' actions on calculator for 2 minutes.

If I connect using RDP to the resource PC while process is running, it will "pause" for a few seconds and then starts clicking buttons again. If I close the RDP session, control room will show the process as "running", but it will not clic on any button, like if it paused forever. 

Any friendly suggestion? ​

------------------------------
David Vizcaíno
RPA Programmer
EY
America/Bogota
------------------------------

[AmiBarrett]

@DavidVizcaíno

That is the intent of the bat file, yes.

Treat a VM like any desktop without an attached monitor. If you RDP into a physical box, the local system will display a lock screen, while the active session is being virtualized. Upon logging out, there is no longer any virtualization, thus the local system has no desktop to interact with. Rather than permenantly pause, this should cause a failure on anything requiring accessibility, UIA, or Region mode access. 

Its also worth noting, that because it has no screen attached, the default resolution will be somewhere around 800x600. You can change this via powershell on disconnect, to 1200x800, which I think is the max resolution without an attaches screen. These resolution changes may have an effect on your automation. 

The pause you're experiencing may have to do with it reconfiguring the session between the local and remote connections.

------------------------------
Ami Barrett
Sr Product Consultant
Blue Prism
Plano, TX
------------------------------

[DavidVizcaíno]

@AmiBarrett

​Hi, I've configured a new task to run a the .bat, triggering on RDP disconnection according to the following event IDs:

• EventID – 24 (Remote Desktop Services: Session has been disconnected) – a user has disconnected from the RDP session;
• EventID – 25 (Remote Desktop Services: Session reconnection succeeded) – a user has reconnected to the existing RDP session on the server;
• EventID – 39 (Session <A> has been disconnected by session <B>) – a user has disconnected from RDP session by selecting the corresponding menu option (instead of just closing the RDP client window). If the session IDs are different, a user has been disconnected by another user (or an administrator);
• EventID – 40 (Session <A> has been disconnected, reason code <B>). Here you must view the disconnection reason code in the event description. 
• EventID 23 (Remote Desktop Services: Session logoff succeeded), refers to the user logoff from the system.

I noticed the bug, as you said the task also triggers on login, so I implemented some logic into a powershell script to check for the session state and only run the .bat when session state is disconnected. When connecting with RDP: session is active, closing RDP: session is dissconected. I also managed to run the powershell script on hidden mode so no interference with the UI is shown. 

Your .bat runs OK, only when I close the RDP sessions it brings back the console session. The only problem is that is shows the 20 cmd windows, one for each tscon line you mentioned on previous comments. Therefore my process fails to find the buttons on my application due to all the cmd windows opening and closing. 

I noticed the "@ECHO off"​ on first line of the.bat, which will mean everything runs without windows displaying. Am I running an outdated version of the .bat, maybe? 

I also tried changing the user account under which the task runs on the task scheduler to SYSTEM, or checking the "Run whether is logged on or not" checkbox. The cmd Windows don't show but the .bat doesn't bring back the session to console. 

Do you have any updated version of the .bat? How did you get this to work?

Thank you

------------------------------
David Vizcaíno
RPA Programmer
EY
America/Bogota
------------------------------

[RaúlHernández]

This solution is pretty amazing and stunning.

I would like to understand to source of this problem (not able to see the history conversation).

Do you have a control room to start the process to avoid to enter into the Runtime Resource? Or, do you any a special business requirement to do that?

Just to know.

Best regards.

------Original Message------

@AmiBarrett

​Hi, I've configured a new task to run a the .bat, triggering on RDP disconnection according to the following event IDs:

• EventID – 24 (Remote Desktop Services: Session has been disconnected) – a user has disconnected from the RDP session;
• EventID – 25 (Remote Desktop Services: Session reconnection succeeded) – a user has reconnected to the existing RDP session on the server;
• EventID – 39 (Session <A> has been disconnected by session <B>) – a user has disconnected from RDP session by selecting the corresponding menu option (instead of just closing the RDP client window). If the session IDs are different, a user has been disconnected by another user (or an administrator);
• EventID – 40 (Session <A> has been disconnected, reason code <B>). Here you must view the disconnection reason code in the event description. 
• EventID 23 (Remote Desktop Services: Session logoff succeeded), refers to the user logoff from the system.

I noticed the bug, as you said the task also triggers on login, so I implemented some logic into a powershell script to check for the session state and only run the .bat when session state is disconnected. When connecting with RDP: session is active, closing RDP: session is dissconected. I also managed to run the powershell script on hidden mode so no interference with the UI is shown. 

Your .bat runs OK, only when I close the RDP sessions it brings back the console session. The only problem is that is shows the 20 cmd windows, one for each tscon line you mentioned on previous comments. Therefore my process fails to find the buttons on my application due to all the cmd windows opening and closing. 

I noticed the "@ECHO off"​ on first line of the.bat, which will mean everything runs without windows displaying. Am I running an outdated version of the .bat, maybe? 

I also tried changing the user account under which the task runs on the task scheduler to SYSTEM, or checking the "Run whether is logged on or not" checkbox. The cmd Windows don't show but the .bat doesn't bring back the session to console. 

Do you have any updated version of the .bat? How did you get this to work?

Thank you

------------------------------
David Vizcaíno
RPA Programmer
EY
America/Bogota
------------------------------