Control Room & Scheduler

Expand all | Collapse all

Control Room suddenly useless - Can start any process on any machine whether it's logged in or not - obvious error and swiftly terminates

  • 1.  Control Room suddenly useless - Can start any process on any machine whether it's logged in or not - obvious error and swiftly terminates

    Posted 04-06-2019 00:07
    Hi, In BP Control room, whole system became suddenly useless today. We have 10 resources, have multiple schedules that run and occaisionally kick off process manually when required. Suddenly today, control room stopped workign normally.  Now all process permanently show as logged out. -Using LoginAgent I can login to any resource however, while the session marks itself as a success and completes.  The machine is NOT logged into. -Can do this over and over, always a success but the machines status never changes. -Likewise I can manually kick off processes on any machine.  Normally, if the machine is not logged into, the process will always terminate immeadiately as it's not on the windows home screen.  However now, all process will allow the process to "start".  This morning I could not run a process on a machine that wasn't logged into.  Then process fails because obviously the machine is not logged into. Have tried - Rebooting all the machines, problem persists - With IT, remoting into resources.  This allows the BP listern to start workign normally so i can run processes.  But as soon as the machine logs out.  The same behavior outlined abvoe resumed. - With the machine remoted into, terminating the BP Listener service and restartign it, then logging out.  Does not work. Only way we can currently run our BAU is to have a PC remote int othe Run Time Resource so that the Listener starts working, run processes manually while moving the mouse. Naturally this is completely unacceptable and cannot continue. Been trying to solve the problem all day and have gotten nowhere. Open to any and all suggestions. Thanks in advance.


  • 2.  In reality all the login…

    Posted 04-08-2019 19:11
    In reality all the login process does is pass the credentials to Windows, it does not wait for Windows to complete the user log in. So it would be worth double checking that 1) the Windows user/password are valid, 2) the user/password in the BP credential store are OK, 3) after logging into Windows the resource PC starts automatically, 4) the event log on the resource machine.


  • 3.  I'm curious about your usage…

    Posted 04-08-2019 21:41
    I'm curious about your usage of wording when you say ""the machine logs out"". Do you actually mean that the user account is logged out, or that the workstation simply goes to the lock screen? If the former, try setting up to just lock instead of logging the account out entirely. (Though if these are VMs, it seems silly to require it to lock to begin with.) Theoretically, you should only need to use Login Agent to enter the credentials while the VM is locked, rather than entirely logged out. This should save time and create additional stability. Just make sure that the VM is configured through Windows to automatically login the proper account on reboot (hint: check out netplwiz ). Likewise, how are you remoting into the system? If it's RDP, that's known to break Login Agent. Try using something like TeamViewer.  


  • 4.  If you absolutely have to…

    Posted 04-08-2019 21:49
    If you absolutely have to use RDP to connect, I'm attaching a batch script as a txt file that'll unlock the system from an RDP disconnect (Assuming the forum will play nicely - if not, I'll have to make another post). Set this up as a scheduled task to kick off on remote disconnect from the resource, and make sure it runs with elevated privileges.   I know it may seem excessive to have this many tscon lines - but sometimes there are more sessions than you may be aware of. We had a resource reach 15 once. (Not at all a pretty sight.) Please note that there is a known bug with this particular scheduled task trigger that may cause the script to kick off on RDP connection, rather than just disconnection. The attached file has some logic included to check for this. In case of failure, you should be able to RDP in as any other account and disable the task.


  • 5.  RE: If you absolutely have to…

    Posted 06-17-2019 15:32
      |   view attached
    Had a request earlier to re-attach the script. (Also, fun fact: Edit won't allow you to attach anything after the fact. :P )

    ------------------------------
    Ami Barrett
    Lead RPA Software Developer
    Solai & Cameron
    America/Chicago
    ------------------------------

    Attachment(s)

    txt
    unlockMe.txt   1K 1 version


  • 6.  RE: If you absolutely have to…

    Posted 04-25-2020 02:17
    Hi @Ami Barrett

    Will this .bat allow me to close RDP and have my processes still running? Sadly RDP is the only tool we have to connect to server and resource PCs. 

    ​I've configured 2 VMs, one as server and the other as client. I've scheduled 2 different process at different schedules. One uses Login Agent to login, and the other does 'Global Mouse Clic Center' actions on calculator for 2 minutes.

    If I connect using RDP to the resource PC while process is running, it will "pause" for a few seconds and then starts clicking buttons again. If I close the RDP session, control room will show the process as "running", but it will not clic on any button, like if it paused forever. 

    Any friendly suggestion? ​

    ------------------------------
    David Vizcaíno
    RPA Programmer
    EY
    America/Bogota
    ------------------------------



  • 7.  RE: If you absolutely have to…

    Posted 04-26-2020 17:46
    @David Vizcaíno

    That is the intent of the bat file, yes.

    Treat a VM like any desktop without an attached monitor. If you RDP into a physical box, the local system will display a lock screen, while the active session is being virtualized. Upon logging out, there is no longer any virtualization, thus the local system has no desktop to interact with. Rather than permenantly pause, this should cause a failure on anything requiring accessibility, UIA, or Region mode access. 

    Its also worth noting, that because it has no screen attached, the default resolution will be somewhere around 800x600. You can change this via powershell on disconnect, to 1200x800, which I think is the max resolution without an attaches screen. These resolution changes may have an effect on your automation. 

    The pause you're experiencing may have to do with it reconfiguring the session between the local and remote connections.

    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 8.  RE: If you absolutely have to…

    Posted 04-27-2020 17:59
    @Ami Barrett

    ​Hi, I've configured a new task to run a the .bat, triggering on RDP disconnection according to the following event IDs:

    • EventID – 24 (Remote Desktop Services: Session has been disconnected) – a user has disconnected from the RDP session;
    • EventID – 25 (Remote Desktop Services: Session reconnection succeeded) – a user has reconnected to the existing RDP session on the server;
    • EventID – 39 (Session <A> has been disconnected by session <B>) – a user has disconnected from RDP session by selecting the corresponding menu option (instead of just closing the RDP client window). If the session IDs are different, a user has been disconnected by another user (or an administrator);
    • EventID – 40 (Session <A> has been disconnected, reason code <B>). Here you must view the disconnection reason code in the event description. 
    • EventID 23 (Remote Desktop Services: Session logoff succeeded), refers to the user logoff from the system.
    I noticed the bug, as you said the task also triggers on login, so I implemented some logic into a powershell script to check for the session state and only run the .bat when session state is disconnected. When connecting with RDP: session is active, closing RDP: session is dissconected. I also managed to run the powershell script on hidden mode so no interference with the UI is shown. 

    Your .bat runs OK, only when I close the RDP sessions it brings back the console session. The only problem is that is shows the 20 cmd windows, one for each tscon line you mentioned on previous comments. Therefore my process fails to find the buttons on my application due to all the cmd windows opening and closing. 

    I noticed the "@ECHO off"​ on first line of the.bat, which will mean everything runs without windows displaying. Am I running an outdated version of the .bat, maybe? 

    I also tried changing the user account under which the task runs on the task scheduler to SYSTEM, or checking the "Run whether is logged on or not" checkbox. The cmd Windows don't show but the .bat doesn't bring back the session to console. 

    Do you have any updated version of the .bat? How did you get this to work?

    Thank you

    ------------------------------
    David Vizcaíno
    RPA Programmer
    EY
    America/Bogota
    ------------------------------



  • 9.  RE: If you absolutely have to…

    Posted 04-27-2020 18:03

    This solution is pretty amazing and stunning.

     

    I would like to understand to source of this problem (not able to see the history conversation).

     

    Do you have a control room to start the process to avoid to enter into the Runtime Resource? Or, do you any a special business requirement to do that?

     

    Just to know.

     

    Best regards.

     

     

     

     






  • 10.  RE: If you absolutely have to…

    Posted 04-27-2020 18:21
    @Raúl Hernández​ - This is specifically if there is no other method setup to remotely connect to the resource(s) than RDP. Ideally, you'd use something like VNC or TeamViewer.

    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 11.  RE: If you absolutely have to…

    Posted 04-27-2020 18:41

    Hi, Ami.

     

    I understand. So, there is no login agent working here.

     

    Best regards.

     

     

     

     






  • 12.  RE: If you absolutely have to…

    Posted 04-27-2020 18:51
    The RDP logon would be for manual intervention, not automatic login.

    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 13.  RE: If you absolutely have to…

    Posted 04-27-2020 21:44

    Ok, So, is this situation solved now?

     

    Best regards.

     

     

     

     






  • 14.  RE: If you absolutely have to…

    Posted 04-27-2020 18:19
    Edited by Ami Barrett 04-27-2020 20:29

    @David Vizcaíno - Below is the latest version I have, which includes a session check, as you found might be needed. The 20 windows are expected, unfortunately. In normal cases, you might be able to get away with only using the first one or two tsdiscon lines, though.

    The only way I know of to launch something hidden, would be via additional code. For example, VB (also below). You could make a .vbs file to launch the bat file, then tie that into your task. I haven't tested it on this particular bat file, so I don't know if the child windows would also be hidden. "@ECHO off" only specifies that it shouldn't output text to the cmd window.

    Have you checked that the task is running as administrator? It's required to use tsdiscon.

    Dim startInfo As New ProcessStartInfo("unlockme.bat")
    startInfo.WindowStyle = ProcessWindowStyle.Hidden
    
    Process.Start(startInfo)
    @ECHO off
    
    setlocal EnableDelayedExpansion
    
    
    
    for /f "tokens=2,4" %%a in ('QUERY USER ^| FINDSTR ">"') DO (
    
     set "consoleTemp=%%a"
    
     set "connectionState=%%b"
    
     set "consoleType=!consoleTemp:~0,7!"
    
    )
    
    
    
    SET consoleTypeCorrect=0
    
    IF "!consoleType!"=="rdp-tcp" SET consoleTypeCorrect=1
    
    
    
    SET connectionStateCorrect=1
    
    IF %connectionState%==Active SET connectionStateCorrect=1
    
    
    
    set TRUE=%connectionStateCorrect%%consoleTypeCorrect%
    
    
    
    IF %TRUE%==11 (
    
     endlocal
    
    ) else (
    
     start "" tscon 1 /dest:console
    
     start "" tscon 2 /dest:console
    
     start "" tscon 3 /dest:console
    
     start "" tscon 4 /dest:console
    
     start "" tscon 5 /dest:console
    
     start "" tscon 6 /dest:console
    
     start "" tscon 7 /dest:console
    
     start "" tscon 8 /dest:console
    
     start "" tscon 9 /dest:console
    
     start "" tscon 10 /dest:console
    
     start "" tscon 11 /dest:console
    
     start "" tscon 12 /dest:console
    
     start "" tscon 13 /dest:console
    
     start "" tscon 14 /dest:console
    
     start "" tscon 15 /dest:console
    
     start "" tscon 16 /dest:console
    
     start "" tscon 17 /dest:console
    
     start "" tscon 18 /dest:console
    
     start "" tscon 19 /dest:console
    
     start "" tscon 20 /dest:console
    
    )​


    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 15.  RE: If you absolutely have to…

    Posted 04-27-2020 21:58
    @Ami Barrett

    So by now, the .bat shows the windows, right? Does the .bat stop launching cmd windows depending on which tscon line got the correct connection? ​Could you please explain in a more technical way how the .bat actually does the trick? 

    I've tried your code example with VB with no luck, it throws error 800a0401. I tried with this and the windows still popup. 

    command = "powershell.exe -nologo -command unlockMe.bat"
    
    set shell = CreateObject("WScript.Shell")
    
    shell.Run command,0​

    Set oShell = CreateObject ("Wscript.Shell") 
    Dim strArgs
    strArgs = "cmd /c unlockMe.bat"
    oShell.Run strArgs, 0, false​

    CreateObject("Wscript.Shell").Run "unlockMe.bat", 0, True


    I'm running the task with elevated rights. 

    Question: When you tried this .bat, the windows flashing up didn't represent a problem??




    ------------------------------
    David Vizcaíno
    RPA Programmer
    EY
    America/Bogota
    ------------------------------



  • 16.  RE: If you absolutely have to…

    Posted 04-27-2020 22:45
    The script is meant as a shotgun-blast approach. Even if it hits the correct session number, it will still continue through. 20 was an extreme case that I've seen useful in exactly one deployment. Pretty much every other time only required the first one or two sessions before it had a successful flip.

    I was messing around with the VB solution, and was not able to get it to hide the child windows as-is. I found a way to make it hide the child windows in powershell, but it won't hide the main window. 

    This got me poking around it some more. When I had first built this bat file against some Server 2012r2 VMs, there was a technical need to spawn each tsdiscon in a new window. Seems something's changed along the way, so you can remove the "start """ from each line and contain it into a single window. So yay, no more child windows to mess with. I was then able to fix the VBS file to get the desired effect.

    PowerShell:
    PowerShell.exe -windowstyle hidden { "-command unlockMe.bat" }

    VB:

    1) Create file: unlockme.vbs (or whatever you want to call it), and modify the path to the bat file:

    CreateObject("Wscript.Shell").Run "c:\tools\unlockme.bat",0,False

    2) Modify unlockme.bat

    @ECHO off
    
    setlocal EnableDelayedExpansion
    
    for /f "tokens=2,4" %%a in ('QUERY USER ^| FINDSTR ">"') DO (
     set "consoleTemp=%%a"
     set "connectionState=%%b"
     set "consoleType=!consoleTemp:~0,7!"
    )
    
    SET consoleTypeCorrect=0
    
    IF "!consoleType!"=="rdp-tcp" SET consoleTypeCorrect=1
    SET connectionStateCorrect=1
    
    IF %connectionState%==Active SET connectionStateCorrect=1
    set TRUE=%connectionStateCorrect%%consoleTypeCorrect%
    
    IF %TRUE%==11 (
     endlocal
    ) else (
      tscon 1 /dest:console
      tscon 2 /dest:console
      tscon 3 /dest:console
      tscon 4 /dest:console
      tscon 5 /dest:console
      tscon 6 /dest:console
      tscon 7 /dest:console
      tscon 8 /dest:console
      tscon 9 /dest:console
      tscon 10 /dest:consol
      tscon 11 /dest:console
      tscon 12 /dest:console
      tscon 13 /dest:console
      tscon 14 /dest:console
      tscon 15 /dest:console
      tscon 16 /dest:console
      tscon 17 /dest:console
      tscon 18 /dest:console
      tscon 19 /dest:console
      tscon 20 /dest:console
    )


    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 17.  RE: If you absolutely have to…

    Posted 04-28-2020 19:47
    @Ami Barrett

    Thank you so much for your help. For some strange reason the session check you mentioned was not working on my test environment.  So I implemented some logic into a powershell script to check for the session state and only run the .bat when session state is disconnected (RDP session is closed). Then I modified the .bat as you mentioned on the previous comment, created the VBS linked to the .bat, and ran the VBS accordingly from the powershell script.

    Things are working now, I've done two tests: 

    1) Use RDP to login into resource PC, run a process from control room, close RDP session. Process continued running on console and everything succeeded (as long as I did not interfere with Blue Prism pressing buttons while closing the RDP window) 

    2) Use scheduler to Login into a resource PC using Login Agent and run a process, RDP to the resource PC while the process is running (process will seem to pause for a few seconds and then starts working again), closed the RDP session to let the process finish. 

    Neither the powershell script I created for session check, nor the cmd windows popped up.

    Thank you again.


    ------------------------------
    David Vizcaíno
    RPA Programmer
    EY
    America/Bogota
    ------------------------------



  • 18.  RE: If you absolutely have to…

    Posted 04-28-2020 19:50
    Awesome! Glad you got it working!

    ------------------------------
    Ami Barrett
    Sr Product Consultant
    Blue Prism
    Plano, TX
    ------------------------------



  • 19.  Thanks @John & @AmiBarrett…

    Posted 04-09-2019 15:36
    Thanks @John & @AmiBarrett for your input. Issue was resolved yesterday afternoon. It was a perfect storm of several small human errors which were not caught in time.  - Two environments were not in sync when a developer edited a VBO in our QA environment  (all edits should have been done in Dev) (mistake #1) - The edit broke the object - The QA object was imported back to our Dev environment which is against our protocols (mistake #2) - A package was created which included the overwritten object which was then imported into PROD, it should not have included the object (mistake #3) - The object is used in a few custom VBOs including a modified LoginAgent. - Package was imported into Production on Friday - The modified VBO causes LoginAgent to essentially do nothing and always report a success. - No changes were made at a process level so it took us longer to diagnose the issue (i ll call this mistake #4, i should noticed this sooner) It was a face palm moment when we figured it out. @Amibarett When I said ""the machine logs out"", I meant that when we remoted into the resource, it reported to the control that it was available to run processes so long as the remote desktop session is dispalyed on a monitor. So everything works normally from then.  But when we ran a 'LogOut' or a 'MachineRestart' process.  We could still not use the LoginAgent's 'Login' function. The thing that was throwing me off was that when we ran the Login, the it reported successful completion and the Log showed all steps completed successfully. But once we synced up our environments with the correct version everything returned to normal.