Scheduler error: Resource XXXX:8001 is offline. But it's not in control room.

[RuneHedegaard]

Hello We got som problems with Scheduler. For some reason it random get stuck after login, and says "Resource UDCBLUERUN7:8001 is offline - retried 3 times" We created a simple process to check resolution from a object utility to test the problem. Below the logs from Recent Activity from control room. I have made some comments to the logs below.   Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:10:00Z 2018-05-02 13:09:59Z 2018-05-02 13:11:12Z  Task Completed Login DIRXLYA  2018-05-02 13:09:59Z 2018-05-02 13:10:04Z  Session Completed abbd1ea2-8fad-4102-8601-a34179c7aa7a  2018-05-02 13:10:01Z 2018-05-02 13:10:04Z  Task Completed Bent - Check resolution  2018-05-02 13:10:04Z 2018-05-02 13:11:09Z  Session Completed 53b7b66c-43c7-4939-b167-91e3a560fe4e  2018-05-02 13:11:05Z 2018-05-02 13:11:09Z  Task Completed Logout  2018-05-02 13:11:09Z 2018-05-02 13:11:12Z  Session Completed 1748a191-0f43-408b-bbe4-167188bce5d6  2018-05-02 13:11:10Z 2018-05-02 13:11:12Z         Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:15:00Z 2018-05-02 13:14:59Z 2018-05-02 13:21:04Z  Task Completed Login DIRXLYA  2018-05-02 13:14:59Z 2018-05-02 13:15:03Z  Session Completed 96a2d462-2146-459e-9890-11ea5bed3c18  2018-05-02 13:15:00Z 2018-05-02 13:15:03Z  Task Terminated Bent - Check resolution  2018-05-02 13:15:03Z 2018-05-02 13:18:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:18:03Z 2018-05-02 13:21:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times               Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:25:00Z 2018-05-02 13:24:59Z 2018-05-02 13:31:04Z  Task Completed Login DIRXLYA  2018-05-02 13:24:59Z 2018-05-02 13:25:04Z  Session Completed f236b260-0e8d-4439-bf13-9529be7ed871  2018-05-02 13:25:00Z 2018-05-02 13:25:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:25:04Z 2018-05-02 13:28:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:28:04Z 2018-05-02 13:31:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:35:00Z 2018-05-02 13:34:59Z 2018-05-02 13:41:04Z  Task Completed Login DIRXLYA  2018-05-02 13:34:59Z 2018-05-02 13:35:04Z  Session Completed 47410358-44a3-4a49-b0bf-604eec5e23c2  2018-05-02 13:35:01Z 2018-05-02 13:35:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:35:04Z 2018-05-02 13:38:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:38:04Z 2018-05-02 13:41:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times        Comment: We did manuelle run logout (by drag and drop logout process to the resource UDCBLUERUN7:8001 in the control room and waited for det next schedule. Se below       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:45:00Z 2018-05-02 13:45:00Z 2018-05-02 13:46:10Z  Task Completed Login DIRXLYA  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Session Completed a0cf5ed3-6d23-4d57-846b-4137163106f3  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Task Completed Bent - Check resolution  2018-05-02 13:45:03Z 2018-05-02 13:46:06Z  Session Completed 703fad23-8dd6-4544-9349-33fbc89aa5db  2018-05-02 13:46:04Z 2018-05-02 13:46:06Z  Task Completed Logout  2018-05-02 13:46:06Z 2018-05-02 13:46:10Z  Session Completed c1aa4568-edde-4746-bbb1-a4632e3ff669  2018-05-02 13:46:07Z 2018-05-02 13:46:10Z         Comment: The first run did work fine. But the 2 schedule failed again. But for some reason the logout did not fail. (se below)       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:50:00Z 2018-05-02 13:50:00Z 2018-05-02 13:57:09Z  Task Completed Login DIRXLYA  2018-05-02 13:50:00Z 2018-05-02 13:50:04Z  Session Completed ce0a34fd-0831-4674-a58e-f472f2e16a03  2018-05-02 13:50:01Z 2018-05-02 13:50:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:50:04Z 2018-05-02 13:53:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 13:53:04Z 2018-05-02 13:57:09Z  Session Completed 2be860d5-6972-43ea-ab84-05adb08cd0c1  2018-05-02 13:57:05Z 2018-05-02 13:57:09Z                Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:00:00Z 2018-05-02 14:00:00Z 2018-05-02 14:01:13Z  Task Completed Login DIRXLYA  2018-05-02 14:00:00Z 2018-05-02 14:00:04Z  Session Completed 20399468-456b-487c-8207-8965940fb63c  2018-05-02 14:00:01Z 2018-05-02 14:00:04Z  Task Completed Bent - Check resolution  2018-05-02 14:00:04Z 2018-05-02 14:01:09Z  Session Completed 822b824b-66ef-4be4-897e-19ec803584a4  2018-05-02 14:01:05Z 2018-05-02 14:01:09Z  Task Completed Logout  2018-05-02 14:01:09Z 2018-05-02 14:01:13Z  Session Completed ddfd9075-9ab3-4f10-a91b-79c3844aefbb  2018-05-02 14:01:10Z 2018-05-02 14:01:13Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:05:00Z 2018-05-02 14:05:00Z 2018-05-02 14:09:07Z  Task Completed Login DIRXLYA  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Session Completed aeb58cff-f34b-4b59-8dcb-51d2a9752932  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:05:03Z 2018-05-02 14:08:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:08:04Z 2018-05-02 14:09:07Z  Session Completed 46a152fb-3888-4b8f-8e37-90290e5c70e3  2018-05-02 14:09:04Z 2018-05-02 14:09:07Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:10:00Z 2018-05-02 14:09:59Z 2018-05-02 14:14:06Z  Task Completed Login DIRXLYA  2018-05-02 14:09:59Z 2018-05-02 14:10:04Z  Session Completed efd20bd2-1296-41bd-b3c5-1fee4f60f104  2018-05-02 14:10:00Z 2018-05-02 14:10:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:10:04Z 2018-05-02 14:13:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:13:04Z 2018-05-02 14:14:06Z  Session Completed 792886f4-28ee-46f7-b1d6-fbf8e108c3ce  2018-05-02 14:14:04Z 2018-05-02 14:14:06Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:15:00Z 2018-05-02 14:14:59Z 2018-05-02 14:19:08Z  Task Completed Login DIRXLYA  2018-05-02 14:14:59Z 2018-05-02 14:15:04Z  Session Completed 842e1616-658e-4fb6-baf5-84b22104734a  2018-05-02 14:15:01Z 2018-05-02 14:15:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:15:04Z 2018-05-02 14:18:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:18:04Z 2018-05-02 14:19:08Z  Session Completed 721e4968-33a1-46a6-b283-c9e69d791c5b  2018-05-02 14:19:05Z 2018-05-02 14:19:08Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:20:00Z 2018-05-02 14:19:59Z 2018-05-02 14:27:07Z  Task Completed Login DIRXLYA  2018-05-02 14:19:59Z 2018-05-02 14:20:03Z  Session Completed 51571d7c-6c0b-4969-9ffe-d4e86cd6244f  2018-05-02 14:20:00Z 2018-05-02 14:20:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:20:03Z 2018-05-02 14:23:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:23:03Z 2018-05-02 14:27:07Z  Session Completed c57c7f86-6089-402d-8f10-a5ea353850d6  2018-05-02 14:27:04Z 2018-05-02 14:27:07Z         Comment: For som reason the logout task also failed after some tryes. Se below       Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:30:00Z 2018-05-02 14:29:59Z 2018-05-02 14:38:03Z  Task Completed Login DIRXLYA  2018-05-02 14:29:59Z 2018-05-02 14:30:03Z  Session Completed 99e761e6-d1d0-45b6-b668-9210f7840c5c  2018-05-02 14:30:00Z 2018-05-02 14:30:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:30:03Z 2018-05-02 14:33:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 14:33:03Z 2018-05-02 14:38:03Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Comment: Now the login also fails, because of the logout problem. Se blow Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:40:00Z2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task Terminated Login DIRXLYA2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:45:00Z2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task Terminated Login DIRXLYA2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times

[PetriKaukua]

Hi Experts, This is what i am getting, termination reason -> ""Resouce XXX was unavailable : The state given as : InUse"" Note : There are no pending processes , doing it on the local machine Please  help, thanks in advanced.""   This means someone is logged in to the target ResourcePC and is using Blue Prism client. Br, Petri Kaukua

[JereVekka]

Hi you all, We have been sending now logs to Blue Prism for couple months and having multiple online meetings about this issue. And now they said that they may have identified the issue with runtime resources running in virtual environments*. The issue is with login agent doing it stuff too fast and for some reason in wrong order. We have now new speculative fix which is compiled againts login agent 6.2.1 in testing, but we can't give any results yet as it has been running only short period of time. If the problem does not re-occur with new login agent, the fix is coming only to Blue Prism 6.3 so we have to wait some time to get it to production. * Is anyone really running Blue Prism runtime resources on laptop/workstations instead of virtual machines?

[Mustafa_UlasYig]

Hi We were getting the same error in the schedule log viewer. Furthermore we were also getting the error ""The action was disconnected because of timeout"" in session log viewer when executing jobs directly from session management. So we were not able to execute from session management nor scheduler. The issue turned out to be the ""named pipes"" protocol on our SQL server that had been disabled. Enabling this protocol solved these issues. Even though this protocol had been disabled for quite some time, we were able to execute certain scheduled jobs, but as soon as we stopped the schedule for those jobs, we could not restart the scheduling for those same jobs and got the same errors mentioned in this thread. So just an idea, it to check whether or not the ""named pipes"" protocol on your SQL server has been disabled. BR, Mustafa

[Mustafa_UlasYig]

*correction Even though this protocol had been disabled for quite some time, we were able to execute certain scheduled jobs - that had been scheduled prior to the protocol being disabled - but as soon as we stopped the schedule for those jobs, we could not restart the scheduling for those same jobs and got the same errors mentioned in this thread. BR, Mustafa

[AndrewWatts]

Having the Same issue here.   Last night Scheduler successfully logged out the resource PC. However the Scheduler failed to start the Login Agent process, giving the error mentioned by the original poster. Is it possibly anything to do with a long period of inactivity between logout then login next morning that's causing it. has the application server gone to sleep or the resource itself? we have installations on multiple sights and this morning it happened on both. Some help please   Regards Andy  

[JoelPaajanen1]

@jvekka -> any results after that speculative fix? We have also the same issue here. Blue Prism version: 6.2.1.4272 Connection type: WCF BR, Joel

[PetriKaukua]

@paajajo So far what we have tested the speculative fix does not help. Br, Pete

[Anonymous]

Has anyone checked to see it the runtime resources are using DHCP or Static IP addresses. I have noticed that DCHP address can cause a issue if DNS does not get updated correctly. We had this very issue and the fix was using static IP addresses for our Runtime Resources. 

[PetriKaukua]

We are using stating IPs so this is not the cause.   -Pete

[PetriKaukua]

We have done extensive investigation on tcp packet level and on .NET framework level (with Wireshark, PaloAlto tools, SteelCentral AppInternals) and only thing we could conclude with a 3rd party that something happens in BP code that stops the scheduler working as intended. -Pete