Scheduler error: Resource XXXX:8001 is offline. But it's not in control room.

[RuneHedegaard]

Hello We got som problems with Scheduler. For some reason it random get stuck after login, and says "Resource UDCBLUERUN7:8001 is offline - retried 3 times" We created a simple process to check resolution from a object utility to test the problem. Below the logs from Recent Activity from control room. I have made some comments to the logs below.   Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:10:00Z 2018-05-02 13:09:59Z 2018-05-02 13:11:12Z  Task Completed Login DIRXLYA  2018-05-02 13:09:59Z 2018-05-02 13:10:04Z  Session Completed abbd1ea2-8fad-4102-8601-a34179c7aa7a  2018-05-02 13:10:01Z 2018-05-02 13:10:04Z  Task Completed Bent - Check resolution  2018-05-02 13:10:04Z 2018-05-02 13:11:09Z  Session Completed 53b7b66c-43c7-4939-b167-91e3a560fe4e  2018-05-02 13:11:05Z 2018-05-02 13:11:09Z  Task Completed Logout  2018-05-02 13:11:09Z 2018-05-02 13:11:12Z  Session Completed 1748a191-0f43-408b-bbe4-167188bce5d6  2018-05-02 13:11:10Z 2018-05-02 13:11:12Z         Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:15:00Z 2018-05-02 13:14:59Z 2018-05-02 13:21:04Z  Task Completed Login DIRXLYA  2018-05-02 13:14:59Z 2018-05-02 13:15:03Z  Session Completed 96a2d462-2146-459e-9890-11ea5bed3c18  2018-05-02 13:15:00Z 2018-05-02 13:15:03Z  Task Terminated Bent - Check resolution  2018-05-02 13:15:03Z 2018-05-02 13:18:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:18:03Z 2018-05-02 13:21:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times               Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:25:00Z 2018-05-02 13:24:59Z 2018-05-02 13:31:04Z  Task Completed Login DIRXLYA  2018-05-02 13:24:59Z 2018-05-02 13:25:04Z  Session Completed f236b260-0e8d-4439-bf13-9529be7ed871  2018-05-02 13:25:00Z 2018-05-02 13:25:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:25:04Z 2018-05-02 13:28:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:28:04Z 2018-05-02 13:31:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:35:00Z 2018-05-02 13:34:59Z 2018-05-02 13:41:04Z  Task Completed Login DIRXLYA  2018-05-02 13:34:59Z 2018-05-02 13:35:04Z  Session Completed 47410358-44a3-4a49-b0bf-604eec5e23c2  2018-05-02 13:35:01Z 2018-05-02 13:35:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:35:04Z 2018-05-02 13:38:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:38:04Z 2018-05-02 13:41:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times        Comment: We did manuelle run logout (by drag and drop logout process to the resource UDCBLUERUN7:8001 in the control room and waited for det next schedule. Se below       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:45:00Z 2018-05-02 13:45:00Z 2018-05-02 13:46:10Z  Task Completed Login DIRXLYA  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Session Completed a0cf5ed3-6d23-4d57-846b-4137163106f3  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Task Completed Bent - Check resolution  2018-05-02 13:45:03Z 2018-05-02 13:46:06Z  Session Completed 703fad23-8dd6-4544-9349-33fbc89aa5db  2018-05-02 13:46:04Z 2018-05-02 13:46:06Z  Task Completed Logout  2018-05-02 13:46:06Z 2018-05-02 13:46:10Z  Session Completed c1aa4568-edde-4746-bbb1-a4632e3ff669  2018-05-02 13:46:07Z 2018-05-02 13:46:10Z         Comment: The first run did work fine. But the 2 schedule failed again. But for some reason the logout did not fail. (se below)       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:50:00Z 2018-05-02 13:50:00Z 2018-05-02 13:57:09Z  Task Completed Login DIRXLYA  2018-05-02 13:50:00Z 2018-05-02 13:50:04Z  Session Completed ce0a34fd-0831-4674-a58e-f472f2e16a03  2018-05-02 13:50:01Z 2018-05-02 13:50:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:50:04Z 2018-05-02 13:53:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 13:53:04Z 2018-05-02 13:57:09Z  Session Completed 2be860d5-6972-43ea-ab84-05adb08cd0c1  2018-05-02 13:57:05Z 2018-05-02 13:57:09Z                Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:00:00Z 2018-05-02 14:00:00Z 2018-05-02 14:01:13Z  Task Completed Login DIRXLYA  2018-05-02 14:00:00Z 2018-05-02 14:00:04Z  Session Completed 20399468-456b-487c-8207-8965940fb63c  2018-05-02 14:00:01Z 2018-05-02 14:00:04Z  Task Completed Bent - Check resolution  2018-05-02 14:00:04Z 2018-05-02 14:01:09Z  Session Completed 822b824b-66ef-4be4-897e-19ec803584a4  2018-05-02 14:01:05Z 2018-05-02 14:01:09Z  Task Completed Logout  2018-05-02 14:01:09Z 2018-05-02 14:01:13Z  Session Completed ddfd9075-9ab3-4f10-a91b-79c3844aefbb  2018-05-02 14:01:10Z 2018-05-02 14:01:13Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:05:00Z 2018-05-02 14:05:00Z 2018-05-02 14:09:07Z  Task Completed Login DIRXLYA  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Session Completed aeb58cff-f34b-4b59-8dcb-51d2a9752932  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:05:03Z 2018-05-02 14:08:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:08:04Z 2018-05-02 14:09:07Z  Session Completed 46a152fb-3888-4b8f-8e37-90290e5c70e3  2018-05-02 14:09:04Z 2018-05-02 14:09:07Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:10:00Z 2018-05-02 14:09:59Z 2018-05-02 14:14:06Z  Task Completed Login DIRXLYA  2018-05-02 14:09:59Z 2018-05-02 14:10:04Z  Session Completed efd20bd2-1296-41bd-b3c5-1fee4f60f104  2018-05-02 14:10:00Z 2018-05-02 14:10:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:10:04Z 2018-05-02 14:13:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:13:04Z 2018-05-02 14:14:06Z  Session Completed 792886f4-28ee-46f7-b1d6-fbf8e108c3ce  2018-05-02 14:14:04Z 2018-05-02 14:14:06Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:15:00Z 2018-05-02 14:14:59Z 2018-05-02 14:19:08Z  Task Completed Login DIRXLYA  2018-05-02 14:14:59Z 2018-05-02 14:15:04Z  Session Completed 842e1616-658e-4fb6-baf5-84b22104734a  2018-05-02 14:15:01Z 2018-05-02 14:15:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:15:04Z 2018-05-02 14:18:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:18:04Z 2018-05-02 14:19:08Z  Session Completed 721e4968-33a1-46a6-b283-c9e69d791c5b  2018-05-02 14:19:05Z 2018-05-02 14:19:08Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:20:00Z 2018-05-02 14:19:59Z 2018-05-02 14:27:07Z  Task Completed Login DIRXLYA  2018-05-02 14:19:59Z 2018-05-02 14:20:03Z  Session Completed 51571d7c-6c0b-4969-9ffe-d4e86cd6244f  2018-05-02 14:20:00Z 2018-05-02 14:20:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:20:03Z 2018-05-02 14:23:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:23:03Z 2018-05-02 14:27:07Z  Session Completed c57c7f86-6089-402d-8f10-a5ea353850d6  2018-05-02 14:27:04Z 2018-05-02 14:27:07Z         Comment: For som reason the logout task also failed after some tryes. Se below       Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:30:00Z 2018-05-02 14:29:59Z 2018-05-02 14:38:03Z  Task Completed Login DIRXLYA  2018-05-02 14:29:59Z 2018-05-02 14:30:03Z  Session Completed 99e761e6-d1d0-45b6-b668-9210f7840c5c  2018-05-02 14:30:00Z 2018-05-02 14:30:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:30:03Z 2018-05-02 14:33:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 14:33:03Z 2018-05-02 14:38:03Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Comment: Now the login also fails, because of the logout problem. Se blow Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:40:00Z2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task Terminated Login DIRXLYA2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:45:00Z2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task Terminated Login DIRXLYA2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times

[JereVekka]

Has Blue Prism provide any solution to anybody? New Login Agent has not solved the problem and error ""Session creation failed on resource 'RESOURCE': Failed to create session on Resource - No reply from Resource PC"" still occurs multiple times per hour.

[JereVekka]

We have been able to reproduce this error with multiple combinations including different OS (2012R2, 2016, 2019), Blue Prism versions (5, 6.2.1, 6.3), connection types (WCF, .NET), platforms (Hyper-V, Azure, Vmware) and different ports for login agent. Our investigation appoints that the problem exists with Blue Prism scheduler as it cant handle the load which can be reproduced with under 10 runtime resources.

[JosepArques]

Yesterday we upgraded our pre-production environment to 6.3 version and for the first time we have faced this error. The runtime resource is configured to start a runtime process automatically when a windows user logs in (to run processes by connecting with login agent). But, as a PRE machine, developpers are allowed as well to log in and start Control Room to develop and schedule processes. I guess that the problem is related when scheduling a process and there's a Control Room session running in the same machine.

[JereVekka]

Just to update for you all. We have been working with Blue Prism to resolve this problem as it has been identified. The problem exists (usually) when you have more than 5 runtime resources which are doing login-do the tasks-logout multiple times a day. If you are doing login on the morning and logout on the evening, you probably don't have this problem. It would be nice if Blue Prism itself would update this thread, but we, in Digital Workforce, hope that the fix will be released in this month as we are already testing versions which includes it.

[RuneHedegaard]

Thanks for the update Jvekka We only got one running runtime resource. Its correct that logout in the evening and in the morning is working, but we need to do it manually from the control room.  

[WitoldKochman]

Hello all. We are working on version 6.3 and facing same issue with the scheduler. Thanks to jvekka for giving some tips that could help. I have just removed logouts after each task, so now all the logouts will be done at the end of day. Hope it helps. In a meantime - did anybody check if the latest release (6.3.1) fixes the problem? This is really annoying that it takes so long for BP to resolve such critical issue... 

[RuneHedegaard]

Hello W.Kockman   Version 6.3.1 did not fix the problem for us.

[AmiBarrett]

Out of curiosity, have you tried connecting the resources to the DB instead of the controller? Despite what BP Support has told us on the phone, we're getting full functionality out of this solution and have had no issues with login agent-controlled VMs. To clarify, BP Support has told us that this solution will not support scheduling or encryption. We use both, going directly to the SQL server. I have also configured an environment going through the controller, and have noticed no difference in functionality. (Just that the controller pass-through is slower.)

[ApurvaVerma]

We are also having the same problem where we are able to run the process through Control Room, but same process won't run through the scheduler. It will fail after 'x' number of retries. Please let me know if someone has a viable solution, we are on 6.3.1.

[JosepArques]

Has anybody tried with 6.3.2 or 6.4.0 versions? We need this problem to be fixed as soon as possible.