Scheduler error: Resource XXXX:8001 is offline. But it's not in control room.

[RuneHedegaard]

Hello We got som problems with Scheduler. For some reason it random get stuck after login, and says "Resource UDCBLUERUN7:8001 is offline - retried 3 times" We created a simple process to check resolution from a object utility to test the problem. Below the logs from Recent Activity from control room. I have made some comments to the logs below.   Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:10:00Z 2018-05-02 13:09:59Z 2018-05-02 13:11:12Z  Task Completed Login DIRXLYA  2018-05-02 13:09:59Z 2018-05-02 13:10:04Z  Session Completed abbd1ea2-8fad-4102-8601-a34179c7aa7a  2018-05-02 13:10:01Z 2018-05-02 13:10:04Z  Task Completed Bent - Check resolution  2018-05-02 13:10:04Z 2018-05-02 13:11:09Z  Session Completed 53b7b66c-43c7-4939-b167-91e3a560fe4e  2018-05-02 13:11:05Z 2018-05-02 13:11:09Z  Task Completed Logout  2018-05-02 13:11:09Z 2018-05-02 13:11:12Z  Session Completed 1748a191-0f43-408b-bbe4-167188bce5d6  2018-05-02 13:11:10Z 2018-05-02 13:11:12Z         Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:15:00Z 2018-05-02 13:14:59Z 2018-05-02 13:21:04Z  Task Completed Login DIRXLYA  2018-05-02 13:14:59Z 2018-05-02 13:15:03Z  Session Completed 96a2d462-2146-459e-9890-11ea5bed3c18  2018-05-02 13:15:00Z 2018-05-02 13:15:03Z  Task Terminated Bent - Check resolution  2018-05-02 13:15:03Z 2018-05-02 13:18:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:18:03Z 2018-05-02 13:21:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times               Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:25:00Z 2018-05-02 13:24:59Z 2018-05-02 13:31:04Z  Task Completed Login DIRXLYA  2018-05-02 13:24:59Z 2018-05-02 13:25:04Z  Session Completed f236b260-0e8d-4439-bf13-9529be7ed871  2018-05-02 13:25:00Z 2018-05-02 13:25:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:25:04Z 2018-05-02 13:28:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:28:04Z 2018-05-02 13:31:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:35:00Z 2018-05-02 13:34:59Z 2018-05-02 13:41:04Z  Task Completed Login DIRXLYA  2018-05-02 13:34:59Z 2018-05-02 13:35:04Z  Session Completed 47410358-44a3-4a49-b0bf-604eec5e23c2  2018-05-02 13:35:01Z 2018-05-02 13:35:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:35:04Z 2018-05-02 13:38:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 13:38:04Z 2018-05-02 13:41:04Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times        Comment: We did manuelle run logout (by drag and drop logout process to the resource UDCBLUERUN7:8001 in the control room and waited for det next schedule. Se below       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:45:00Z 2018-05-02 13:45:00Z 2018-05-02 13:46:10Z  Task Completed Login DIRXLYA  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Session Completed a0cf5ed3-6d23-4d57-846b-4137163106f3  2018-05-02 13:45:00Z 2018-05-02 13:45:03Z  Task Completed Bent - Check resolution  2018-05-02 13:45:03Z 2018-05-02 13:46:06Z  Session Completed 703fad23-8dd6-4544-9349-33fbc89aa5db  2018-05-02 13:46:04Z 2018-05-02 13:46:06Z  Task Completed Logout  2018-05-02 13:46:06Z 2018-05-02 13:46:10Z  Session Completed c1aa4568-edde-4746-bbb1-a4632e3ff669  2018-05-02 13:46:07Z 2018-05-02 13:46:10Z         Comment: The first run did work fine. But the 2 schedule failed again. But for some reason the logout did not fail. (se below)       Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 15:50:00Z 2018-05-02 13:50:00Z 2018-05-02 13:57:09Z  Task Completed Login DIRXLYA  2018-05-02 13:50:00Z 2018-05-02 13:50:04Z  Session Completed ce0a34fd-0831-4674-a58e-f472f2e16a03  2018-05-02 13:50:01Z 2018-05-02 13:50:04Z  Task Terminated Bent - Check resolution  2018-05-02 13:50:04Z 2018-05-02 13:53:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 13:53:04Z 2018-05-02 13:57:09Z  Session Completed 2be860d5-6972-43ea-ab84-05adb08cd0c1  2018-05-02 13:57:05Z 2018-05-02 13:57:09Z                Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:00:00Z 2018-05-02 14:00:00Z 2018-05-02 14:01:13Z  Task Completed Login DIRXLYA  2018-05-02 14:00:00Z 2018-05-02 14:00:04Z  Session Completed 20399468-456b-487c-8207-8965940fb63c  2018-05-02 14:00:01Z 2018-05-02 14:00:04Z  Task Completed Bent - Check resolution  2018-05-02 14:00:04Z 2018-05-02 14:01:09Z  Session Completed 822b824b-66ef-4be4-897e-19ec803584a4  2018-05-02 14:01:05Z 2018-05-02 14:01:09Z  Task Completed Logout  2018-05-02 14:01:09Z 2018-05-02 14:01:13Z  Session Completed ddfd9075-9ab3-4f10-a91b-79c3844aefbb  2018-05-02 14:01:10Z 2018-05-02 14:01:13Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:05:00Z 2018-05-02 14:05:00Z 2018-05-02 14:09:07Z  Task Completed Login DIRXLYA  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Session Completed aeb58cff-f34b-4b59-8dcb-51d2a9752932  2018-05-02 14:05:00Z 2018-05-02 14:05:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:05:03Z 2018-05-02 14:08:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:08:04Z 2018-05-02 14:09:07Z  Session Completed 46a152fb-3888-4b8f-8e37-90290e5c70e3  2018-05-02 14:09:04Z 2018-05-02 14:09:07Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:10:00Z 2018-05-02 14:09:59Z 2018-05-02 14:14:06Z  Task Completed Login DIRXLYA  2018-05-02 14:09:59Z 2018-05-02 14:10:04Z  Session Completed efd20bd2-1296-41bd-b3c5-1fee4f60f104  2018-05-02 14:10:00Z 2018-05-02 14:10:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:10:04Z 2018-05-02 14:13:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:13:04Z 2018-05-02 14:14:06Z  Session Completed 792886f4-28ee-46f7-b1d6-fbf8e108c3ce  2018-05-02 14:14:04Z 2018-05-02 14:14:06Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:15:00Z 2018-05-02 14:14:59Z 2018-05-02 14:19:08Z  Task Completed Login DIRXLYA  2018-05-02 14:14:59Z 2018-05-02 14:15:04Z  Session Completed 842e1616-658e-4fb6-baf5-84b22104734a  2018-05-02 14:15:01Z 2018-05-02 14:15:04Z  Task Terminated Bent - Check resolution  2018-05-02 14:15:04Z 2018-05-02 14:18:04Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:18:04Z 2018-05-02 14:19:08Z  Session Completed 721e4968-33a1-46a6-b283-c9e69d791c5b  2018-05-02 14:19:05Z 2018-05-02 14:19:08Z         Schedule Completed Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:20:00Z 2018-05-02 14:19:59Z 2018-05-02 14:27:07Z  Task Completed Login DIRXLYA  2018-05-02 14:19:59Z 2018-05-02 14:20:03Z  Session Completed 51571d7c-6c0b-4969-9ffe-d4e86cd6244f  2018-05-02 14:20:00Z 2018-05-02 14:20:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:20:03Z 2018-05-02 14:23:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Completed Logout  2018-05-02 14:23:03Z 2018-05-02 14:27:07Z  Session Completed c57c7f86-6089-402d-8f10-a5ea353850d6  2018-05-02 14:27:04Z 2018-05-02 14:27:07Z         Comment: For som reason the logout task also failed after some tryes. Se below       Schedule Terminated Z Test  - DIRXLYA (Citrix 7) 2018-05-02 16:30:00Z 2018-05-02 14:29:59Z 2018-05-02 14:38:03Z  Task Completed Login DIRXLYA  2018-05-02 14:29:59Z 2018-05-02 14:30:03Z  Session Completed 99e761e6-d1d0-45b6-b668-9210f7840c5c  2018-05-02 14:30:00Z 2018-05-02 14:30:03Z  Task Terminated Bent - Check resolution  2018-05-02 14:30:03Z 2018-05-02 14:33:03Z Task: Bent - Check resolution threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Task Terminated Logout  2018-05-02 14:33:03Z 2018-05-02 14:38:03Z Task: Logout threw an exception: Resource UDCBLUERUN7:8001 is offline - retried 3 times Comment: Now the login also fails, because of the logout problem. Se blow Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:40:00Z2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task Terminated Login DIRXLYA2018-05-02 14:39:59Z2018-05-02 14:43:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times Schedule Terminated Z Test  - DIRXLYA (Citrix 7)2018-05-02 16:45:00Z2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task Terminated Login DIRXLYA2018-05-02 14:44:59Z2018-05-02 14:48:00Z Task: Login DIRXLYA threw an exception: Resource UDCBLUERUN7 is offline - retried 3 times

[JuhoLindblad]

Hi Rune, We are facing the exactly same issue with the scheduler. This occurs randomly maybe with 10-20% of the scheduled runs. When the tasks are triggered by drag-and-dropping in Control Room, this never occurs. Also after the login agent, we can see that the resource is in ""Idle"" status and that the Resource PC is in mode ""Listener started."". Which communication method you are using? We moved now from .Net Remoting to WFC. It seems to be more stable with WFC, but the change was just done, so it's too early to say yet. Please, let me know, if you find any solution for the issue.  

[JuhoLindblad]

Sorry, WCF not WFC..

[RuneHedegaard]

Hi Juho We are also using WCF. The resource is also in idlemode with us.

[JuhoLindblad]

Hi Rune, Maybe we could compare our environment setups and see if we can find similarities, which could cause this issue? If you would like to have a call, please send me an email to juho.lindblad@dpdhl.com.

[stuartfraser]

We are having very similar issues, scheduals seem to work fine for a while after run resources and app server have all been rebooted, then start randomly failing anything from a few days to a week or more after. Control room shows them connected and Telnet from app server confirms listening with LoginAgent or resource as appropriate but schedular gives the offline error. From what I have read the scheduler runs as the BPServer service and it is that process (running under a service account in our case) that can't see the resources as being online (where as control room running as an admin user can), but we can't work out why! Restarting BPserver service normally fixes it, but not feasible workaround on a prod platform! we are running 6.1 and already using WCF connection type.

[RuneHedegaard]

Hello again. Thanks for your responds. We will try to update to version 6.2 to see if it fix the problem. We run 6.0 atm

[JuhoLindblad]

Hi Rune, We are already running 6.2 and having the issue. Ver 6.2.1 is out, but I couldn't see in the Release notes anything which would point into solving this issue. I have opened ticket to BP support, but I haven't received reply to that yet. Still, if you would be willing to compare the environment setups, feel free to drop me an email.

[RuneHedegaard]

Hi Juho Any news? Did you hear from BP Support?

[JuhoLindblad]

Hi Rune, Got only suggestion to update into v6.2.1. Did that yesterday, but it didn't solve the issue. Now waiting for their response.