January 2021 Windows Security Patches - Breaking Single Sign-on to Blue Prism?

[BrentonWestwood]

Hello all.   On any desktop where the January 2022 Windows Security Patches have been installed, I can no longer sign into Blue Prism.   On desktops where the security patch is not installed, everything works as normal.   The patches were installed on 2 of my desktops last night but others have not been patched yet.   We use single sign-on (AD setup).   I get this error:   

Error: Could not connect to '{connection name}'.

SOAP security negotiation with 'http://{appserver}.southernco.com:8187/bpserver' for target 'http:/{appserver}.southernco.com:8187/bpserver' failed. See inner exception for more details.

System.ComponentModel.Win32Exception: Either the client credential was invalid or there was an error collecting the client credentials by the SSPI.
at System.ServiceModel.Security.WindowsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetOutgoingBlobProxy.GetOutgoingBlob(ChannelBinding channelBinding)
at System.ServiceModel.Security.RequestSecurityToken.GetBinaryNegotiation()
at System.ServiceModel.Security.WSTrust.Driver.WriteRequestSecurityToken(RequestSecurityToken rst, XmlWriter xmlWriter)
at System.ServiceModel.Security.RequestSecurityToken.OnWriteTo(XmlWriter writer)
at System.ServiceModel.Security.RequestSecurityToken.WriteTo(XmlWriter writer)
at System.ServiceModel.Security.RequestSecurityToken.OnWriteBodyContents(XmlDictionaryWriter writer)
at System.ServiceModel.Channels.BodyWriterMessage.OnWriteBodyContents(XmlDictionaryWriter writer)

------------------------------
Brenton Westwood
Systems Analyst
Southern Company
------------------------------

[EmmaBurns]

We have experienced the same issue here as well. Have had to roll back the update, which is not ideal!

Edit: It was update KB5009543

------------------------------
Emma Burns
Ground Control Ltd
Europe/London
------------------------------

[diane.sanzone]

We have the same. Uninstalling the patches did help, and based on the "support" page, it seems BP is already aware and working on it.  I also found this site which pulls out three patches included in the CU that are specifically "AD" related.  We haven't gone to the level of uninstalling individual KBs yet, but perhaps this is a good place to start if you're not up for rolling back the whole thing:

https://dirteam.com/sander/2022/01/11/three-active-directory-vulnerabilities-were-addressed-during-microsofts-january-2022-patch-tuesday/

If anyone identifies the specific KB and can post it here, please do!  I'll be sure to do the same as well.

Also, this was from our PC level event viewer log and might also help shed some light.  I'm trying to get this over to BP support but, of course, having issues accessing the ticketing pages.  Oh, the irony!

The Security System has detected a downgrade attempt when contacting the 3-part SPN

 HTTP/[servernameremoved]:8199/BPServer

 with error code "The SAM database on the Windows Server does not have a computer account for this workstation trust relationship.

(0xc000018b)". Authentication was denied.

------------------------------
Diane Sanzone
------------------------------

[steven.boggs]

Hi Brenton,

Our latest updates and guidance for this scenario can be found in our Knowledge Base here: https://help.blueprism.com/Alerts/1784860762/Latest-on-Windows-updates-from-11th-January-2022-causing-authentication-issues-in-Blue-Prism.htm

This page will be continually updated as information about this becomes available.

------------------------------
Steve Boggs
Senior Software Support Engineer
Blue Prism
Austin, TX
------------------------------

[BrentonWestwood]

Thanks and yes, Blue Prism Support is aware.   I created a case and found that out as well.    Also, removing the patch did allow things to get back to working on a desktop that I performed that operation on.    

From the Control Panel > Programs > Programs and Features, I removed the KB5009545 Windows Security Update from a desktop that had the issue and after a restart and logging back into the desktop, Blue Prism launched and signed in!

After the uninstall, the KB5008206 Windows Security Update is listed.

------------------------------
Brenton Westwood
Systems Analyst
Southern Company
------------------------------

[diane.sanzone]

I've since learned from my desktop team that the KB will vary based on your Windows build, and that with Windows 10 you can no longer cherry pick which updates in the CU you can apply, so sadly posting the individual KB won't help here since there were only 2 this month.

We are monitoring the BP Alert on this and they indicated it's related to SSO configurations, so we're contemplating removing the AD authentication/SSO and just manually logging in.  Another internal suggestion was to patch the desktops/resource VMs AND the supporting servers (only our desktops were patched last night - servers are still pending).

Has anyone tried this? If your developer/bot workstations are patched AND your BP server is patched AND your BP Database server is patched - does SSO configuration for the login agent work?  

We need to coordinate like 4 different teams on our side to test this but we might do it tomorrow - I'll post the results if we do end up testing it but would appreciate any feedback from anyone else that already tried it.

Thanks!

------------------------------
Diane Sanzone
------------------------------

[BrentonWestwood]

If the patching of the servers breaks the desktops SSO, further Blue Prism development will end tonight.    So, I guess I should have an answer late tonight.   Thanks for posting this thought.   Fingers crossed that the server patching does not break things.    Wow.

------------------------------
Brenton Westwood
Systems Analyst
Southern Company
------------------------------

[BrentonWestwood]

Tonight our Blue Prism Evaluation and Development servers got the Windows Security Patches installed (KB5009546). My client and runtime desktops did not have the patch and the desktops were able to connect to Blue Prism after the server patching. Thus, the desktop and server patches do not need to be in synch, it appears.

After the patching of our Evaluation and Development servers, I tried to connect to those environments from a desktop that has the new January Windows Security Patch (KB5009545 on the desktop). The connection did not work on the patched desktop.   Yet, the connection worked on the desktop for which I had removed the January Windows Security Patch after the servers were patch (as alluded to above).

------------------------------
Brenton Westwood
Systems Analyst
Southern Company
------------------------------

[diane.sanzone]

Thanks for the information, Brenton!  We're likely moving forward with the server patching of our test environment today. Good to know that I'll still probably be able to develop afterward.  Our next thought is to temporarily disable the SSO configuration pending a more permanent fix from BP.  Our IT security department is super strict so we can't afford to wait long without these patches applied.  
I'll be sure to keep posting any information I have here. Hoping BP gets us all a real fix soon!

------------------------------
Diane Sanzone
------------------------------

[AP.Philippa]

Hi Steve,

Is it already known when we can expect a hot-fix patch from Blue Prism to be released?
For now we have been able to roll back the Security Windows update, however this is not a sustainable solution.

Thanks in advance!

With kind regards,

------------------------------
Arthur Philippa
RPA Developer
Port of Rotterdam
Europe/Amsterdam
------------------------------