JWS Signature issue if Body contains Special Chracters

[TonyHann]

Hi,

I have an issue and am hoping someone might have some suggestions as to the cause,

I am looking to build a process, which utilises the Lloyds bank 'Confirmation of Payee' API

I am still working in their development environment at the moment with test data, so please be assured any names & account numbers that may appear in this post are purely sandbox test data.

The banks test data provided includes 131 accounts to run the API against, and includes the responses we should expect for each test cast if our API is working correctly - for 121 of these my request successfully returns the correct data.

For the 10 test cases that are failing, I receive an 'x-jws-signature validation failed' response. On investigating this, I can see that in all 10 of these accounts the 'Account Name' contains either the & or the ' character (none of the other cases which succeed contain either of these characters) 


The below example shows my test process with a working example:-


The body is created by the calculation:-


My Inputs into the HTTP request are :-


The Headers are as below :-


and the JWS Token is created by a custom code stage which I suspect is the cause of the issue.

The code within the code stage is as follows:-

Global Code:-

Stage Code:-
Inputs

Code:-


The results of one of the failed items is below:-


My initial suspicion was that I was not creating the 'Body' correctly when the special character was encountered, however the body looks fine in the above.

I have further validated that my 'Body' is correct, as if I use the same body in a process where I use the jwt.io website to create the JWS Token for me, rather than the above code stage, this results in a successful API call


The Output data created in the code stage again shows the 'name' to appear to be correct within the code stage

Would anyone have any suggestions as to what may be causing my issues in the above?

Many thanks
Tony

[ewilson]

Hi @TonyHann,

I believe Lloyds is using the Open Bank specification in the design of their API. If so, OBI requires that the header, payload, and signature of the request to be Base64Url encoded. I'd say there's something not right in your base64 encoding​. An easy test would be to add a bit of logic to look for those special characters (i.e. & and ') and then replace them with something like %26 (URL encoded &) and %27 (URL encoded ').

If we take an item from your test data (ex. A&G RIMSKY) and input it into an online B64 encoder/decoder utility (ex. https://www.base64url.com/ ), we can see there's a difference in the encoded representation because of padding characters. But if you replace the & with %26 you should see that the encoded output is identical between Base64 and Base64Url.

Cheers,
Eric

[TonyHann]

Hi Eric,

Many thanks for the reply,

There isn't actually any Base64URL encoding taking place in the process any longer, the data items for the signature creation are passed to the code stage as clear text.

Before having the code stage created, I was doing my testing using the jwt.io website to create my tokens and feeding these into my process (this is the second process shown in the original post), and you are absolutely correct, that initially I was simply Base64 encoding the header and body which didn't work, and then I realised that these needed to be base64URl encoded, following which I can get the jwt.io website to generate me tokens that work fine when I feed back into the API call irrespective of it there are special characters in the data or not.

The issue is when I use the code stage to generate the tokens,

The data items are fed to the code stage, to generate the token, and are also used to generate the Body which is fed into the HTTP request, however they are fed into the code stage simply as text,


Using the above example, if I replace the & with %26 in the Account Name (leaving the body unchanged as I know the body is correctly formed, as it works fine when using a token generated by jwt.io) I still get the signature error when running the API.


If I change both the name and the body as per below, then the signature does successfully pass and the API call is made, however the return I receive from the API is simply an 'Incorrect account name' response - this will be because I have simply removed the problem character from the signature creation, so the signature is valid for the payload provided, however the payload is incorrect (there is no account on record at the bank with the name S RICDSON %26 SON LTD.)


When I take the token created by the code stage for one of my data items that do NOT contain the special character, and compare that to a token created by jwt.io I can see they are identical (which I would expect as the API call works successfully), however when I compare the token created by the code stage when my account name has the special character, to the token I get from jwt.io when I pass the correctly Base64URL encoded data I can see the differences in the token, which was why I believe there to be an issue with the code stage, or potentially a missing step/ piece of code within the code stage needed to replace the special character?

Many thanks
Tony

[ewilson]

@TonyHann,

I've been looking over the Open Banking spec, reviewing the JWT spec, and a couple of the RFC's (i.e. 7515 and 7797 ). I could be wrong, but the way I interpret ​all of this is that the header and payload do need to be Base64Url encoded for calculating the signature, but then the payload portion of the JWT is removed from the resulting output before adding it to the x-jws-signature header.

Under the section titled Step 3: Compute the JWS, of the Open Banking spec, it states "The signer must compute the signature as a detached JWS as defined in RFC 7515, Appendix F". When you were b64Url encoding the data is that what you were actually doing?

Cheers,
Eric

​

[TonyHann]

Hi @ewilson

Thanks again for the continued help in trying to resolve this,

Yes, everything you have said is correct regarding  the header and payload need to be base64url encoded, I think the confusion has arisen with me saying 'I am not base64URl encoding the data' is because I am passing  the data into a code stage, I believe the relevant encoding is being done by the code within the code stage.

To try and clarify, I can generate a signature that will successfully work with the banks API when I create the signature externally, rather than using my code stage, using the jwt.io website and a header & payload I have created and base64URL encoded myself - the steps I take to do this are as follows :-

Take the header in clear text

Run this first through a Base 64 encode stage, and then through a calculation stage to convert the Base64 into Base64URL


and store the base64url encoded header (I'm masking the data as this contains my KID)

Then take the bank data items and run through the below calculation stage to create the Body



The body is fed into the Utility - HTTP object as part of the API, but is also ran through the above Base64 object and URL encode calculation stage, to create the base64URL encoded payload:-


I then join the 2 together to generate the content that I want to sign:-


If I then paste my 'Content to Sign' into the jwt.io website, you can see it correctly decodes back to my correct header and payload.



And then when pasting in my private key, I get the signature that I need to carry on my process


I then create an 'x-jws-signature' data item by taking by base64url encoded header and combining it with the (blue portion) JWS signature created by the jwt.io website, omitting the payload as this isn't required. 




Then passing all the data into the Utility - HTTP request object:-

This generates a successful call and response to the banks API (the matched:false / reason code BAMM response is the expected result from this test data,) 


The above method works correctly irrespective of if the Name data item contains any special characters or not.

The issue I am having is when I try and replace the reliance on the jwt.io website with a Blue prism code stage.

My process now looks like the below:-


The 'Create Body' calculation stage is the same calculation as in my jwt.io process
The Populate Headers and HTTP-Utility stages are also identical and have the same inputs

The bank data items are passed as inputs to the code stage, along with the Private Key and KID, I'm passing my bank data as 'clear text' and I'm assuming all of the encoding is done within the code stage:-


The global code section of the above object is:-


This code stage generates a JWS output in the correct 'x-jws-signature' format that I can pass directly into my Header for the HTTP request, and the API call is successful and returns the expected result as per the test data expected.


The only issue being, if the Name data item contains the & or ' character, the signature generated is incorrect and the API call fails:-


This was why I believe the issue to be with the code stage, rather than my base64url encoding of the data, as I no longer carry out any base64url encoding of the data, I simply pass the data to the code stage.

Trying to resolve the issue with the code stage , the developer who wrote the code for me is not seeing the same issue that I get when they run the code outside of Blue Prism, the code works correctly for them as standalone code even when special characters are encountered - unfortunately, they do not have access to a Blue Prism environment, so we cant verify if their code would work within BP for them.

Just for completeness, the libraries and namespaces in use are as below:-
 


Hope this gives a clearer insight into the issue I'm trying to resolve and thanks again for the help.

Cheers
Tony

[ewilson]

@TonyHann,

Ah, now I get it. Sorry, a little slow on the upload here. 😂 I took a look at the source code and documentation for JOSE-JWT on GitHub.​ The library does not apply Base64URL encoding by default. The only encoding it applies it UTF8 when it's ready string data to binary.

There's a note on the GitHub repos that discusses how to create a detached payload with encoding. You can find it here. Basically you just need to include a JwtOptions instance in your call to JWT.Encode() in your Global Code section. Something like this:

return JWT.Encode(payload, rsa, JwsAlgorithm.RS256, headers, options: new JwtOptions { DetachPayload = true, EncodePayload = true});​

Cheers,
Eric

[TonyHann]

Hi @ewilson

Thanks for your post, unfortunately the ​suggestion did not resolve the problem, however It's led me to look at trying at a few things that I think point towards where the issue might be, the stage I have got to now is as follows.

Firstly, trying your suggestion, I have noted that irrespective of it I use my original code of

return JWT.Encode(payload, rsa, JwsAlgorithm.RS256, headers );​

or your suggested replacement of

return JWT.Encode(payload, rsa, JwsAlgorithm.RS256, headers, options: new JwtOptions { DetachPayload = true, EncodePayload = true});​

having run several test cases with both code variants, and compared the signatures generated (for both test cases that contain special characters, and those that don't) - the signature is always identical for a given payload irrespective of which of the lines of code I use.

However, if I use the code:-

return JWT.Encode(payload, rsa, JwsAlgorithm.RS256, headers, options: new JwtOptions { DetachPayload = true, EncodePayload = false});​

then the signature is always different to that generated by both the above lines, and the API call fails, so I'm pretty sure that the base64URL encoding option is enabled by default, the document you pointed me towards in your post seems to supports that:-


I believe the DetachPayload option has no effect to the signature, as the signature is generated by the code line


which is already ensuring that only the header and the signature are used in the token, and the payload is not returned.

So, in order to do some more testing, I changed the above code line to:-  


and also ensured Detachpayload was set to false, so that I could compare the entire signature being generated in my test cases.

Then I took several of the test cases where the name contains no special characters, where I know the code is working and generated a signature through the jwt.io website, and also with my code stage and compared them both. As expected, they were absolutely identical.

Then I took a test case where the name has a special character and did the same, so I could compare both of the output:-


The section I have highlighted red above is the encoded payload, the top line was generated by the code stage, the bottom line was generated by the jwt.io website. so, now I can see what the encoded payload looks like for both a signature that works (jwt.io) and one that doesn't (my code stage), I can use Blue Prim to decode the base64url back to plain text to examine the difference.

The payload generated by jwt.io was 

eyJhY2NvdW50Ijp7InNvcnRDb2RlIjoiMTEwMDI2IiwiYWNjb3VudE51bWJlciI6IjExNDU5NTYwIiwibmFtZSI6IlAuRCZFLlAgREFTQ0hFUiIsImFjY291bnRUeXBlIjoiUGVyc29uYWwifX0

and decoding back to plain text gives:-

which is correct, as this was the original payload input,

However, the payload generated by my code stage was:-

eyJhY2NvdW50Ijp7InNvcnRDb2RlIjoiMTEwMDI2IiwiYWNjb3VudE51bWJlciI6IjExNDU5NTYwIiwibmFtZSI6IlAuRFx1MDAyNkUuUCBEQVNDSEVSIiwiYWNjb3VudFR5cGUiOiJQZXJzb25hbCJ9fQ

Which when decoded gives:-


So its clear that my code is replacing the '&' with '\u0026' for some reason, however I am no closer to understanding why?

I don't suppose, now we can see what the difference are, you have any suggestions as to what may be causing this?

Just to confirm, I get the same results when I used the 'EncodePayload=True' option.

Many thanks
Tony








 

[ewilson]

Hi @TonyHann,

I've spent some more time looking into this. I was able to get some test code, in Visual Studio, to generate a JWT that validates in JWT.io with a payload that includes the & character. However, I used a completely different library than what you're using. I went with Jwt.Net instead of Jose-JWT.

I also noticed that the type of key I was using seemed play a part in this too. I ended up generating a test key using OpenSSL in the PKCS#8 format which ultimately worked for me. Where did your key come from and do you know what format it's in?

Cheers,
Eric​

[ewilson]

@Tony Hann,

FWIW - Here's the code I put together using Jwt.Net.

The following code should be added to the Global Code section of your VBO:

// Utility function to build an instance of a JWT encoder that uses the RS256 algorithm.
internal IJwtEncoder GetRS256JWTEncoder(RSAParameters rsaParams)
{
  var csp = new RSACryptoServiceProvider();
  csp.ImportParameters(rsaParams);
  var algorithm = new RS256Algorithm(csp, csp);
  var serializer = new JsonNetSerializer();
  var urlEncoder = new JwtBase64UrlEncoder();
  var encoder = new JwtEncoder(algorithm, serializer, urlEncoder);

  return encoder;
}

// Utility function to build an RSAParameters instance with just a private key (i.e. not actual X.509 certificate).
internal RSAParameters GetRSAParameters(string privateKey)
{
  string pkey = GetPrivateKeyValue(privateKey);
  byte[] keyBytes = Convert.FromBase64String(pkey);
  AsymmetricKeyParameter asymKeyParam = PrivateKeyFactory.CreateKey(keyBytes);
  RsaPrivateCrtKeyParameters rsaPrivateCrtKeyParams = (RsaPrivateCrtKeyParameters)asymKeyParam;
  RSAParameters rsaParams = DotNetUtilities.ToRSAParameters(rsaPrivateCrtKeyParams);

  return rsaParams;
}

// Utility function for extracting just the actual private key value from an RSA private key string.
internal string GetPrivateKeyValue(string privateKey)
{
  var beginPrivateKeyTag = @"-----BEGIN PRIVATE KEY-----";
  var endPrivateKeyTag = @"-----END PRIVATE KEY-----";
  StringBuilder tmpPrivateKey = new StringBuilder();

  // We need to pull out just the value of the private key. We don't need the header/footer. 
  if (privateKey.Contains(beginPrivateKeyTag))
  {
    int start = privateKey.IndexOf(beginPrivateKeyTag, StringComparison.OrdinalIgnoreCase) + beginPrivateKeyTag.Length;
    int end = privateKey.IndexOf(endPrivateKeyTag, StringComparison.OrdinalIgnoreCase);
    tmpPrivateKey.Append(privateKey.Substring(start, (end - start)));
  }
  else
    tmpPrivateKey.Append(privateKey);

  // After extracting just the <VALUE> portion, we need to replace all \n (Unix newlines) with
  // Windows new line values (i.e. \r\n).
  tmpPrivateKey.Replace(@"\n", Environment.NewLine);

  return tmpPrivateKey.ToString().Trim();
}
​

​
And this code would be placed in the Code stage of the specific action you're exposing:

var rsaParams = GetRSAParameters(privateKey);
var encoder = GetRS256JWTEncoder(rsaParams);
var token = encoder.Encode(payloadFinal, new byte[0]);

// You can add your additional logic for pulling out the payload portion and creating the JWS Token here.​

Cheers,
Eric

[TonyHann]

Hi @ewilson

Thanks for the really detailed and helpful reply and for the code provided!​

The key I was using was extracted from a certificate provided by the bank, and was in PEM encoded PKCS#1 format,

Using the code you provided for the JWT.net libraries and converting my key into PKCS#8 format, I can now get the​ API call to work successfully using this code, including the data items containing special characters - so my problem is solved! 

Thanks again for all the help,

Cheers
Tony