Verify JWT - Provided Key Id did not match any Public Key

Google Cloud Apigee
1

I have configured the verify JWT policy below

<VerifyJWT name="VerifyJWT" continueOnError="true"> 
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables> 
  <Algorithm>PS256</Algorithm> 
  <Source>request.content</Source> 
  <PublicKey> 
    <JWKS ref="register.jwks"/> 
  </PublicKey> 
</VerifyJWT>

The value of kid in the header of JWT is present in the provided jwks. But still I am getting

Provided Key Id did not match any Public Key

I am getting the JWKS using a service callout and assigning the response to this variable register.jwks

Could someone help me out on this? Am I missing anything?

1 Like
2

There are multiple reasons for the error " Provided Key Id did not match any Public Key". In fact all these things must match in order for VerifyJWT to succeed in finding a key in the JWKS:

  • kid must be present in JWK, and must match kid in JOSE header
  • kty must be present in JWK, and must be appropriate: “RSA” if alg is RS{256,384,512} or PS{256,384,512}, or “EC” if alg is ES{256,384,512}.
  • use, IF PRESENT in the JWK, must be appropriate: “sig” if using an RSA algorithm (including PSS); and “enc” if using an ECDSA algorithm.
  • if ECDSA, then crv must be present in the JWK and must match the alg specified in VerifyJWT; and the x and y properties must be present in the JWK.
  • if RSA, then the n and e properties must be present in the JWK.

If you post your JWKS, and your JOSE header, I might be able to help you diagnose further.

3

Hi @dchiesa1

We have some strange situation where in some MP node java callout policy is thorwing this error “Could not find PublicKey for provided keyId” where as in some MP nodes it is working fine. Can you explain why it is behaving ?

Follow up question is there a way to bind this proxy to certain MP’s where it is working. Like I have 5 MP nodes it is working on 3 but not working on 2.

4

Hi satyendrasrivas

We have some strange situation where in some MP node java callout policy is thorwing this error “Could not find PublicKey for provided keyId” where as in some MP nodes it is working fine. Can you explain why it is behaving ?

When you use the phrase “java callout” are you referring to THIS Java callout? https://github.com/apigee/iloveapis2015-jwt-jwe-jws

If yes, then… why are you using that? You could be using the built-in Apigee policies that verify JWT. [doc link]

The first thing I would do is double check that the keyid that cannot be found is the same keyid in both cases - the success and the failure cases. If the keyid is different, then … maybe the error is “correct” and expected.

As regards the error that you see on some MP nodes but not others… no, I am sorry, but I do not have any explanations for why it would behave differently on different nodes. I guess you have an OPDK installation, is that right? In that case, I would look in the system.log for the Message Processors, to see if there is any logging there that would indicate an unexpected error in the runtime.

If you are using OPDK, one possibility is that the required Java configuration is different across the different nodes. This won’t happen with a hosted Apigee (Apigee Edge or Apigee X), because Google insures that every node that runs Apigee is identically configured. But some shops that run their own Apigee have gaps in their governance and the MP nodes might be configured differently. As an example, if you are using Java8, prior to Java8_u161 (I think), and you do not have the unlimited strength crypto extension policy enabled, then the Java JVM will not be able to load RSA keys beyond 1024-bit strength. If that is happening in a few of your MP nodes, then you wouldn’t see that error in Trace, but…you might see an error like “could not find publickey…”. Scanning the system.log would provide more evidence for you. I would want to make sure that the Java configuration is exactly the same across all of these MPs.

Another possible source for different behavior from different MPs is a difference in the network configuration that you have for each MP. If you have configured the policy to retrieve public keys from a JWKS endpoint, and some of the MPs have connectivity to the JWKS endpoint, and some of them do not, then… you could see the “could not find publickey…” message. Again, Scanning the system.log would provide more evidence for you.

is there a way to bind this proxy to certain MP’s where it is working. Like I have 5 MP nodes it is working on 3 but not working on 2.

Yes, the way to do that is to use distinct environments within Apigee. Environments are just named subsets of Message Processors. But I think taking this approach is probably not the right thing to do. Better to identify the root cause of the discrepancy in behavior and rectify that problem.

So in summary, I suggest

  1. Verify that the keyID is the same in the success cases and in the error cases.
  2. If you are using an older Java callout, stop using that. Use the builtin, supported JWT policies provided by Apigee.
  3. If you can’t do that, or if you already ARE using the builtin policies, then … verify that the MPs are all prepared identically w.r.t. Java runtime and JCE unlimited strength crypto policy files if necessary. And also that they all have connectivity to any JWKS endpoint.
  4. Check the system.log for the various MPs to see if there are exceptions logged that indicate more specifically what the problem might be.
  5. If you have checked all of that , and still you see the discrepancy in behavior across different MPs, then I suggest contacting Apigee support for additional assistance.
5

Thanks for detailed the reply @dchiesa1 . really appreciate your response.

  1. we are on private cloud installation 4.17.05. The JWT policy wasn’t available in that version.

  2. The first thing I would do is double check that the keyid that cannot be found is the same keyid in both cases..

Yes It is same keyid

  1. we found minor difference in the java version and MP version as the problematic nodes were added later. There are 50+ apiproxy deployed but we are having issue in one particular proxy using JWT verification(java call out).

Following combination is having issue

openjdk version “1.8.0_302”
edge-message-processor-4.17.05-0.0.1432

Nodes with Following version is not having issue

openjdk version “1.8.0_242”
edge-message-processor-4.17.05-0.0.1231

  1. I checked the MP logs but it doesn’t have any information about error. Trace from UI gives stacktrace about this error.

com.google.auth.oauth2.TokenVerifier$VerificationException: Could not find PublicKey for provided keyId: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx at com.google.auth.oauth2.TokenVerifier.verify(TokenVerifier.java:164) at com.xxxxxx.enterprise.oauth.GoogleOauth.verifyJWTToken(GoogleOauth.java:71) at com.xxxxxx.enterprise.oauth.GoogleOauth.execute(GoogleOauth.java:108) at com.apigee.steps.javacallout.JavaCalloutStepDefinition$ClassLoadWrappedExecution.execute(JavaCalloutStepDefinition.java:176) at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution$1.run(JavaCalloutStepDefinition.java:242) at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution$1.run(JavaCalloutStepDefinition.java:240) at java.security.AccessController.doPrivileged(Native Method) at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution.execute(JavaCalloutStepDefinition.java:240) at com.apigee.steps.javacallout.JavaCalloutStepDefinition$CallOutWrapper.execute(JavaCalloutStepDefinition.java:111) at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:146) at com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask.call(AbstractAsyncExecutionStrategy.java:74) at com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask.call(AbstractAsyncExecutionStrategy.java:45) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

  1. what information you want to check for below statement:

verify that the MPs are all prepared identically w.r.t. Java runtime and JCE unlimited strength crypto policy files if necessary.

I see unlimited strength jars are present in <jre_home>lib/security/policy/unlimited dir. crypto.policy is defined as unlimited.

  1. what configuration we needs to check for below:

Another possible source for different behavior from different MPs is a difference in the network configuration that you have for each MP.

Appreciate your help!

6

From your response there are two important pieces of information.

  1. You are using 4.17.xx. This is out of support for 2 years at least.
  2. this is your own Java callout. This is not Java code anyone at Apigee has provided. This is not code I am familiar with at all.

Because of this, I am unable to help you. I suggest you contact whoever wrote the Java code, to help you troubleshoot.

Good luck!