Extract bearer token from Authorization when separated by multiple spaces

[anonymous]

How do you capture an OAuth access token into a variable when the token is separated from the Bearer authentication scheme keyword by multiple spaces?

The Authorization header has this ABNF:

credentials = auth-scheme [ 1*SP ( token68 / [ ( "," / auth-param )
    *( OWS "," [ OWS auth-param ] ) ] ) ]

RFC 6750 says that the auth-scheme in that ABNF is “bearer” (case insensitive), so I have to extract the token from the value after that with any amount of spaces.

I have found only one way to do this and it is very kludgey: Extract the token with preceding spaces and use a bit of JavaScript to trim it. So, this means I have an Extract like this:

<ExtractVariables name="Extract-Access-Token">
    <DisplayName>Extract access token from Authorization request header</DisplayName>
    <Header name="Authorization">
        <!-- No space between "Bearer" and the accessToken variable -->
        <Pattern ignoreCase="true">Bearer{accessToken}</Pattern>
    </Header>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <VariablePrefix>clientRequest</VariablePrefix>
</ExtractVariables>

Then, I can use a bit of JavaScript to trim the leading spaces. This is less than ideal. Is there no better way?

TIA!

[ssvaidyanathan]

HI @Travis Spencer

Welcome to the community !!!

I am assuming you are passing the bearer token to Apigee and validating that using the VerifyAccessToken operation that is available within the OAuth policy

If this policy executes correctly, the token is actually stored in a flow variable called “access_token”. It automatically removes the “Bearer” from it. You can use this variable anywhere in the flow context. The only condition is that this variable gets populated if the policy executes correctly. The policy should handle the spaces and just store the token in that variable.

[anonymous]

Thanks, @Sai Saran Vaidyanathan, for the reply and the nice welcome.

You are right that I’m passing the token to an API proxy in Apigee, but I’m not using the VerifyAccessToken operation because that can only verify tokens issued by Apigee. My token is issued by an instance of Curity, so I’m trying to grab it out of the request and send it to Curity for verification. It is for this reason that I’m trying to use the ExtractVariables operation to snag the token, but struggling with this corner case.

Any suggestions on how to get just the token from such a header as this:

Authorization: bearer      TOKEN123

Seems the JavaScript trim technique is the only one?

[ssvaidyanathan]

@Travis Spencer - Thanks for the clarification. Yeah looks like you will have to extract it via Extract Variables and then use JS policy to remove the spaces or do both in JS itself using Regex.

Just curious - with this approach you have taken, since Apigee is not validating the token, you miss all the features like Analytics, etc that Apigee provides out of the box.

I believe you have exposed an OAuth endpoint for consumers to call (pointing to Curity directly or a plain pass through proxy via Apigee) to generate the token. And for the subsequent functional API call (that is exposed in Apigee), consumer would make the request to Apigee, Apigee would send it to Curity to validate and then send it to the back end service.

Is that right ?

If that is right, I have a recommendation - why don’t you expose the OAuth on Apigee itself but the token provider is still Curity. The process is as follows -

• Apigee authenticate the incoming calls (via apps’s client ID and secret)
• Apigee sends the request to Curity Token service which generates the token and sends it back to Apigee
• Apigee will then store that token (with all the info like expires_in, refresh_token, custom attributes, etc) before sending the response to the caller (as if Apigee minted the token)
• On the subsequent API call, the caller sends the request with the bearer token to Apigee
• Apigee validates the token instead of sending it to Curity but sends it to the backend service after validation

With this you get all the features as an API platform - Security, Traffic Mgmt, Rate Limiting, Analytics, etc. and you are still using Curity as your IDP. You also save the calls to Curity for every call being made to Apigee. The only call to Curity is from the Apigee’s OAuth proxy.

We have many many customers doing this with their own IDP services. There are many posts in the community about this as well. Please read this doc.

Didn’t mean to deviate from the main thread of this post, but wanted to highlight this feature as I am not sure if you knew or considered this already.

Let me know if you still have questions. Happy to help !

[anonymous]

Thanks again, @Sai Saran Vaidyanathan for the help with this. You’re right about what I’m trying to do. It’s like this:

(We see this flow so often with all the various API gateways that we gave it the nickname the “Phantom Token flow”.)

I understand that this approach will mean that many of Apigee’s reporting features are lost; our customers want these, so we need to find another way. Exposing OAuth on Apigee and using Curity as a token provider, however, would mean that most of Curity’s features are lost. Curity is MUCH more than a token issuer and validator. It supports all of the standard OAuth and OpenID Connect flows, some of its own, the device flow, revocation, introspection, token exchange, etc., etc. Not only that, it’s a world-class authentication server, reducing REALLY complex login and SSO flows to simple configuration. Because of this, we can’t simply proxy Curity or relegate it to be a token issuer/validator.

Can we have both Curity’s advanced login/token issuance + Apigee’s API analytics? When I read your comment and that doc you pointed me to, it sounds to me like Apigee’s VerifyAccessToken operation depends on the token being in its database. This makes me wonder if it’s possible to use Curity as the OAuth server but pump its tokens (or some metadata thereof) into Apigree’s token database. Curity can store tokens wherever. It supports a half-dozen different RDBMS, a handful of LDAP server, SCIM 1 and 2, and a simple JSON over HTTP kinda web service; it also has an Java SDK to build others, and we could actually build one just for Apigee without much fuss.

If this is not possible, there may be other way: during token issuance, Curity can do all kinds of tricks; it supports a scripting language where customers can jack into the issuance process. For example, during token issuance from one of these scripts, a Web service call could be made. This could be a policy in Apigee that calls the OAuthV2/GenerateAccessToken operation.

My preference would be the first where we use Apigree as a token store. The sideways call during token issuance means that a lot of scripts for the various flow (like for revocation, CC, ROPC, etc.) need to be customized by customers who are using Curity with Apigee.

What do you think? Could either of these approaches work?

[ssvaidyanathan]

@Travis Spencer

Thanks for the clarification. I am loving this post as its getting into a nice useful resource for all. Thanks for spending time and educating all

I am with you totally. By not validating the token in Apigee, you not only miss the Analytics feature, but more like the Product Management feature itself which can help with Security, Rate Limiting, Product Catalog, etc. Service providers would surely want these and probably bought Apigee for these out of the box capabilities it offers as a platform. Customers who are exposing these micro services would like to categorize them and probably stack them as Product services which Apigee supports out of the box. Customer would miss a lot of things from a platform perspective with Apigee in their tech stack

With Apigee storing the tokens and validating it, it still provides all those capabilities complementing the wonderful features Curity provides. I have come up with a simple sequence diagram (happy flow only is captured) that depicts how this could work. Check it out and let me know if you have further question

Here, App’s first interaction is to get the token. Apigee makes a Service callout to Curity which responds with the token and on the way back, Apigee stores the token in its store along with all the metadata info passed by Curity and then sends to the app.

Now the app makes a call to the microservice exposed via Apigee. Apigee would validate the token and forward the request to the service directly.

In your Phantom Token service flow, you can use Apigee as the Reverse proxy and instead of caching it, Apigee can store it which in this case will be used for validation for subsequent Microservice calls that can be extended for other features.

With this - we get the capabilities of both Curity and Apigee. Curity still holds to be the Identity SOT. Let me know if this clarifies.

[anonymous]

Thanks, @Sai Saran Vaidyanathan, for the continued help. You guys really know how to make an open and useful forum!

We talked about your answers today and we’re thinking that this flow would ensure that joint customers can utilize all of the features of both Curity and Apigee:

Would this work? Can some sort of “store-token” proxy policy be defined in Apigee that does nothing but stores an externally created token as in that doc you pointed me to?

If so, we’re thinking that later on, when the client calls a back-end API, Apigee will still do an introspection call to Curity but this time it will get the “Apigee token” in the introspection response (together with the phantom token, scopes for access control, etc.). With this Apigee token, an OAuthV2 policy like this one could be used to verify the token:

<OAuthV2 name="GenerateAccessToken">
  <Operation>VerifyAccessToken</Operation>
  <AccessToken>apigeeToken</AccessToken> <!-- Assigned from the introspectionResponse object's JSON -->
</OAuthV2>

After this, the policy in Apigee would actually forward Curity’s JWT token, allowing the microservices to verify all identity off-line (i.e., without communication with Curity).

If this technique was used, would the analytics in Apigee still work?

TIA!

[ssvaidyanathan]

@Travis Spencer

Thanks for considering this option. Appreciate you guys taking the time to discuss within your team.

Technically, this looks like a working solution (however, a POC will be good to confirm), but I see some operational challenges. For example, the client now makes call to two different systems - Curity (for the token) and Apigee (for the microservice). Because of this, how are the apps authenticated separately? Are we planning to store the app information on both the systems? how about the Client IDs and secret -will they be the same across the two so that the calling apps don’t have to muddle with two different creds?

To avoid that - I recommended having Apigee as the point of network for the App while Apigee internally communicates with Curity. In this case, Apigee can register the app, generate the client ID and secret.

Curity can consider Apigee as a client and enable security mechanisms for any communication between these two systems

Also - not sure why you recommended a REF token to be stored in Apigee, we could store the actual token itself ?

Lastly - for every Microservice call, I see the network hop happening between Apigee and Curity - for introspection. Do we really need that ? Why not store the token data in Apigee itself and the app pass that in the header ? Apologies if I am missing something.

Would be better if we could just do that for token generation and not for validation so that the overall SLA is high

[ssvaidyanathan]

Hey @Travis Spencer - were you able to do any POC ? any interesting findings ? curious to know if any progress were made.

[anonymous]

Sorry for the slow reply, @Sai Saran Vaidyanathan. We’ve been heads down implementing this. In response to your earlier questions/points:

the client now makes call to two different systems - Curity (for the token) and Apigee (for the microservice).

This is normal OAuth. My app might consume a dozen APIs. They’re all different from the OAuth server, so the client has this burden to bear.

We’re planning to do a fire-and-forget from Cuirty to Apigee during token issuance, so it won’t impose any performance impact on the OAuth clients interacting with Curity. The introspection call from Apigee to Curity will be cached for as long as Curity says to do so. We’ve parsed the Cache-Control response header and used Apigee’s normal caching operations for that.

Because of this, how are the apps authenticated separately?

With separate tokens granted separately. Confidential clients will have different client IDs and secrets (unique per installation even).

Are we planning to store the app information on both the systems? how about the Client IDs and secret -will they be the same across the two so that the calling apps don’t have to muddle with two different creds?

Apigee will only know about the client ID, not the secret. The latter will only be in Curity. We’re going to make an API proxy in Apigee that fronts Curity’s Dynamic Client Registration (DCR) endpoint. The developer portal will only interact with this standardized API. In this Apigee proxy, it’ll forward the call to Curity, store the client ID that Curity generates in the Apigee database, and return Curity’s DCR response to the portal. Consequently, the app will only have one credential and there will be only one secret that is hashed and stored in one database – the database of the OAuth server, Curity.

I recommended having Apigee as the point of network for the App while Apigee internally communicates with Curity.

I think it’ll work to have Apigee proxy DCR, but not much else. The P2P calls, like revocation, introspection, token issuance could be done through a proxy, but we would not recommend that customers do that for anything more than WAF-type protections. Any multi-channel OAuth flow (e.g., hybrid OIDC, code flow, device flow, assisted token flow, etc.) will not work with a proxy in the middle.

Curity can consider Apigee as a client and enable security mechanisms for any communication between these two systems

We’ve built systems like this before we had Curity. It makes the API gateway a honeypot – a God client, and isn’t a pattern I’d recommend, especially now that we have Curity

not sure why you recommended a REF token to be stored in Apigee, we could store the actual token itself ?

There’s an important reason for this that isn’t immediately obvious: revocation. A token can be revoked at any point. To avoid more cross system communication, the “Apigee token” approach will avoid the need to call over to Apigee from Curity when a token is revoked. Furthermore, identity data changes and scopes expire (Curity allows for a TTL on scopes that are < the expiration of the tokens). By always issuing a REF token, the API proxy will always dereference it and get the most-up-to-date info. Furthermore, a REF will ensure that the app isn’t effected by GDPR and other such regulations because it’ll never have any PII. When you put this together with a cache and a fire-and-forget approach to storing the “Apigee tokens,” we expect very good performance and throughput.

for every Microservice call, I see the network hop happening between Apigee and Curity - for introspection. Do we really need that ?

Curity’s REF tokens are globally unique. Therefore, the cache in Apigee is a Global one. This means that all microservices will share that cache. Thus, one microservice’s call to Curity can be used by another.

Why not store the token data in Apigee itself and the app pass that in the header ?

I think this would work in some deployments, but it’s not as dynamic as the other I’m advocating here. The policies that we’ll publish on GitHub will use the “Apigee REF token” idea, but I wouldn’t be surprised if some customers who aren’t concern about data becoming unsynchronized after revocation will take this alternative approach.

Apologies if I am missing something

No need! We’re Apigee babies over here, so we really appreciate your help!

were you able to do any POC ?

Our PoC is ongoing and is supposed to finish by the end of the year. We ran into a snag with some odd errors. That’s cost us a lot of time, but we have a meeting on Monday with some Apigee folks to help us get that cleared up. We’ll keep updating this thread as we continue.

Thanks again for everything so far.

[ssvaidyanathan]

@Travis Spencer - thanks for the details. Really appreciate it.

I see the issue is resolved now. Please let us know if there is anything else we can help with. Curious to know more info on the POC you are working on. Hope it will be pushed into your GitHub repo as reference implementation.

Happy to help !

[anonymous]

Stuck again, but VERY close now. So far, this is what we have:

• An API proxy that stands in front of Curity’s DCR endpoint. If Curity replies with a 201, created, then Apigee’s management API is used to create the client, add Curity’s client_id as the consumerKey and delete the generated consumerKey and consumerSecret. This is working fine; would be great if the client_id could be set in the OAuthV2 GenerateAccessToken policy though.
• An API proxy that stores “Apigee tokens”. This is called asynchronously from Curity whenever a token is created. This is working OK. The only issue is that Curity must always use the client_credentials grant_type, or else something must be done to get a code as implicit and code grant types require that in Apigee; I’m happy to leave it like this, but am unsure if it’ll affect reporting down the road. Here’s a sample of the POST message to the proxy hosted at /token in Apigee:

client_id=f9e0d55d-bbec-4f61-913a-cc23685b06e9&token=2898570a-58f4-4050-ae84-bd5e3e9a9a0b&grant_type=client_credentials

The client_id there is the one that Curity created using DCR and coerced Apigee into using. The token there is this “Apigee token” I keep referring to, not the actual access token. The heart of this proxy is this policy:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<OAuthV2 name="OAuth-v20-1">
    <DisplayName>OAuth GenerateAccessToken</DisplayName>
    <ExternalAccessToken>request.formparam.token</ExternalAccessToken>
    <ExternalAuthorization>true</ExternalAuthorization>
    <Operation>GenerateAccessToken</Operation>
    <GenerateResponse enabled="true"/>
    <GenerateErrorResponse enabled="true"/>
    <StoreToken>true</StoreToken>
    <ExpiresIn>-1</ExpiresIn>
    <SupportedGrantTypes>
        <GrantType>authorization_code</GrantType>
        <GrantType>client_credentials</GrantType>
        <GrantType>implicit</GrantType>
        <GrantType>password</GrantType>
    </SupportedGrantTypes>
</OAuthV2>

Despite the issue with the hard-coded grant_type, this is working.

• Every API that should be protected embeds a shared flow that does a call to Curity’s introspecton endpoint. This is cached, and working nice. The introspection results were updated in Curity to return the “Apigee token.” Now, this is where we get stuck. In the introspection shared flow, we added the following policy:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<OAuthV2 name="Validate-Apigee-Token">
    <DisplayName>Validate Apigee Token</DisplayName>
    <Operation>VerifyAccessToken</Operation>
    <!-- There is no Bearer prefix, so override the default -->
    <AccessTokenPrefix/>
    <AccessToken>introspectionPayload.apigeeToken</AccessToken>
</OAuthV2>

This always fails with the following error:

{
    "fault": {
        "faultstring": "Invalid Access Token",
        "detail": {
            "errorcode": "keymanagement.service.invalid_access_token"
        }
    }
}

To debug this, we added a GetOAuthV2Info policy just before the OAuthV2 one, like this:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<GetOAuthV2Info name="Get-OAuth-v20-Info">
    <DisplayName>Get OAuth v2.0 Info</DisplayName>
    <AccessToken ref="introspectionPayload.apigeeToken"/>
    <IgnoreAccessTokenStatus>true</IgnoreAccessTokenStatus>
</GetOAuthV2Info>

Oddly, this prints info in a trace that makes us think it finds the token:

We can’t do much else to determine why the token is invalid. Is there something missing in our VerifyAccessToken operation? Is there any way to debug this more from the “outside”?

ANY help or clues would be MUCH appreciated.

[ssvaidyanathan]

@Travis Spencer

Just to understand, in your call

client_id=f9e0d55d-bbec-4f61-913a-cc23685b06e9&token=2898570a-58f4-4050-ae84-bd5e3e9a9a0b&grant_type=client_credentials

has the token that needs to be stored in Apigee as OAuth token (Bearer) ? Is that right ?

Have you set the oauth_external_authorization_status to true before the GenerateAccessToken policy ? More info here

Now in your actual API call where you are passing the token, its failing in the VerifyAccessToken call. Can you try it using Bearer ? Thats the default value it supports. So while storing the token in Apigee, it uses Bearer as well. So while passing the token, please include the prefix as well. Also please remove the AccessTokenPrefix in your VerifyAccessToken OAuth policy

Would it be possible to include me to your trial org so that I can take a look at the proxy ?

[anonymous]

I gave you access, @Sai Saran Vaidyanathan.

The policy you want to look at is the token one. That is the one called by Curity after token issuance to store the “Apigee token”. As you can see there, it’s require Curity to authenticate using basic auth. There is no OAuth involved in authenticating this request. The payload looks like an OAuth Client Credential flow, but that’s only to make the use of the OAuthV2 policy easier (defaults apply then). Before that is called, we use AssignMessage to set oauth_external_authorization_status to true, as in that article you linked to.

Here’s the flow and the interesting policy assertions:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
    <PreFlow name="PreFlow">
        <Request>
            <Step>
                <Name>Basic-Authentication-1</Name>
            </Step>
            <Step>
                <Name>getEncrypted</Name>
            </Step>
            <Step>
                <Condition>(oauthServerId != private.oauthServerId) or (oauthServerSecret != private.oauthServerSecret)</Condition>
                <Name>Access-Denied-Fault</Name>
            </Step>
            <Step>
                <Name>assign-oauth-variables</Name>
            </Step>
            <Step>
                <Name>OAuth-v20-1</Name>
            </Step>
        </Request>
        <Response/>
    </PreFlow>
    <PostFlow name="PostFlow">
        <Request/>
        <Response/>
    </PostFlow>
    <Flows/>
    <HTTPProxyConnection>
        <BasePath>/token</BasePath>
        <Properties/>
        <VirtualHost>default</VirtualHost>
    </HTTPProxyConnection>
    <RouteRule name="noroute"/>
</ProxyEndpoint>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage name="assign-oauth-variables">
    <DisplayName>Set Mandatory variables to use external authorization</DisplayName>
    <AssignVariable>
        <Name>oauth_external_authorization_status</Name>
        <Value>true</Value>
    </AssignVariable>
</AssignMessage>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<OAuthV2 name="OAuth-v20-1">
    <DisplayName>OAuth GenerateAccessToken</DisplayName>
    <ExternalAccessToken>request.formparam.token</ExternalAccessToken>
    <ExternalAuthorization>true</ExternalAuthorization>
    <Operation>GenerateAccessToken</Operation>
    <GenerateResponse enabled="true"/>
    <GenerateErrorResponse enabled="true"/>
    <StoreToken>true</StoreToken>
    <ExpiresIn>-1</ExpiresIn>
    <SupportedGrantTypes>
        <GrantType>authorization_code</GrantType>
        <GrantType>client_credentials</GrantType>
        <GrantType>implicit</GrantType>
        <GrantType>password</GrantType>
    </SupportedGrantTypes>
</OAuthV2>

Now in your actual API call where you are passing the token, its failing in the VerifyAccessToken call.

That’s correct. That’s in the introspection shared flow, at the end.

Can you try it using Bearer ? Thats the default value it supports. So while storing the token in Apigee, it uses Bearer as well. So while passing the token, please include the prefix as well. Also please remove the AccessTokenPrefix in your VerifyAccessToken OAuth policy

We tried to include "Bearer " in the shared flow, but not in the GenerateAccessToken operation. That has always been unprefixed. We will try to prefix that and remove the AccessTokenPrefix tag from the OAuthV2 VerifyAccessToken operation tomorrow.

Meanwhile, if you find any more clues or think of any new ideas, please share them.

Thanks for the continued help. VERY much appreciated!

[ssvaidyanathan]

As suspected - was an issue with the AccessTokenPrefix. Once you remove it, should work

[anonymous]

After much discussion with @Sai Saran Vaidyanathan, we managed to get this working. We published the step-by-step guide on our web site, and a summary article has been published in this forum.

[ssvaidyanathan]

Thats awesome @Travis Spencer.

[JudithKramp]

Is it possible to extract attributes from an invalid AccessToken?
We are beta testing a new API and have only granted access to some customers (approve API Product in their Developer App).
We would like to be able to see which customers are trying to send requests to our new API without access to it.
The information about the customer is inside (custom) attributes of the OAuth2 AccessToken.
Both the VerifyAccessToken and the GetOAuthInfo policies fail and do not output the attributes.
Any suggestions?

[dchiesa1]

Hi - if you want an answer, you need to ask new questions on NEW THREADS, not as comments on 6-yr old unrelated threads. No one sees your questions when you embed them this way.

Post a new question!