Apigee allows connections to "insecure" targetServer?

Google Cloud Apigee
1

A backend api team asked me to create a proxy to their service on aws and to secure the backend connection with mutual TLS. I noticed right away, they have not configured TLS correctly. They used a certificate issued to other backend services we have, where the cert has a wildcard like “CN=*.mycompany.com”. This team didn’t create a custom DNS name, so their host name is like “whatever.amazonaws.com”.

If you open their host in a browser it says “Not Secure”. It crosses out the “HTTPS” and shows an error message “NET::ERR_CERT_COMMON_NAME_INVALID.” You can click the ‘advanced’ option and proceed anyway.. but this is certainly a bad practice.

Also, if you try to curl their backend, it gives an error “SSL: no alternative certificate subject name matches target host name.” You can add “–insecure” and then curl allows a connection, but again, this is a bad practice.

Anyway, I created a proxy (on Edge cloud) and was surprised to see that Apigee doesn’t care about the SSL error and made a successful request!

I don’t want to enable connections to servers with misconfigured TLS, so any thoughts on how can I prevent this?

For reference, here is the targetServer definiton I used.

	{
        "name": "targetServer",
        "host": "whatever.amazonaws.com",
        "isEnabled": true,
        "port": 443,
        "sSLInfo": {
            "ciphers": [],
            "clientAuthEnabled": "true",
            "enabled": "true",
            "ignoreValidationErrors": false,
            "protocols": [],
            "trustStore":"ref://trustStore-outbound-test-ref",
            "keyStore":"ref://keystore-outbound-test-ref", 
            "keyAlias": "keystore-outbound-test" 
        }
    }
2

You are trying southbound MTLS, which is apigee to backend service. The key alias you are using is produced by apigee to the backend service. And you have the backend service’s root and intermediate certificate added to your truststore. So, Apigee will trust the certificate.

To overcome this you can add one file including full certificate chain of the certificate in truststore. That will ensure only certificate in the file will be trusted by Apigee.

3

No, I think you missed the problem. My issue here is that Apigee is NOT throwing any warning or error when this other server appears to have stolen somebody else’s certificate. Its like if I was able to steal the public and private certificates for “google.com” and then use them on my website “fakeGoogle.com”. You should not trust my server is “google” because “fakeGoogle.com” is not “google.com”.

We have MTLS setup using this same certificate chain because we have other targetServers that are correctly using that certificate.

4

Can we check one more thing?

Can you also show the configuration for the targetconnection?

I want to see what you get from querying

/v1/o/$ORG/apis/$PROXY/revisions/$REV/targets/$TARGET

In case it is not clear, the $TARGET in the above is not the targetserver name. It’s something different. It’s the name of the targetendpoint in the proxybundle. That targetendpoint can possibly refer to a targetserver.

5

I think it is trusting till it has the root certificate in the trust store. You see that in the browser as the browser tries to verify that with CA. Apigee is not going to verify that. But yes, if you want you can verify the cn information and target url, if that doesn’t match you can throw the error.

6

Here is the response from that path. I replaced some of the content with […] as it didnt seem relevant to have each policy and I suspect you’re looking for the “connection” details.

{
  "connection": {
    "connectionType": "httpConnection",
    "loadBalancer": {
      "maxFailures": 0,
      "retryEnabled": true,
      "server": [
        {
          "isEnabled": true,
          "isFallback": false,
          "name": "targetServer",
          "weight": 1
        }
      ],
      "targetDisableSecs": 300
    }
  },
  "connectionType": "httpConnection",
  "description": "",
  "faultRules": [...],
  "flows": [...],
  "name": "target",
  "postFlow": {
    "name": "PostFlow",
    "request": {
      "children": []
    },
    "response": {
      "children": []
    }
  },
  "preFlow": {
    "name": "PreFlow",
    "request": {
      "children": [...]
    },
    "response": {
      "children": []
    }
  },
  "type": "Target"
}
7

OK Thanks. Yes I Was interested in the connection in particular.

I agree with you; the behavior you’re describing is not the behavior I would expect. I would expect Apigee to throw an error with an invalid hostname.

Can you please check this response and try what is suggested there?

https://community.apigee.com/answers/60407/view.html

See also, this doc page

https://docs.apigee.com/api-platform/system-administration/configuring-ssl-edge-backend-service

10222-screenshot-20200813-134324.png
10222-screenshot-20200813-134324.png1912×610 84.7 KB

The documentation is quite unsatisfying for 2 reasons: First, that it flatly states “Edge does not validate”. That seems to be true but is surprising (As you point out), and not really acceptable. The doc isn’t wrong on this point, apparently. It’s just not satisfying. Second, the documentation does not mention the CommonName element and what it does. I just checked the test cases, and yes, CommonName is there and is fully tested.

I think the reason the CommonName exists is… at some point in the wayback, Apigee did not validate the common name at all. And some customers began to depend on that behavior. And then the eng team introduced a way for the Apigee target endpoint to enforce CN on the peer cert, but we dropped the ball on documenting it.

1 Like
8

Thanks Dino, I had not found that “CommonName” property before.

Yes, this at least makes it fail (although error is not obvious, so that’s another issue…). I updated the targetServer definitions’ “sSLInfo” object with this:

 "commonName": {
 "value": "*.amazonaws.com",
 "wildcardMatch": true
 }

In my situation, the cert’s cname does not match this value, and now the proxy throws a ‘500 Internal Server Error’ and this pretty generic error:

{
    "fault": {
        "faultstring": "Internal Server Error",
        "detail": {
            "errorcode": "messaging.adaptors.http.flow.InternalServerError"
        }
    }
}

Its unfortunate the error is so vague and doesn’t just indicate there was a TLS error. I can see this will be a difficult error for my company’s apigee developers to troubleshoot whenever a backend hasn’t configured the TLS correctly.

9

you can set custom error, which will help your users to debug easily.

10

Mark,

yes, I see what you’re saying.

Some additional thoughts on this.

From the perspective of the client calloing into Apigee, this really is, should be, a 500 error. Apigee isn’t configured correctly for the backend connection. So that is a “server error” from the client’s perspective. The client shouldn’t see, shouldn’t be able to see, that Apigee’s TLS config isn’t correct when connecting to the upstream. So I rather think the 500 is appropriate.

On the other hand you make a good point about diagnosing such an error. Within the proxy itself, ideally Apigee emits something into the context (an error variable or something) that allows the proxy developer or the proxy operator to determine that the problem is with the TLS configuration. While I am fantasizing, we’d like it to be particularly helpful, in your case something like “CN agreement error” or something like that. I think the fault.name variable should be something like “protocol.http.CommonNameMismatch”.

And I don’t know - I haven’t tested this recently - if that is the case. Can you check? What do you see in the trace context?

11

@Priyadarshi Not every '500 Internal Server Error" is a TLS issue, so how would we write logic to detect that this was a TLS error?

12

I’m not seeing anything more in the trace except basically the same details as the json response, unless there’s some other way to see fault variables in the trace that I’m not aware of? I downloaded the XML form of the trace too and can’t see anything additional hiding in there either.

We also deal with issues where a backend team hasn’t setup IP whitelisting correctly, or maybe they haven’t registered their DNS as ‘public’, and those traces look similar.

Really just any indicator to tell the developer that theres a TLS issue would be useful. Its possible they configured the targetServer definition wrong or need to allow wildcards, etc..

13

I’m away from my desk at the moment, but when I get back, I’ll investigate using badssl.com and see if I can provide you some satisfaction on this.

1 Like
14

https://wrong.host.badssl.com/ is this exact scenario