Distributed Trace Support - Jaeger, Cloud Trace, Zipkin

Google Cloud Apigee
, , ,
1

Hi:

We’ve stumbled through awkward conversations with Support and our Account Team with respect to Distributed Trace and hope we can get some help here.

We have, with some pain, been able to get Distributed Trace fed into a Collector using exporter=JAEGER. We are logging every transaction’s Zipkin B3 Headers to Splunk. Our External Load Balancer is enabled for Logging. From my team’s Apigee-centric view, we’ve left the Target Backend enablements to the backend teams

Given that Distributed Trace is pre-GA, and with the understanding that we can get by with it (whatever it is capable of) for now on an ad-hoc basis to diagnose some issues, we have the following questions:

  1. What does Distributed Trace actually do at this time? The documentation suggests Apigee will participate in an existing trace, but we don’t find that to be the case. The Trace Spans sent by Apigee do not include the parent span of the request that our backend receives from Apigee
  2. Is there a difference between what is sent for JAEGER exporter vs. CLOUD_TRACE?
  3. Why does the JAEGER Configuration not provide the ability to inject Headers to the collector callback? This requires us to stand up an intermediate proxy collector to inject Authorization?
  4. Understanding there are no promises from Product Management, is there any broad-strokes outlook for when Distributed Tracing will no longer be considered pre-GA. This post suggests it’s been sitting in pre-GA for at least 18 months and perhaps much longer.

Thanks,
Andrew

2

I think I need the Product team to engage on this question. As you saw, this feature was launched in preview, a long while ago. We observed less “traction” and attention on the feature from customers, than we had expected. And so, as you observed, this distributed trace feature has languished.

At the same time, the distributed trace standards were evolving and shifting. Jaeger uses Open Census. But Cloud Distributed Trace doesn’t. (hence the different parameters in the policy depending on which option you use). Since the feature was launched, the industry has evolved to embrace Open Telemetry.

Regarding the trace spans, I understand what you’re saying. At the time it was released pre-GA, it worked appropriately. Since that time, ASM and the other supporting tech has evolved, and that may have caused a change in the behavior w.r.t. Spans.

As for injecting authorization… I need the Product team to engage on this question.

Maybe @gsjurseth might have a perspective.

3

Distributed tracing with Apigee is something I’ve looked into on behalf of a few customers and while I can’t directly explain the behavior, I can observe it.

First off, there is some interaction with various trace headers by the components involved.

  • GCP Load Balancers recommend using traceparent but also support x-cloud-trace-context. If these headers are not provided by the client, they get added by GCP, or perhaps by the Envoy based GLB.
  • Apigee uses x-b3-traceid and x-b3-spanid plus a few others. If these headers are not provided by the client, they get added by Apigee, perhaps by the Envoy based ingress.

I’ve been able to correlate an API call from Client → GCP GLB → Apigee X → Cloud Run using GCP Trace Explorer in Cloud Monitoring by setting specific headers.

For example, this call to an Apigee API proxy:

TRACEID=12345678901234567890123456789444
curl https://$HOST/v1/cloud-run-multi-region/internal/v1/hello \
    -H "traceparent: 00-${TRACEID}-1234567890123456-01" \
    -H "x-b3-spanid:1" \
    -H "x-b3-traceid:${TRACEID}"

Results in these headers being used:

"traceparent": "00-12345678901234567890123456789444-c18510c9d284eab4-01",
    "x-cloud-trace-context": "12345678901234567890123456789444/13944570280229006004;o=1",
    "x-b3-traceid": "12345678901234567890123456789444, 12345678901234567890123456789444",
    "x-b3-spanid": "f304954ef0c72c9a, b48778de95500d42",
    "x-b3-parentspanid": "0000000000000001",
    "x-b3-sampled": "0, 1",

I can then see the correlated trace in Cloud Trace Explorer

kurtkanaskie_0-1721243844885.png
kurtkanaskie_0-1721243844885.png982×256 32.7 KB

4

Quick follow up, through testing Jaeger “exporter” and configuring an Apigee API Proxy as the “endpoint” I’ve noticed that the trace ID used is the x-b3-traceid.

5

Inject Authorization Header with Apigee Proxy

I will answer the second question in item 3 - “This requires us to stand up an intermediate proxy collector to inject Authorization?”

Yes, you can create a Trace helper proxy in Apigee that could inject the Authorization header into the Splunk request. You can also include an API key in the query parameter of the traceConfig. I’ll outline three high-level things that you can do to secure the Apigee proxy below.

  1. Deploy the proxy to an environment that is only accessible from within your VPC network.
  2. Include an Access Control policy in the request preflow to validate the request is from an internal/external IP address.
  3. Include a Verify API Key policy in the proxy and pass the API Key in the traceConfig payload as a query parameter (see example below).

Secure the Trace Helper proxy

Create an Apigee proxy that includes a Verify API policy in the request preflow. You can pass an API key in a query parameter in the Jaeger traceConfig request payload as shown below.

"endpoint": "https://api.exco.com/jaeger?apikey=some-unique-key"
Allow Internal Requests

You can use this approach if you want your Trace helper proxy to receive requests from within your network only.

  1. Create an internal only Application Load Balancer (ALB).
  2. Use Cloud DNS to register a domain name to the Internal ALB’s private IP address.
  3. Create an Apigee Environment and Environment Group that has the ALB’s hostname.
  4. Add an Access Control policy in Apigee Trace helper proxy and include the Allow Action to allow IP Ranges from your Apigee VPC Peering range.
Allow External Requests
  1. Configure Apigee NAT addresses for Apigee runtime instances.
  2. Configure Access Control policy to Allow Apigee NAT IP addresses and Deny all other IPs.

Your API Traffic will Increase

If you decide to use an Apigee proxy, then that will increase your API call volume. Cloud Trace allows you to configure a probability sampler between .1 and .5, which means that the Distributed Tracing will randomly select a request and send that to your Trace tool. The sample rate and the amount of time Distributed Tracing is running will determine how much your API traffic will increase.

For example, if you enter .5 or 50%, then Distributed Tracing will randomly select 50% of your API traffic and send it to your Trace helper proxy. If you send 1M requests, then 50% of those requests will be sent to your Trace helper proxy, bringing your total API volume to 1.5M requests. If you are ok with increasing your API volume in Apigee, then this is the best approach. Alternatively, you could accomplish your goal of injecting the Auth header without an Apigee proxy by writing custom code and deploying it within Cloud Run or a Cloud Function.

Missing Span IDs

I copied and pasted the documentation that we have regarding the missing span IDs. This might be the reason why you see Missing Span ID in your distributed tracing tool.

“In distributed tracing systems, incomplete traces are expected. A trace is incomplete when a sampled span contains a reference to another span that hasn’t been received. The unresolved reference can occur for the following reasons:

  • The referenced span wasn’t sampled.
  • The referenced span was sampled but hasn’t yet been received by Cloud Trace or the span was received but not stored.”

See this documentation link for more information.

https://cloud.google.com/trace/docs/troubleshooting#missing-span-id

Thanks @kurtkanaskie for your help on reviewing this.

6

Thanks @williamssean for writing this up! I’ve actually tested the internal request route, so you’ve been fact checked in that regard. :wink:

Regarding using an API proxy and increased traffic count, it should be noted that using distributed tracing is not something you turn on and leave on as it can degrade performance by 20% as per: https://cloud.google.com/apigee/docs/api-platform/develop/enabling-distributed-trace#perf-considerations