Hi everyone,
I recently enabled the DNS endpoint to access the control plane of my GKE cluster. Everything’s working smoothly, and I was able to remove the public IP from the control plane.
However, I’m running into an issue with streaming logs. When I run:
kubectl logs -f
I get a blank output. But if I fetch logs without streaming, like this:
kubectl logs --since=1m
it works just fine.
Has anyone else encountered this? Any ideas on what might be causing it?
Thanks!
Hi, @mehdi-relevanc .
Have you tried using this command (kubectl logs -f <pod-name> --v=9) to get more verbose log output and identify where it might be hanging or failing? If not, could you please try this? If it doesn’t reveal the root cause, feel free to share the output here to look further.
Regards,
Mokit
Here are the logs. When I perform this command nothing happens and it seems to hang
klf external-dns-5497678c79-c46g6 -v=9
I0425 09:06:13.222376 23799 loader.go:402] Config loaded from file: /Users/mehdi.mahfoudi/.kube/config
I0425 09:06:13.222566 23799 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0425 09:06:13.222574 23799 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0425 09:06:13.222577 23799 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0425 09:06:13.222579 23799 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0425 09:06:13.225330 23799 helper.go:105] "Request Body" body=""
I0425 09:06:13.225370 23799 round_trippers.go:473] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.32.3 (darwin/arm64) kubernetes/32cc146" 'https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6'
I0425 09:06:13.235689 23799 round_trippers.go:502] HTTP Trace: DNS Lookup for gke-xxxx.europe-west1.gke.goog resolved to [{2001:4860:4802:32::1b } {216.239.32.27 }]
I0425 09:06:13.237622 23799 round_trippers.go:517] HTTP Trace: Dial to tcp:[2001:4860:4802:32::1b]:443 succeed
I0425 09:06:13.394419 23799 round_trippers.go:560] GET https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6 200 OK in 169 milliseconds
I0425 09:06:13.394475 23799 round_trippers.go:577] HTTP Statistics: DNSLookup 1 ms Dial 1 ms TLSHandshake 19 ms ServerProcessing 136 ms Duration 169 ms
I0425 09:06:13.394485 23799 round_trippers.go:584] Response Headers:
I0425 09:06:13.394496 23799 round_trippers.go:587] Content-Length: 6590
I0425 09:06:13.394503 23799 round_trippers.go:587] Audit-Id: 1e8499b1-7422-4d4f-aac2-103aac08c886
I0425 09:06:13.394507 23799 round_trippers.go:587] Cache-Control: no-cache, private
I0425 09:06:13.394510 23799 round_trippers.go:587] Date: Fri, 25 Apr 2025 07:06:13 GMT
I0425 09:06:13.394513 23799 round_trippers.go:587] X-Content-Type-Options: nosniff
I0425 09:06:13.394516 23799 round_trippers.go:587] X-Directpath-Agent-Latency: 25
I0425 09:06:13.394518 23799 round_trippers.go:587] Content-Type: application/json
I0425 09:06:13.394533 23799 round_trippers.go:587] X-Frame-Options: SAMEORIGIN
I0425 09:06:13.394536 23799 round_trippers.go:587] X-Gkfe-Proxy-Success: true
I0425 09:06:13.394538 23799 round_trippers.go:587] X-Kubernetes-Pf-Flowschema-Uid: 48ce012f-cbb0-444d-9e02-fbb92a55be19
I0425 09:06:13.394541 23799 round_trippers.go:587] X-Kubernetes-Pf-Prioritylevel-Uid: c4863bdc-863c-416a-9d92-09481cad2b6b
I0425 09:06:13.394544 23799 round_trippers.go:587] X-Xss-Protection: 0
I0425 09:06:13.394779 23799 helper.go:105] "Response Body" body=<
{"kind":"Pod","apiVersion":"v1","metadata":{"name":"external-dns-5497678c79-c46g6","generateName":"external-dns-5497678c79-","namespace":"external-dns","uid":"39a5cf77-5ea7-41b2-861d-6b4bad2e8a1f","resourceVersion":"308901905","creationTimestamp":"2025-04-23T06:52:49Z","labels":{"app.kubernetes.io/instance":"external-dns","app.kubernetes.io/name":"external-dns","pod-template-hash":"5497678c79"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"external-dns-5497678c79","uid":"42ad3882-37c5-4b2c-a7ca-b670dcdc8913","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2025-04-23T06:52:49Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/name":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"42ad3882-37c5-4b2c-a7ca-b670dcdc8913\"}":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"external-dns\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":7979,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:privileged":{},"f:readOnlyRootFilesystem":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:fsGroup":{},"f:runAsNonRoot":{},"f:seccompProfile":{".":{},"f:type":{}}},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2025-04-23T06:53:09Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodReadyToStartContainers\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:hostIPs":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.1.2.11\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]},"spec":{"volumes":[{"name":"kube-api-access-rx59n","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"external-dns","image":"registry.k8s.io/external-dns/external-dns:v0.16.1","args":["--log-level=info","--log-format=json","--interval=1m","--source=service","--source=ingress","--policy=upsert-only","--registry=txt","--provider=google"],"ports":[{"name":"http","containerPort":7979,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"kube-api-access-rx59n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":"http","scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":2},"readinessProbe":{"httpGet":{"path":"/healthz","port":"http","scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":6},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":65532,"runAsGroup":65532,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"external-dns","serviceAccount":"external-dns","automountServiceAccountToken":true,"nodeName":"gke-rc-infra-p-w7ld-k8s-euwe1-in-main-ac586e46-p4eo","securityContext":{"runAsNonRoot":true,"fsGroup":65534,"seccompProfile":{"type":"RuntimeDefault"}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:56Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:49Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:53:09Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:53:09Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:49Z"}],"hostIP":"192.168.192.91","hostIPs":[{"ip":"192.168.192.91"}],"podIP":"10.1.2.11","podIPs":[{"ip":"10.1.2.11"}],"startTime":"2025-04-23T06:52:49Z","containerStatuses":[{"name":"external-dns","state":{"running":{"startedAt":"2025-04-23T06:52:55Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"registry.k8s.io/external-dns/external-dns:v0.16.1","imageID":"registry.k8s.io/external-dns/external-dns@sha256:37d3a7a05c4638b8177382b80a627c223bd84a53c1a91be137245bd3cfdf9986","containerID":"containerd://cc9913993979b5373b52d38bde24e73edb993a29a438d200f0c47e028f047252","started":true,"volumeMounts":[{"name":"kube-api-access-rx59n","mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","readOnly":true,"recursiveReadOnly":"Disabled"}]}],"qosClass":"BestEffort"}}
>
I0425 09:06:13.398564 23799 logs.go:469] "Request Body" body=""
I0425 09:06:13.398672 23799 round_trippers.go:473] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.32.3 (darwin/arm64) kubernetes/32cc146" 'https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6/log?container=external-dns&follow=true'
I did try with cloud shell and it works great, I believe it is because I have Netskope on my macbook.
SO the issue was only on your Mac?
I believe so 