Traffic callout extension not working on Application Load Balancer

Trying to set up a traffic callout extension to an Application Load Balancer with VM instance group backends, following this wiki: https://cloud.google.com/service-extensions/docs/configure-callout

After adding the traffic-ext policy, the load balancer always returns

HTTP/1.1 500 Internal Server Error
date: Fri, 20 Oct 2023 13:19:57 GMT
via: 1.1 google
content-length: 0

Removing this extension makes the load balancer functional again

My extension yaml:

name: traffic-ext
loadBalancingScheme: EXTERNAL_MANAGED
forwardingRules:
- https://www.googleapis.com/compute/v1/projects/<project>/regions/us-west1/forwardingRules/l7-xlb-forwarding-rule
extensionChains:
- name: "chain1"
  matchCondition:
    celExpression: 'request.method=="GET"'
  extensions:
  - name: 'ext11'
    authority: <random url>
    service: https://www.googleapis.com/compute/v1/projects/<project>/regions/us-west1/backendServices/l7-ilb-callout-service
    failOpen: false
    timeout: 1s
    supportedEvents:
    - REQUEST_HEADERS
    - REQUEST_BODY
    - RESPONSE_HEADERS
    - RESPONSE_BODY

On setting failOpen to true, the lb doesnt return 500 anymore, but still my extension application is not at all invoked

The issue was that the grpc server was not listening on a tls port. It is working fine after the connection was made secure. Client auth is not required