Facing 502 errors after switching to Google Cloud Load Balancer

Google Cloud Compute Infrastructure
1

Hello,

Previously we had a setup where Nginx was used as a load balancer. It went as follows:

Angular app in the web (Client) → Nginx in Compute Engine in public subnet, used as load balancer → Application Server in private subnet, where nginx is used to route requests to the application port based on the path.

In the nginx load balancer, we were adding headers as follows to mitigate the CORS issue.


location /app {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' *;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Content-Type,platform,ServicePlatformID,versionid';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 200;
}
proxy_pass [http://127.0.0.1:3000](http://127.0.0.1:3000);
}

In the new setup we are using Google Cloud Load Balancer as follows:

Angular app in the web (Client) -->Google Cloud Load Balancer → Application Server in private subnet, where nginx is used to route requests to the application port based on the path.

So we moved the above nginx configuration from load balancing compute engine to application server compute engine.

When we run the requests, it works fine via Postman but when run via the browser, we are constantly seeing the below error in Google CLB.

{
"insertId": "1bg7ytof0jd92",
"jsonPayload": {
"backendTargetProjectNumber": "projects/32826632432423",
"remoteIp": "124.153.49.xxx",
"statusDetails": "backend_connection_closed_before_data_sent_to_client",
"@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"cacheDecision": [
"RESPONSE_HAS_CONTENT_TYPE",
"CACHE_MODE_USE_ORIGIN_HEADERS"
]
}

We tried the below 2 solutions, but to no avail.

  1. Set Timeout and Connection Draining Timeout to the same value(in our case, 300) as per this.

  2. Set keepalive_timeout 650s; in Nginx as per this.

Since none of these solutions helped, we are now stuck. Would appreciate any help to fix this.

1 Like
2

Hi,

Based on the information you shared specifically in the error message that you are having, I recommend checking this troubleshooting documentation[1] of 5xx errors. From the documentation it will explain the “statusDetails” and “backend_connection_closed_before_data_sent_to_client” error message. There is also a recommendation for setting up the keep alive timeout value.

Also, make sure that you follow the required firewall rule[2] for health check, this is one requirement that you need to set up in your firewall when you use a health check.

[1]https://cloud.google.com/load-balancing/docs/https/troubleshooting-ext-https-lbs#troubleshoot_general_connectivity_issues
[2]https://cloud.google.com/load-balancing/docs/health-checks#firewall_rules

1 Like
3

Hi VannGuce,

Thanks for your reply.

We have already setup keepalive_timeout 650s; as mentioned in the initial post in the second last line.

Also, we are not seeing any issues with health check. It is showing a green checkmark in “Healthy” column for the backend in load balancer details page.

Despite all this, we continue to see the “backend_connection_closed_before_data_sent_to_client” issue. It only happens when sending requests via browser, not when using Postman. It has something to do with CORS and Google CLB but I’m unable to figure it out as the same nginx configuration that I posted in my initial message had no issues when nginx was used as load balancer.

1 Like
4

After lots of experimentation, I was able to resolve the issue.

In the nginx configuration


location /app {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' *;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Content-Type,platform,ServicePlatformID,versionid';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 200;
}
proxy_pass [http://127.0.0.1:3000](http://127.0.0.1:3000/);
}

Replacing return 200; with return 204; fixed the issue. I’m not sure why this is the case as return 200; works for both nginx as well as aws load balancer that we use in other applications, but after lots of trail and error, this is the conclusion I’ve reached.

1 Like
5

How to solve 502 bad gateway in GKE (I used GCE Ingress for LB)

RROR 2024-07-17T15:44:21.507751246Z [resource.labels.containerName: webapp-api] ' <h1> HTTP Error 500.30 - ANCM In-Process Start Failure </h1>\r\n' +
ERROR 2024-07-17T15:44:21.507768258Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507778380Z [resource.labels.containerName: webapp-api] ' <h2> Common solutions to this issue: </h2><ul><li>The application failed to start</li><li>The application started but then stopped</li><li>The application started but threw an exception during startup</li></ul>\r\n' +
ERROR 2024-07-17T15:44:21.507784421Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507790038Z [resource.labels.containerName: webapp-api] ' \r\n' +
ERROR 2024-07-17T15:44:21.507795816Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507802540Z [resource.labels.containerName: webapp-api] ' <h2> Troubleshooting steps: </h2>\r\n' +
ERROR 2024-07-17T15:44:21.507812459Z [resource.labels.containerName: webapp-api] ' <ul>\r\n' +
ERROR 2024-07-17T15:44:21.507818392Z [resource.labels.containerName: webapp-api] ' <li> Check the system event log for error messages </li>\r\n' +
ERROR 2024-07-17T15:44:21.507824128Z [resource.labels.containerName: webapp-api] " <li> Enable logging the application process' stdout messages </li>\r\n" +
ERROR 2024-07-17T15:44:21.507829863Z [resource.labels.containerName: webapp-api] ' <li> Attach a debugger to the application process and inspect </li>\r\n' +
_keepAliveTimeout: 0,
insecureHTTPParser: undefined,
keepAliveMsecs: 1000,
ERROR 2024-07-17T15:44:21.506707105Z [resource.labels.containerName: webapp-api] response: {
ERROR 2024-07-17T15:44:21.506713270Z [resource.labels.containerName: webapp-api] status: 500,
ERROR 2024-07-17T15:44:21.506724253Z [resource.labels.containerName: webapp-api] statusText: 'Internal Server Error',
ERROR 2024-07-17T15:44:21.506022537Z [resource.labels.containerName: webapp-api] defaultPort: 443,
ERROR 2024-07-17T15:44:21.506028401Z [resource.labels.containerName: webapp-api] protocol: 'https:',
ERROR 2024-07-17T15:42:20.861101553Z [resource.labels.containerName: webapp-api] }
ERROR 2024-07-17T15:44:21.504950127Z [resource.labels.containerName: webapp-api] Error fetching the API: AxiosError: Request failed with status code 500 at settle (/app/node_modules/axios/dist/node/axios.cjs:1983:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3085:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:4224:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async EmailService.readingSpec (/app/dist/notification/email_service.js:418:34) at async EmailService.sendPrivacyReport (/app/dist/notification/email_service.js:469:28) at async SensitiveDataService.processDocument (/app/dist/sensitive_data_service/sensitive_data.service.js:269:29) {
ERROR 2024-07-17T15:44:21.505071022Z [resource.labels.containerName: webapp-api] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-17T15:44:21.505076608Z [resource.labels.containerName: webapp-api] config: {
ERROR 2024-07-17T15:44:21.505221535Z [resource.labels.containerName: webapp-api] transitional: {
ERROR 2024-07-17T15:44:21.505230123Z [resource.labels.containerName: webapp-api] silentJSONParsing: true,
ERROR 2024-07-17T15:44:21.505235562Z [resource.labels.containerName: webapp-api] forcedJSONParsing: true,
ERROR 2024-07-17T15:44:21.505241112Z [resource.labels.containerName: webapp-api] clarifyTimeoutError: false
ERROR 2024-07-17T15:44:21.505246849Z [resource.labels.containerName: webapp-api] },
ERROR 2024-07-17T15:44:21.505253225Z [resource.labels.containerName: webapp-api] adapter: [ 'xhr', 'http', 'fetch' ],
ERROR 2024-07-17T15:44:21.505259837Z [resource.labels.containerName: webapp-api] transformRequest: [ [Function: transformRequest] ],
ERROR 2024-07-17T15:44:21.505265618Z [resource.labels.containerName: webapp-api] transformResponse: [ [Function: transformResponse] ],
ERROR 2024-07-17T15:44:21.505270953Z [resource.labels.containerName: webapp-api] timeout: 0,
ERROR 2024-07-17T15:42:20.861101553Z [resource.labels.containerName: webapp-api] }
ERROR 2024-07-17T15:39:37.055237807Z [resource.labels.containerName: perfai-engine] },
ERROR 2024-07-17T15:39:37.055243893Z [resource.labels.containerName: perfai-engine] socketPath: undefined,
ERROR 2024-07-17T15:39:37.055249982Z [resource.labels.containerName: perfai-engine] method: 'POST',
ERROR 2024-07-17T15:39:37.055257154Z [resource.labels.containerName: perfai-engine] maxHeaderSize: undefined,
ERROR 2024-07-17T15:39:37.055263321Z [resource.labels.containerName: perfai-engine] insecureHTTPParser: undefined,
ERROR 2024-07-17T15:39:37.055269423Z [resource.labels.containerName: perfai-engine] path: '/api-analyzer/fetch-api-spec',
ERROR 2024-07-17T15:39:37.055275495Z [resource.labels.containerName: perfai-engine] _ended: true,
ERROR 2024-07-17T15:39:37.055282051Z [resource.labels.containerName: perfai-engine] res: IncomingMessage {
ERROR 2024-07-17T15:39:37.055288993Z [resource.labels.containerName: perfai-engine] _readableState: [ReadableState],
ERROR 2024-07-17T15:39:37.055301607Z [resource.labels.containerName: perfai-engine] _events: [Object: null prototype],
ERROR 2024-07-17T15:39:37.055313787Z [resource.labels.containerName: perfai-engine] _eventsCount: 4,
ERROR 2024-07-17T15:39:37.055321059Z [resource.labels.containerName: perfai-engine] _maxListeners: undefined,
ERROR 2024-07-17T15:39:37.055343082Z [resource.labels.containerName: perfai-engine] socket: [TLSSocket],
ERROR 2024-07-17T15:39:37.055349871Z [resource.labels.containerName: perfai-engine] httpVersionMajor: 1,
ERROR 2024-07-17T15:39:37.055356241Z [resource.labels.containerName: perfai-engine] httpVersionMinor: 1,
ERROR 2024-07-17T15:39:37.055362507Z [resource.labels.containerName: perfai-engine] httpVersion: '1.1',
ERROR 2024-07-17T15:39:37.055369587Z [resource.labels.containerName: perfai-engine] complete: true,
ERROR 2024-07-17T15:39:37.055376143Z [resource.labels.containerName: perfai-engine] rawHeaders: [Array],
ERROR 2024-07-17T15:39:37.055387644Z [resource.labels.containerName: perfai-engine] rawTrailers: [],
ERROR 2024-07-17T15:39:37.055399576Z [resource.labels.containerName: perfai-engine] aborted: false,
ERROR 2024-07-17T15:39:37.055407889Z [resource.labels.containerName: perfai-engine] upgrade: false,
ERROR 2024-07-17T15:39:37.055414310Z [resource.labels.containerName: perfai-engine] url: '',
ERROR 2024-07-17T15:39:37.055420523Z [resource.labels.containerName: perfai-engine] method: null,
ERROR 2024-07-17T15:39:37.055426958Z [resource.labels.containerName: perfai-engine] statusCode: 502,
ERROR 2024-07-17T15:39:37.055433121Z [resource.labels.containerName: perfai-engine] statusMessage: 'Bad Gateway',
ERROR 2024-07-17T15:39:37.055439371Z [resource.labels.containerName: perfai-engine] client: [TLSSocket],
ERROR 2024-07-17T15:39:37.055445722Z [resource.labels.containerName: perfai-engine] _consuming: false,
ERROR 2024-07-17T15:39:37.055451849Z [resource.labels.containerName: perfai-engine] _dumped: false,
ERROR 2024-07-17T15:39:37.055458097Z [resource.labels.containerName: perfai-engine] req: [Circular *1],
ERROR 2024-07-17T15:39:37.055465585Z [resource.labels.containerName: perfai-engine] responseUrl: '[https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec](https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec)',
ERROR 2024-07-17T15:39:37.055529982Z [resource.labels.containerName: perfai-engine] redirects: [],
ERROR 2024-07-17T15:39:37.049334386Z [resource.labels.containerName: perfai-engine] }
ERROR 2024-07-17T15:39:37.053966315Z [resource.labels.containerName: perfai-engine] Error updating field details with AI service: [https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec](https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec) AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {
ERROR 2024-07-17T15:39:37.054107712Z [resource.labels.containerName: perfai-engine] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-17T15:39:37.054114006Z [resource.labels.containerName: perfai-engine] config: {
ERROR 2024-07-17T15:39:37.054121026Z [resource.labels.containerName: perfai-engine] transitional: {
ERROR 2024-07-17T15:39:37.054126952Z [resource.labels.containerName: perfai-engine] silentJSONParsing: true,
ERROR 2024-07-17T15:39:37.054132876Z [resource.labels.containerName: perfai-engine] forcedJSONParsing: true,
ERROR 2024-07-17T15:39:37.054138781Z [resource.labels.containerName: perfai-engine] clarifyTimeoutError: false
ERROR 2024-07-17T15:39:37.054145246Z [resource.labels.containerName: perfai-engine] },
ERROR 2024-07-17T15:39:37.054152116Z [resource.labels.containerName: perfai-engine] adapter: [ 'xhr', 'http' ],
ERROR 2024-07-17T15:39:37.054158867Z [resource.labels.containerName: perfai-engine] transformRequest: [ [Function: transformRequest] ],
ERROR 2024-07-17T15:39:37.054165172Z [resource.labels.containerName: perfai-engine] transformResponse: [ [Function: transformResponse] ],
ERROR 2024-07-17T15:39:37.054171281Z [resource.labels.containerName: perfai-engine] timeout: 0,
1 Like