How to Filter Specific Fields for GCLB Logs in Log Sink?

Google Cloud Data Analytics
1

created a log sink in Google Cloud Platform (GCP) to export Google Cloud Load Balancer (GCLB) logs to a BigQuery dataset. Currently, all fields related to GCLB logs are being pushed to the destination table.

I only want the following fields to be pushed:

resource.labels.project_id, timestamp and httpRequest.status

Is it possible to filter logs so that only these fields are included in the sink? If yes, could you guide me on how to configure it?

Thanks in advance!

2

Anyone can help me on this pls

3

Hi @RosarioLandy ,

Welcome to Google Cloud Community!

There is no direct way to filter specific fields within a log sink’s filter. Log sinks select entire log entries; they don’t allow selective field extraction.

To get only resource.labels.project_id, timestamp, and httpRequest.status, you need to process the data after it’s in BigQuery. After the logs have been exported to BigQuery, you can use BigQuery’s capabilities (like SQL queries) to select and create a new table containing only the desired fields. This involves creating a new table from the existing one, choosing only the specified columns.

The most efficient way is to create a BigQuery view, here’s a sample query:

CREATE OR REPLACE VIEW `your-project.your-dataset.your-view-name` AS
SELECT
    resource.labels.project_id,
    timestamp,
    httpRequest.status
FROM
    `your-project.your-dataset.your-gclb-logs-table`;

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.