Production Patterns: Exporting Apigee Logs and Analytics

Google Cloud Apigee Knowledge Hub
,
1

Acknowledgement: This article’s author is Maria Sofia Almeida.

In today’s API-driven world, understanding the performance, usage, and operational health of your APIs is paramount. Apigee offers robust analytics, but for a truly comprehensive view and deeper analysis, exporting this valuable data to powerful platforms like BigQuery and Cloud Storage becomes indispensable.

The provided sources in this article highlight a clear pathway to achieve this, transforming raw data into actionable insights for improved API management and business intelligence. This article will cover the following topics:

  1. Apigee Analytics Export process

  2. Custom Apigee Logs Export process (via Log Sinks)

  3. Log-based Alerting

Apigee Analytics: Beyond the Dashboard for Strategic Insights

Apigee API Analytics, once enabled as an add-on, provides a treasure trove of information about your APIs. Apigee Analytics provides metrics such as:

  • Response time

  • Request latency

  • Request size

  • Target errors

  • API product name

  • Developer email address or AppGroup Id

  • App name

  • Transaction revenue

The full list of available metrics can be found here. Additionally, any custom analytics data you capture via a Data Capture policy is also included in the Apigee Analytics data collection.

As Apigee Analytics already makes available a wide range of useful data, you might ask yourself “Why go beyond the intuitive Apigee Analytics UI?”, and there are several compelling reasons to go beyond Apigee’s standard analytics dashboards. The primary benefits are:

  • The ability to perform complex querying that is not possible with the standard user interface. By exporting Apigee data, you can seamlessly integrate it with other business intelligence (BI) tools and combine your API analytics with other business data sources. This allows you to gain deeper insights into how API usage correlates with overall business performance, empowering stakeholders across your organization.

  • If you require a longer retention period for the analytics data for compliance or long-term analysis, you can configure data export jobs to export your analytics data to a service like BigQuery or Cloud Storage, where you can store it indefinitely. The data retention period for Apigee Analytics data is 14 months.

In order to execute these extra steps to your analysis processes, exporting this data is the first step. To manage data exports and run export jobs, a user must have both Apigee Analytics Editor and Apigee Org Administrator permissions. The data export tasks run asynchronously in the background, allowing data to be exported in two common formats: CSV (comma-delimited by default) or JSON (newline-delimited by default), and the data can be sent to the following destinations:

  • Google Cloud Storage (GCS), typically used for archival purposes.

  • Cloud BigQuery, leveraged as a powerful data warehousing and analysis solution.

To facilitate these exports, Apigee mandates specific permissions for the Apigee Service Agent service account which vary depending on the chosen destination (including BigQuery Job User, BigQuery Data Editor, and Storage Admin), to ensure secure and controlled data transfers. Alternatively, if these roles are too broad, you can also create a custom role and add the following permissions, as needed:

  • bigquery.jobs.create

  • storage.objects.create

  • storage.objects.delete

  • storage.objects.list

While exports are limited to a 24 hour range and a quota of 15 calls per day per organisation, these measures are in place to manage the cost of these powerful and potentially expensive data operations.

Then, the Apigee Analytics export process requires the following steps:

  1. Configure the destination

    1. BigQuery dataset (must be configured carefully as it has location restrictions)

    2. Cloud Storage bucket (no specific location restrictions)

  2. Ensure the BigQuery API is enabled on the GCP project (when the destination is GCS since Apigee uses the BigQuery API to leverage BigQuery export features when exporting to Cloud Storage)

  3. Ensure the Apigee Service Agent service account has the required permissions (depending on the intended destination)

  4. Test the datastore configuration (ensures the datastore has all required permissions)

  5. Create a datastore

  6. Export the analytics data

Configuration samples, covering process steps 4 through 6 (which follow the initial prerequisites, steps 1–3), are available in this repository :up_right_arrow:.

The export process covered above relates to Apigee X and Hybrid and the full description of the export process can be found here :up_right_arrow:.

Apigee Edge and OPDK possess similar processes available for the export of Analytics, which differ from the one described here (e.g. the Management API used to run the export jobs is different from the one used in Apigee X/Hybrid). The process for Apigee Edge and OPDK is further described in this link :up_right_arrow:.

Apigee Analytics Export Benefits

  • Comprehensive Performance Metrics: Gain detailed insights into critical performance indicators such as request/response processing latency, request/response size, and total response time.

  • Traffic and Usage Patterns: Analyse traffic to understand peak usage times, geographical distribution, and overall API adoption.

  • Optimising Resource Utilisation: Examine cache details to ensure caching policies are effectively reducing backend load and improving response times.

  • Business Value Measurement: Track monetisation metrics to directly link API usage to business outcomes, crucial for API product managers and business stakeholders.

  • Customisable Data Collection: Leverage custom analytics with DataCapture to track specific business events or unique API parameters, tailoring insights to your exact needs.

  • Flexible Data Destinations for Advanced Analysis: Exporting data to Cloud Storage and BigQuery unlocks the potential for advanced data warehousing, complex queries, and machine learning initiatives. Data can be exported in CSV or JSON formats, catering to various analytical tools and workflows.

  • Seamless Integration with Looker Studio: A significant benefit of Apigee Analytics is its ability to integrate with Looker Studio. This allows for the creation of rich, interactive dashboards and reports, bridging the gap between raw data and understandable visualizations. The integration empowers stakeholders across the organization with clear and actionable insights.

    • Looker Studio provides two pre-built templates for Apigee, one focused on Analytics and the other on Monetization. These templates allow you to easily integrate data from a specific Apigee project. To use them, the user executing the application must have been granted the apigee.analyticsEditor role.

    • Looker Studio is free for individual use and sharing, however, access to more advanced enterprise features and management capabilities requires a subscription to Looker Studio Pro.

Custom Proxy Logs: Granular Operational Visibility

Beyond the aggregated analytics that contain general Apigee metrics, Apigee proxies can also generate custom logs to Cloud Logging. This allows you to capture data that standard metrics may not cover, such as payload information, which can be essential for more detailed analysis and troubleshooting. Exporting these custom logs provides a crucial layer of granular operational visibility, allowing organisations to capture specific event details or debug complex issues, according to their requirements.

Cloud Logging serves as the central hub for all logs in Google Cloud, offering a scalable and robust platform for log management. The primary mechanism for exporting custom logs from Apigee are Log Sinks, which are configurations that route a filtered set of logs to a chosen destination. Log sinks can be created and managed through the Google Cloud console, the Cloud Logging API, or the Google Cloud CLI.

Log sinks offer flexible routing to several Google Cloud destinations for analysis and storage:

  • BigQuery: Best for enabling advanced analytics and joining custom proxy log data with other business data. The BigQuery dataset must be write-enabled, linked datasets are not supported, and tables can be date partitioned.

  • Google Cloud Storage: Ideal for long-term, cost-effective archival. Logs are processed hourly and stored in easily accessible JSON format.

  • Other Google Cloud Projects or Log Buckets: Used for centralized log aggregation or routing data across organizational boundaries.

When configuring log sinks, it’s important to know some key technical constraints, amongst others:

  • To successfully create a log sink, the user account must have the Logs Configuration Writer IAM role.

  • There’s a limit of 200 sinks per Google Cloud project.

  • Log sink filters are highly flexible and can target specific criteria like resource.type, severity, or any other log field. However, be aware that the length of a single filter is limited to 20,000 characters, and you can only create up to 50 exclusion filters per sink.

When emitting custom logs from your Apigee proxies, the PostClientFlow is the recommended flow phase. This stage executes after the proxy has sent its response to the client, ensuring that any logging logic does not impact the API’s perceived latency. Some of the policies allowed in this flow which can address logging requirements are:

  • MessageLogging policy: This policy allows you to log custom messages to Cloud Logging or syslog.

  • ServiceCallout policy: This policy enables a proxy to make a call to an external service. In this context, it can be used to invoke the Cloud Logging API directly, offering a more flexible way to publish custom log data (allowing for the publishing of multiple log entries with a single API call).

A sample of a proxy emitting custom logs can be found in this repository :up_right_arrow:.

Therefore, any log data that needs to be emitted should be prepared during the request flow and then published to Cloud Logging using one of the allowed policies in the PostClientFlow.

Then, the custom log proxy export process requires the following steps:

  1. Have a proxy producing custom logs (e.g. via MessageLogging)

  2. Deploy the proxy with a dedicated Service Account with LogWriter permission so logs are successfully written to Cloud Logging

  3. Create the destination location (e.g. BQ dataset, GCS bucket)

  4. Select a filter that extracts the desired custom logs by configuring inclusion and exclusion filters

  5. Create a log sink to the configured destination

  6. Ensure the sink’s writerIdentity (registered in the log sink’s details) has the required permissions (e.g. BigQuery DataEditor permission for a BQ destination)

  7. Validate the destination contains the expected logs

    1. BQ table is created with all filtered logs in near realtime

    2. JSON files are created hourly in GCS with all filtered logs

Apigee Custom Logs Export Benefits

Key Insights and Benefits from Custom Proxy Log Exports:

  • Tailored Log Collection: Proxies can be configured to produce custom logs, capturing precisely the information relevant to your specific API logic or business processes, not covered by standard analytics.

  • Powerful Log Sinks: Google Cloud’s Log Sinks mechanism is the backbone of this capability, allowing you to route these custom log entries to various destinations, including BigQuery datasets and Cloud Storage buckets.

  • Precision Filtering: Inclusion and exclusion filters can be applied when creating log sinks, ensuring that only the desired custom logs are exported, reducing noise and focusing on critical data.

  • Structured Data in BigQuery: Routing to BigQuery means your custom logs are stored in a structured, queryable format, enabling rapid ad-hoc analysis and complex data correlations. It is crucial that the BigQuery dataset is write-enabled.

  • Archival and Event-driven Processing with Cloud Storage: Exporting to Cloud Storage provides a cost-effective solution for long-term archival and can trigger event-driven processing for further analysis or integration with other systems. JSON files are created hourly for these exports, providing fresh data regularly.

  • Secure and Controlled Export: Deploying proxies with a dedicated Service Account possessing LogWriter permission, along with ensuring the log sink’s writerIdentity has BigQuery DataEditor or Storage DataEditor permissions, guarantees secure and auditable log exports.

Actionable Alerts from Logs: Turning Data into Immediate Action

The insights gleaned from both Apigee Analytics and custom Apigee proxy logs are further amplified by the ability to configure alerting based on logs. This crucial capability transforms reactive analysis into proactive intervention.

By setting up log-based alerts, organisations can be immediately notified of specific patterns or thresholds, allowing for rapid response to potential issues, security breaches, or critical business events, thereby minimising downtime and impact.

In conclusion, the comprehensive strategies for exporting Apigee Analytics and custom proxy logs to BigQuery and Cloud Storage, coupled with powerful visualisation and alerting capabilities, provide organisations with a robust analysis framework. This framework enables not only for a deep analysis of API performance and usage but also proactive operational management, ensuring that APIs remain performant, secure, and aligned with business objectives.

:thought_balloon: Get in touch with a Google Cloud Sales Specialist

References

Export Apigee Analytics to BigQuery and GCS: https://cloud.google.com/apigee/docs/api-platform/analytics/export-data

https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/Flexible-Apigee-Analytics-in-BigQuery-and-Data-Studio/ta-p/169918

Exported Analytics data: https://cloud.google.com/apigee/docs/api-platform/analytics/analytics-reference

Apigee Analytics Limitations: https://cloud.google.com/apigee/docs/api-platform/reference/limits#analytics-apis

Display Analytics data in Looker Studio: https://cloud.google.com/apigee/docs/api-platform/analytics/looker

https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/Apigee-Analytics-in-BigQuery-for-Fun-and-Profit/ta-p/176812

https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/How-to-analyze-and-report-on-Apigee-API-data-with-Looker-Studio/ta-p/841268

Looker Studio Apigee templates: https://cloud.google.com/looker/docs/studio/connect-to-apigee

Looker Studio roles and IAM:

https://cloud.google.com/looker/docs/looker-core-access-control

https://cloud.google.com/iam/docs/roles-permissions/looker

https://cloud.google.com/looker/docs/studio/ways-to-share-your-reports

Routing GCP logs to supported destinations:

https://cloud.google.com/logging/docs/export/configure_export_v2

https://cloud.google.com/logging/docs/routing/overview

https://cloud.google.com/logging/docs/export/aggregated_sinks

https://cloud.google.com/logging/docs/routing/user-managed-service-accounts

https://www.youtube.com/watch?v=3Wtbde9fB_Y

https://www.youtube.com/watch?v=vcZN4FvY9bs

Alerting based on logs

https://cloud.google.com/logging/docs/alerting/log-based-alerts

https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts

3 Likes