Building an autonomous SRE incident remediation pipeline using Gemini and OpenTelemetry

Google Cloud Integration Services
,
1

Hi everyone!

I recently built a proof-of-concept for an autonomous SRE incident remediation engine (called MuseOps) for a hackathon, and I’m currently exploring how to port and evaluate this architecture using Google Cloud’s Gemini models and OpenTelemetry.

:hammer_and_wrench: What the Hackathon PoC Did

The project intercepts raw runtime error logs (like container OOMKilled events), processes them through a 4-stage deterministic pipeline:

  1. Telemetry Parsing: Isolates high-signal error signatures.
  2. Contextual LLM Reasoning: Synthesizes Root Cause Analysis (RCA) and dynamic HCL Terraform patches.
  3. AST Static Validation: Runs local safety checks (terraform fmt, init, validate) to prevent hallucinations.
  4. Dry-Run Staging: Compiles binary execution plans (tfplan) under auto_apply=False for human oversight.

:microscope: Exploring the Same with Gemini & OpenTelemetry

I am now looking at how to implement this natively within GCP ecosystems:

  • Telemetry Collection: Intercepting infrastructure error logs using custom OpenTelemetry processors in the opentelemetry-operations-collector.
  • Model Inference: Passing structured telemetry payloads to Gemini 1.5 Pro / Flash via Vertex AI to leverage its massive context window and native JSON schema output.

Has anyone experimented with triggering generative AI remediation workflows directly from OpenTelemetry collector processors or using Gemini for structural HCL/Kubernetes manifest generation? I’d love to hear your insights or feedback!