In the world of Big Data, the journey from writing code to running a production Spark job has traditionally been a manual, multi-step marathon. Engineers often find themselves juggling Git commands, build tools like SBT, cloud storage uploads, and complex infrastructure APIs.
We are now entering a paradigm shift: moving from Prompt Engineering (simply asking an AI for code snippets) to Agentic Workflows (tasking an AI to autonomously execute the entire lifecycle). By leveraging Google’s Agent Development Kit (ADK), Gemini 2.5 Flash, and Dataproc Serverless, we can transform an LLM into an autonomous DevOps and Data Engineering partner.
Series overview
This technical guide is structured as a two-part series:
-
Part 1: The foundation layer — Automating the Git, SBT & cloud storage lifecycle
Focus: How to eliminate local build friction, enforce Java 11 runtime compatibility, compile shaded Scala/Spark fat JARs via SBT, and reliably push binaries to Google Cloud Storage (GCS) through deterministic Python automation. -
Part 2: The intelligence & orchestration layer — Google ADK, Dataproc Serverless & Streamlit
Focus: Hooking our build pipeline into Google ADK FunctionTools, designing an operational state-machine prompt with safety confirmation gates, provisioning Dataproc Serverless Interactive Sessions, and presenting a conversational chat interface via Streamlit.
Part 1: Eliminating Spark developer friction — Automating the Git, SBT, and cloud storage lifecycle
How we turned a 20-minute, error-prone Scala build and deployment cycle into a reliable, autonomous Python pipeline ready for AI agent integration.
Note: This is Part 1 of a two-part series on building intelligent agentic workflows on Google Cloud. In Part 1, we establish the foundational build and packaging automation layer. In Part 2, we connect this engine to Google ADK (Agent Development Kit), Gemini 2.5 Flash, and Dataproc Serverless Interactive Sessions.
Introduction: The agony of the inner loop in Spark engineering
In the world of Big Data, the journey from writing code to running a production Spark job has traditionally been a manual, multi-step marathon. Engineers often find themselves juggling Git commands, build tools like SBT, cloud storage uploads, and complex infrastructure APIs.
This iterative cycle is what data engineers call the “Inner Loop”. In web development, hot module reloading makes feedback instantaneous. In big data engineering, the inner loop frequently grinds to a halt:
Here are the primary friction points:
-
Environment mismatches: Local Java versions conflict with Spark cluster runtimes (e.g., local Java 17 vs. Dataproc Java 11 bytecode version mismatches).
-
Heavy packaging overheads: Running
sbt clean compile assemblyto build shadedfat JARstakes several minutes of intensive CPU and memory consumption. -
Manual cloud navigation: Developers context-switch into the Google Cloud Console or run long, error-prone CLI commands.
-
Interactive notebook delays: Manually attaching new JARs to interactive notebook environments requires digging through nested Dataproc properties.
What if this entire pipeline could be executed programmatically, encapsulated cleanly, and handed over to an autonomous AI agent that accepts plain English instructions?
In this first part, we dissect the architecture of an automated Git-to-GCS Scala/SBT build pipeline engineered specifically to serve as an executable tool for LLM agents.
Architecture: Decoupling build and upload for agentic consumption
To make build systems safe and reliable for an AI agent to execute, tools must adhere to three design principles:
-
Deterministic execution: The script must handle environment validation (e.g. verifying Java 11) before starting expensive compile steps.
-
Clean isolation: Builds must occur in temporary, isolated directories with automatic cleanup to prevent disk bloat.
-
Standardized I/O: Every stage must output machine-parseable logs and explicit return statuses.
Step-by-step implementation
1. Enforcing Java 11 and build discovery (artifact_builder.py)
Apache Spark runtimes on Dataproc Serverless require binary compatibility with Java 11. Developers often have Java 17 or 21 set as their system default, which causes bytecode version mismatch errors (UnsupportedClassVersionError: 61.0 vs 55.0).
Our ArtifactBuilder enforces Java 11 at runtime by inspecting installed JDKs and binding JAVA_HOME directly to the execution subprocess:
import os
import subprocess
import logging
class ArtifactBuilder:
"""Builds Scala JAR artifacts from Git repositories using SBT."""
def __init__(self, repo_url: str, branch: str = "main"):
self.repo_url = repo_url
self.branch = branch
def clone_repo(self, dest_dir: str):
"""Clone the given Git repository to the destination directory."""
logging.info(f"Cloning {self.repo_url} (branch={self.branch}) into {dest_dir}")
subprocess.run(["git", "clone", "-b", self.branch, self.repo_url, dest_dir], check=True)
def find_build_dir(self, root_dir: str) -> str:
"""Recursively search for SBT build file."""
for dirpath, _, filenames in os.walk(root_dir):
if "build.sbt" in filenames:
logging.info(f"Detected SBT build file in {dirpath}")
return dirpath
raise FileNotFoundError(f"No build.sbt file found in {root_dir}")
def check_java_installed(self):
"""Ensure Java 11 is available and active in environment."""
env = os.environ.copy()
env["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home"
env["PATH"] = f"/opt/homebrew/opt/openjdk@11/bin:{env.get('PATH', '')}"
result = subprocess.run(
["java", "-version"],
check=True,
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True
)
output = result.stdout + result.stderr
if "version" in output:
version_str = output.split("version")[1].split()[0].strip('"')
major_version = int(version_str.split(".")[0])
if major_version != 11:
raise EnvironmentError(f"Java 11 required, but detected Java {major_version}")
logging.info("Java 11 verified successfully.")
def build_artifact(self, repo_dir: str) -> str:
"""Compile and assemble fat JAR."""
build_dir = self.find_build_dir(repo_dir)
self.check_java_installed()
env = os.environ.copy()
env["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home"
env["PATH"] = f"/opt/homebrew/opt/openjdk@11/bin:{env.get('PATH', '')}"
logging.info("Running: sbt clean compile assembly...")
subprocess.run(["sbt", "clean", "compile", "assembly"], cwd=build_dir, env=env, check=True)
target_dir = os.path.join(build_dir, "target")
return self._find_file(target_dir, ".jar")
def _find_file(self, directory: str, extension: str) -> str:
for root, _, files in os.walk(directory):
for f in files:
if f.endswith(extension):
return os.path.join(root, f)
raise FileNotFoundError(f"No {extension} file found in {directory}")
2. Reliable cloud storage publishing (uploader.py)
Rather than relying on shell-level gsutil or gcloud sub-shells which can fail silently on authentication edge-cases, we use the official Google Cloud Storage Python Client Library:
from google.cloud import storage
import logging
import os
class ArtifactUploader:
"""Handles secure upload of artifacts to Google Cloud Storage."""
def __init__(self):
self.client = storage.Client()
def upload_to_gcs(self, bucket_name: str, artifact_path: str, dest_path: str):
if not os.path.exists(artifact_path):
raise FileNotFoundError(f"Artifact not found: {artifact_path}")
logging.info(f"Uploading {artifact_path} → gs://{bucket_name}/{dest_path}")
bucket = self.client.bucket(bucket_name)
blob = bucket.blob(dest_path)
blob.upload_from_filename(artifact_path)
logging.info("Upload to GCS completed successfully.")
3. Backend orchestration & CLI Wrapper (run_build_and_upload.py)
Under the hood, we wrap the builder and uploader into a unified ArtifactPipeline with a deterministic CLI interface.
[NOTE] Zero Manual CLI Overhead: As developers, we never need to manually type or memorize these CLI commands. We simply provide our natural language instruction to the AI Agent in the frontend (e.g.
Build my repo on main branch and upload to my GCS bucket). The Agent’s backend autonomously translates our intent, extracts the parameters, and fires the underlying command behind the scenes:
# Executed autonomously in the background by the Agent's backend tool:
python3 artifact_pipeline/run_build_and_upload.py \
--repo "https://github.com/<username>/demo-pipeline.git" \
--branch main \
--bucket "demo-spark-sandbox-bucket" \
--dest "spark-jobs/spark-serverless-job_test.jar" \
--cleanup
Here is the terminal log captured while the agent is executing the build tool in real-time:
Backend Terminal Execution: The Google ADK Agent interacting with Gemini 2.5 Flash on Vertex AI. Notice how the agent parses user chat instructions and automatically invokes artifact_pipeline/run_build_and_upload.py in the background with the exact parameters.
Visualizing the build pipeline in action
1. Cloud storage verification
Once the uploader finishes publishing the binary, the fat JAR is immediately available in the target GCS bucket:
Google Cloud Storage Console: The compiled Spark fat JAR (gs:///spark-jobs/spark-serverless-job_test.jar - 186.8 MB) uploaded and ready to be attached to Dataproc.
2. Agent execution lifecycle in Streamlit
When hooked up to our conversational agent interface, the build and upload lifecycle is driven seamlessly through natural language:
Streamlit Chat UI: The developer supplies the Git repo, branch, bucket, and destination path. The Agent plans and executes the underlying build tools.
Streamlit Chat UI: The artifact build and GCS upload complete successfully. Following our operational state machine, the agent asks for confirmation before spinning up a serverless notebook.
Resilient error handling & diagnostics in action
A production-ready pipeline must fail gracefully. When build or environment errors occur, the agent intercepts the underlying stderr and provides human-friendly, actionable diagnostic guidance rather than crashing:
Scenario 1: Git authentication & clone errors
When a user points to a private repository or encounters authentication failures, the pipeline traps Git’s stderr (Invalid username or token / Password authentication is not supported) and presents clear instructions on SSH/token configuration:
Error Handling Example: Intercepting Git authentication errors and providing actionable remediation guidance to the user.
Scenario 2: Pipeline runtime & environment errors
If an unexpected Python environment or runtime error occurs (e.g. missing dependencies or ModuleNotFoundError), the agent cleanly isolates the error, reports the exact traceback, and offers recovery suggestions:
Error Handling Example: Gracefully reporting runtime build errors in the UI with interactive retry prompts without breaking the chat session.
Key takeaways from part 1
-
Encapsulate domain complexity: If a process takes 4 distinct steps, encapsulate them into a single idempotent command with explicit input/output contracts.
-
Build for programmatic invocation: Avoid interactive prompts inside your build scripts; every parameter should be passable via arguments or API payloads.
-
Pave the way for agents: This deterministic foundation is what allows an LLM agent to reliably trigger complex builds without hallucinating shell commands.
-
Design for failure modes: Provide clean error interception so the AI agent can diagnose issues (auth, missing files, broken dependencies) and explain them to the user.
Conclusion
Building reliable agentic systems starts from the bottom up. By replacing manual compilation steps with an isolated, Java 11-compliant Python engine, we transformed an error-prone, 20-minute development bottleneck into a deterministic building block.
Now that our Git-to-GCS pipeline is wrapped as a self-contained, idempotent tool, it is ready to serve as the executable “hands and feet” of an AI agent.
In Part 2, we connect this foundation to Google Agent Development Kit (ADK), configure Gemini 2.5 Flash, orchestrate Dataproc Serverless Interactive Sessions, and bring everything to life in a reactive Streamlit user interface.
References & further reading (part 1)
-
Full Project Codebase — GitHub Gist
Complete, runnable multi-file Gist containing all 8 source files for this series. -
Google Cloud Dataproc Serverless Documentation
Official overview and architecture guide for Dataproc Serverless Spark workloads. -
Google Cloud Storage Python Client Reference
API documentation and best practices for managing Cloud Storage buckets and blobs using Python.
Thank you for reading! Have questions? Ask me in the comments below!