TL;DR: LoRA fine-tuning has a several hyperparameters that interact in ways no default captures. This post shows how to hand that search to AlphaEvolve, Google DeepMind’s LLM-guided evolutionary framework, running on Google Cloud. You wrap one function in EVOLVE-BLOCK markers, and AlphaEvolve generates and tests candidate configurations, guided by evaluation loss you define on a held-out split. All the code is examples/llm_fine_tuning in the AlphaEvolve Cloud sample repo.
The problem: Which fine-tuning hyperparameters actually matter?
You want to fine-tune a small language model to call functions reliably. You reach for LoRA (Low-Rank Adaptation) because it is cheap, and then you hit the wall every practitioner hits: LoRA has a stack of knobs, and the defaults are guesses.
What rank do you use — 8, 16, 64? Should lora_alpha track the rank? Is the learning rate that worked for full fine-tuning still right for adapters? Do you trade sequence length for a bigger batch, or the other way around? Each choice interacts with the others, the search space is combinatorial, and the only honest way to compare two configurations is to train both and measure held-out loss.
This is one shape of the problem AlphaEvolve is built for: a config you can express as code, an automatable evaluation, and a scalar score to optimize. Instead of you guessing, an LLM proposes configurations, a real training run scores each one, and the scores steer the next generation of proposals.
What we’ll build: an evolutionary search over LoRA configs
The system has two halves. On the client side, AlphaEvolve evolves a single Python function that returns a hyperparameter dictionary. On the server side, a persistent Ray cluster on GKE turns each candidate dictionary into an actual LoRA fine-tuning run on a GPU and reports back the metrics.
The model being tuned is Gemma 4 E2B-IT. The dataset is NousResearch/hermes-function-calling-v1. Each evaluation runs a fixed, short training budget — 200 steps of LoRA on a bf16 model — so that candidates are comparable and cheap. A full default run evaluates 20 candidate programs with 4 evaluations in parallel, one per GPU worker.
The zoomed-out loop looks like this:
AlphaEvolve only rewrites code between two markers. Everything else in the file is fixed scaffolding it cannot touch. The seed program is deliberately tiny and it is just the decision surface:
# EVOLVE-BLOCK-START
def get_training_config():
"""Return hyperparameter configuration for LoRA fine-tuning of Gemma 4 E2B-IT."""
return {
# LoRA configuration
"lora_r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
# Optimizer and learning rate schedule
"learning_rate": 5e-5,
"lr_scheduler_type": "cosine",
"warmup_ratio": 0.03,
"weight_decay": 0.01,
"optim": "adamw_8bit",
"max_grad_norm": 1.0,
# Batch and data
"per_device_train_batch_size": 2,
"gradient_accumulation_steps": 4,
"max_seq_length": 512,
# Precision
"bf16": True,
}
# EVOLVE-BLOCK-END
The EVOLVE-BLOCK-START / EVOLVE-BLOCK-END markers are the whole trick. AlphaEvolve rewrites only get_training_config(), the evaluation harness, the constraint checks, and the training container all live outside the markers and stay constant. That separation is what makes the search safe — the LLM can propose any config it likes, but it cannot change how a config is scored.
This is the parameter surface AlphaEvolve is searching for.
| Parameter | Seed value | Range explored |
|---|---|---|
| lora_r | 16 | 4–64 |
| lora_alpha | 32 | 8–128 |
| lora_dropout | 0.05 | 0.0–0.2 |
| learning_rate | 5e-5 | 1e-5 – 1e-3 (most impactful) |
| lr_scheduler_type | cosine | cosine, linear, constant, constant_with_warmup |
| warmup_ratio | 0.03 | 0.0–0.1 |
| weight_decay | 0.01 | 0.0–0.1 |
| optim | adamw_8bit | adamw_torch, adamw_8bit, adafactor |
| max_grad_norm | 1.0 | 0.1–5.0 |
| per_device_train_batch_size | 2 | 1–8 |
| gradient_accumulation_steps | 4 | 1–16 |
| max_seq_length | 512 | 256–1024 |
| bf16 | True | keep True |
Table 1: The evolvable surface.
How the remote GPU evaluator works
The client never touches a GPU. The evaluation_function in evaluate.py serializes the candidate’s files and POSTs them to the GKE gateway, then waits for JSON metrics back:
def evaluation_function(program_candidate: dict) -> dict:
files = program_candidate.get("content", {}).get("files", [])
payload = {"files": files}
data = json.dumps(payload).encode("utf-8")
req = urllib.request.Request(EVALUATOR_URL, data=data,
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=2400) as response:
result_json = json.loads(response.read().decode("utf-8"))
return result_json.get("metrics", {}) # flat dict, e.g. {"neg_eval_loss": -0.70, ...}
The long timeout=2400 (40 minutes) is not an accident. Remote GPU evaluations take minutes, not milliseconds, so every timeout in this system is sized to outlast a full training run. Get that wrong and the controller cancels in-flight jobs while the backend is still waiting on scores to evolve the next generation.
On the server side, a persistent RayCluster runs on GKE with an autoscaling pool of NVIDIA L4 GPU workers. A lightweight Flask gateway receives each HTTP request and creates a RayJob on the cluster; the model and dataset live in a GCS bucket and are mounted into the training pod through the GCS FUSE CSI driver. Because there are 4 workers, 4 candidates train concurrently, the GPU pool autoscales up under load and back down to zero between experiments, so you pay for GPUs only while evaluations are actually running.
Every evaluation returns a score AND a reason
The evaluator reports the following metrics :
- neg_eval_loss, primary metric, higher is better
- eval_perplexity, lower is better
- train_loss, final training loss
- training_time_seconds, Wall-clock time for the 200-step run
When a candidate fails because of an out-of-memory error, an invalid config, a NaN, the evaluator returns neg_eval_loss = -100.0 plus an insight string explaining what went wrong. Those insights are not just logging. They are fed back into AlphaEvolve’s LLM so the next generation can avoid the same mistake. A config that OOMs teaches the model something about the memory budget, and the model adjusts. This feedback loop is why AlphaEvolve tends to stop proposing invalid configs after a few generations.
Seeing it in action: seed vs. evolved config
Here you can see an example of what a running loop looks like. For better monitoring, I created a custom dash powered by Gemini to look at the evolution and collect insights on its status.
After the loop finishes, the best program is written to evolved_program/program.py and its score to result.json. Here is an example of what the search might found, side by side with the seed:
| Parameter | Seed | Evolved | Change |
|---|---|---|---|
| lora_r | 16 | 64 | 4× rank — more adapter capacity |
| lora_alpha | 32 | 128 | 4× scaling (effective alpha/r stays 2) |
| learning_rate | 5e-5 | 7e-5 | higher LR for the larger, higher-capacity adapter |
| per_device_train_batch_size | 2 | 4 | doubled batch |
| max_seq_length | 512 | 256 | halved to stay under the memory budget |
| lora_dropout | 0.05 | 0.05 | unchanged |
| lr_scheduler_type | cosine | cosine | unchanged |
| optim | adamw_8bit | adamw_8bit | unchanged |
| gradient_accumulation_steps | 4 | 4 | unchanged |
Table 2: The seed configuration and the best evolved configuration. Unchanged rows are omitted after the batch/sequence trade.
Two things stand out. First, AlphaEvolve preserved the effective LoRA scale: it scaled lora_r and lora_alpha together from 16/32 to 64/128, keeping alpha/r = 2 while quadrupling raw adapter capacity. It did not blindly maximize one knob. It kept a ratio that matters. Second, it made the batch/sequence trade explicitly. It doubled the per-device batch from 2 to 4 and halved sequence length from 512 to 256, keeping the run inside the memory ceiling instead of blowing past it.
The result from result.json:
{
"metric": "neg_eval_loss",
"score": -0.7043247,
"eval_loss": 0.7043247
}
The evolved configuration reached an evaluation loss of 0.7043 on the held-out function-calling split. For a run-level view, the loop also writes report/evolution_progress.png (best-so-far neg_eval_loss over iterations) and report/score_distribution.png (a histogram of eval loss across all evaluated programs), so you can see the search converge rather than take one number on faith.
What it costs and what you need
The example is built for efficiency: the GPU pool autoscales back to zero between experiments, keeping idle costs negligible while variable expenses scale linearly with the evaluation count. A complete run typically lands between $3.40–$5.20. This total is driven by a few primary components: approximately $1.80–$3.60 for the evaluations themselves and a $1.60 gateway overhead. The supporting infrastructure consists of a persistent CPU node at ~$0.13/hour, autoscaling GPU workers at ~$1.10/hour, and minimal GCS storage fees at ~$0.02/GB/month, all running on a managed GKE cluster at no extra management cost.
Try it yourself
In the example, every stage is a make target. From the examples/llm_fine_tuning directory:
# 1. Install deps and create .env from the template
make setup
make auth
# Edit .env with your PROJECT_ID, GE_APP_ID, and HF_TOKEN
# 2. Provision GKE, the RayCluster, IAM, and (optional) monitoring
make infra
# 3. Build training + gateway images, stage model/dataset to GCS, deploy the gateway
make deploy
# 4. Confirm the cluster is healthy (Ray Dashboard at http://localhost:8265)
make ray-dashboard
# 5. Run the AlphaEvolve experiment
make run
# When you are done, tear everything down
make clean
make run uploads the seed config, starts the evolutionary search (20 candidates, 4 concurrent evaluations), and prints the top configurations when it finishes. Expect each evaluation to take about 5–10 minutes on an L4, and the whole default run to land in the single-digit-dollar range. If you enable monitoring, make monitoring-portforward opens Grafana at http://localhost:3000 for train-loss, active-job, and eval-score dashboards.
What’s next
The pattern here is not specific to LoRA. Anything you can express as a function that returns a config, scores with an automatable evaluation, and reduces to a scalar is a candidate for AlphaEvolve.
From this example, the natural next moves are:
- Widen the search. Add more parameters to the EVOLVE-BLOCK.
- Change the objective. Swap neg_eval_loss for a task metric (function-call accuracy, JSON validity) and evolve toward what you care about at inference time.
- Scale the budget. Raise MAX_PROGRAMS_EVALUATED and the worker count together. More parallel GPU workers means more candidates per wall-clock hour.
The whole point is to let AlphaEvolve do the hyperparameter tuning sweeping and see if it can surprise you. And this is just an example on how to use AlphaEvolve on Google Cloud, but AlphaEvolve can do much more such as evolving entire architectures.
If you build something with this, I would love to hear about it — find me on LinkedIn or X.
Happy building!