Running Java gae-datastore library on an F1 instance

Google Cloud Database
1

When upgrading the Java lib gae-datastore from version 2.33.2 to the latest version 3.1.0 there was a large increase in machine memory usage.

This is despite not switching to gRPC. Ie: Still using the HTTP transport:

DatastoreOptions.Builder builder = DatastoreOptions.newBuilder()
    .setTransportOptions(HttpTransportOptions.newBuilder().build());

And also telling Java to limit its memory in the app.yaml entrypoint:

entrypoint: java -Xms96m -Xmx96m -XX:+UseCompactObjectHeaders -XX:MaxMetaspaceSize=80m -XX:ReservedCodeCacheSize=40m -Xss256k -XX:+UseSerialGC -jar my-app.war

The end result is version 2.33.2 runs fine on the F1 instances, while version 3.1.0 causes the F1 instances to run out of machine memory.

Has anyone been able to run version 3.1.0 on an F1 instance? Are there any tricks to get it working?

2

I added a DebugController (code below) to try to figure out what is using all the memory in 3.1.0.

3.1.0 is using 67 MB more maching memory than 2.33.2. However, it’s not clear where that is being used, as the Java heap is only 10 MB bigger, and the Java non-heap is only 15 MB bigger, so where is the extra 42 MB being used?

Results:

Using gae-datastore 2.33.2 :

=== JVM Heap ===
  Used:      46 MB
  Committed: 92 MB
  Max:       92 MB

=== JVM Non-Heap (Metaspace, Code Cache, etc.) ===
  Used:      75 MB
  Committed: 75 MB

=== JVM Memory Pools ===
  Metaspace                                [Non-heap memory]  used=51 MB  committed=52 MB
  Tenured Gen                              [Heap memory]  used=33 MB  committed=64 MB
  Eden Space                               [Heap memory]  used=12 MB  committed=25 MB
  Survivor Space                           [Heap memory]  used=0 MB  committed=3 MB
  Compressed Class Space                   [Non-heap memory]  used=10 MB  committed=10 MB
  CodeCache                                [Non-heap memory]  used=12 MB  committed=12 MB

=== Process (OS view) ===
  Committed virtual memory: 1431 MB

=== Machine Memory ===
  Total: 384 MB
  Used:  298 MB
  Free:  85 MB

Using gae-datastore 3.1.0 :

=== JVM Heap ===
  Used:      55 MB
  Committed: 92 MB
  Max:       92 MB

=== JVM Non-Heap (Metaspace, Code Cache, etc.) ===
  Used:      90 MB
  Committed: 91 MB

=== JVM Memory Pools ===
  Metaspace                                [Non-heap memory]  used=64 MB  committed=64 MB
  Tenured Gen                              [Heap memory]  used=34 MB  committed=64 MB
  Eden Space                               [Heap memory]  used=21 MB  committed=25 MB
  Survivor Space                           [Heap memory]  used=0 MB  committed=3 MB
  Compressed Class Space                   [Non-heap memory]  used=13 MB  committed=13 MB
  CodeCache                                [Non-heap memory]  used=12 MB  committed=12 MB

=== Process (OS view) ===
  Committed virtual memory: 1441 MB

=== Machine Memory ===
  Total: 384 MB
  Used:  365 MB
  Free:  18 MB

The DebugController code that outputs the above results:

import com.sun.management.OperatingSystemMXBean;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryUsage;
import java.util.List;

@RestController
public class DebugController {

	private static final long MB = 1024L * 1024L;

	@GetMapping(value = "/debug/memory", produces = "text/plain;charset=UTF-8")
	public String getMemoryUsage() {
		StringBuilder sb = new StringBuilder();

		// --- JVM Heap & Non-Heap ---
		MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
		MemoryUsage heap = memBean.getHeapMemoryUsage();
		MemoryUsage nonHeap = memBean.getNonHeapMemoryUsage();

		sb.append("=== JVM Heap ===\n");
		sb.append(String.format("  Used:      %d MB%n", heap.getUsed() / MB));
		sb.append(String.format("  Committed: %d MB%n", heap.getCommitted() / MB));
		sb.append(String.format("  Max:       %d MB%n", heap.getMax() / MB));

		sb.append("\n=== JVM Non-Heap (Metaspace, Code Cache, etc.) ===\n");
		sb.append(String.format("  Used:      %d MB%n", nonHeap.getUsed() / MB));
		sb.append(String.format("  Committed: %d MB%n", nonHeap.getCommitted() / MB));

		// --- Memory Pools (what is using heap/non-heap) ---
		sb.append("\n=== JVM Memory Pools ===\n");
		List<MemoryPoolMXBean> pools = ManagementFactory.getMemoryPoolMXBeans();
		for (MemoryPoolMXBean pool : pools) {
			MemoryUsage usage = pool.getUsage();
			if (usage == null) continue;
			sb.append(String.format("  %-40s [%s]  used=%d MB  committed=%d MB%n",
					pool.getName(),
					pool.getType(),
					usage.getUsed() / MB,
					usage.getCommitted() / MB));
		}

		// --- OS / Process memory ---
		OperatingSystemMXBean osBean = (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
		long processCommitted = osBean.getCommittedVirtualMemorySize();
		long machineTotal = osBean.getTotalMemorySize();
		long machineFree = osBean.getFreeMemorySize();
		long machineUsed = machineTotal - machineFree;

		sb.append("\n=== Process (OS view) ===\n");
		sb.append(String.format("  Committed virtual memory: %d MB%n", processCommitted / MB));

		sb.append("\n=== Machine Memory ===\n");
		sb.append(String.format("  Total: %d MB%n", machineTotal / MB));
		sb.append(String.format("  Used:  %d MB%n", machineUsed / MB));
		sb.append(String.format("  Free:  %d MB%n", machineFree / MB));

		return sb.toString();
	}
}
3

Other findings:

1: Off heap memory allocation:

I believe gRPC uses Netty which allocates off heap memory (for performance reasons). If gRPC is used for other things (like monitoring), then this would explain the extra 42 MB of system memory that gets used with 3.1.0 that doesn’t show up in my Java DebugController.

2: Logs get spammed:

3.1.0 also seems to log a lot of errors. I get this error on start-up:

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[0] (metric.type="otel.sdk.metric_reader.collection.duration", metric.labels={"client_uid": "00037f", "service": "datastore.googleapis.com", "otel_component_type": "periodic_metric_reader", "otel_component_name": "periodic_metric_reader/0"}, resource.type="global", resource.labels={"project_id": [PII Removed by Staff]}): The metric type must be a URL-formatted string with a domain and non-empty path.

And then every 30 seconds it logs this:

i.o.s.m.e.PeriodicMetricReader - Exporter failed

3: Partial memory success:

I did have some success with 3.1.0 by limiting the number of gRPC connections:

InstantiatingGrpcChannelProvider channelProvider =
        DatastoreAdminSettings.defaultGrpcTransportProviderBuilder()
                .setChannelPoolSettings(
                        ChannelPoolSettings.builder()
                                .setInitialChannelCount(1) // Num DB connections to start with
                                .setMaxChannelCount(2) // Max num DB connections
                                .build())
                .build();
DatastoreOptions.Builder builder = DatastoreOptions.newBuilder()
        .setChannelProvider(channelProvider)
        .setTransportOptions((TransportOptions)GrpcTransportOptions.newBuilder().build());

This did reduce the memory usage, and my app ran for about 10 minutes before the F1 instance ran out of memory and restarted.

4

Figured out Open Telemetry was the thing using all the memory. Now running gae-datasore 3.2.0 with Open Telemetry switched it off:

DatastoreOptions.Builder builder = DatastoreOptions.newBuilder()
    .setTransportOptions(HttpTransportOptions.newBuilder().build())
    .setOpenTelemetryOptions(DatastoreOpenTelemetryOptions
        .newBuilder().setTracingEnabled(false).build());

And the increase in memory is gone. Actually, the memory usage has dropped from when I was running version 2.33.2! It’s now using ~20 MB less Java heap memory, and ~10 MB less Java non-heap memory. Hooray! :tada: