Google cloud build error when saving docker image

[Oka_Widhyartha]

I’m facing issue when deploying my Next js application via firebase app hosting. In cloud build console, the build finished successfully but the deployment failed and display this error

ERROR: failed to export: saving image: failed to fetch base layers: saving image with ID "sha256:b474cf9f404275f6da2666a6b8d94e12edd241bcd3f792xxx" from the docker daemon: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/images/get?names=sha256%3Ab474cf9f404275f6da2666a6b8d94e12edd241bcd3f792xxx": EOF

it’s happening in us-east4 region

[Coder_Tacina]

Same issue for me, for firebase app hosting

[Coder_Tacina]

Hey @Oka_Widhyartha ,

This thread helped me, hope it will help you fix your issue too

App Hosting Deployment Error

I have the same problem. I was just following the steps at the Firebase Doc’s get started section (https://firebase.google.com/docs/app-hosting/get-started) and made the boilerplate app with npm init @apphosting, pushed it to a git repo, then followed the web console prompts to setup the github integration and name a backend, and it magically started deploying. Then it failed on Step #2 with the same messages you got. Scrolling up in the logs, I see that it did not find a nextjs config but I’m n…

[ben-neuraforge]

Getting this as well with builds of my node app (Nuxt.js) in us-east4.

Running a retry on a previously-successful build is failing. I went through and cleared old revision histories and old artifacts thinking that maybe I was hitting some sort of storage quota, but no luck in sorting this out:

[exporter] Setting default process type ‘web’
[exporter] Saving us-east4-docker.pkg.dev/(PII Removed by Staff)/cloud-run-source-deploy/app/app-webapp:24a082d908a35769de02b0XXXXXXXXXXXXXX…
[exporter] ERROR: failed to export: saving image: failed to fetch base layers: saving image with ID “sha256:40480efff39fd8a05268b976e9806c87e77XXXXXXXXXXXXXX” from the docker daemon: error during connect: Get “http://%2Fvar%2Frun%2Fdocker.sock/v1.41/images/get?names=sha256%3A40480efff39fd8a05268b976e9806c87e77XXXXXXXXXXXXXX”: write unix @->/run/docker.sock: write: broken pipe
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 62
Finished Step #0 - “Buildpack”
ERROR
ERROR: build step 0 “gcr.io/k8s-skaffold/pack” failed: step exited with non-zero status: 1

The ERROR has actually changed recently, and it was previously looking like this:

[exporter] ERROR: failed to export: saving image: failed to fetch base layers: saving image with ID “sha256:40480efff39fd8a05268b976e9806c87e77XXXXXXXXXXXXXX” from the docker daemon: error during connect: Get “http://%2Fvar%2Frun%2Fdocker.sock/v1.41/images/get?names=sha256%3A40480efff39fd8a05268b976e9806c87e77XXXXXXXXXXXXXX”: EOF

The difference here is instead of “EOF” at the end I now have “write unix @->/run/docker.sock: write: broken pipe”.

I’ve run into issues where my build logs didn’t log down any errors, but I was able to figure out something in my app that was breaking the build and fix it. In this case I haven’t changed anything recently and it looks like this is something internal to Google’s platform they’ll have to resolve with the way they use Docker.

Pls help Google!

[kymfx]

After many hours of debugging this issue, I came up with a work around: adding --publish to pack build and removing the images: section from the Cloud Build config.

Minimal change to the official example:

  steps:
  - name: gcr.io/k8s-skaffold/pack
    entrypoint: pack
    args:
    - build
    - LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME
    - --builder
    - gcr.io/buildpacks/builder:latest
    - --network
    - cloudbuild
    - --publish                                                    # <-- add this
  # images:                                                        # <-- remove this
  # - LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME

The root cause appears to be the Docker daemon socket failing during the image export phase. Using --publish skips that phase completely because pack pushes the image directly to the registry instead of saving it to the local Docker daemon.

This works with both gcr.io/k8s-skaffold/pack and docker.io/buildpacksio/pack (the recommended replacement since the skaffold image is deprecated).

[ben-neuraforge]

Thanks! I’ll give this a try. I just let Google automatically setup all this stuff after I connected it to my Github repository when I migrated away from AWS App Runner. It took a little bit of work but I got it all setup without having to burn too much time with devops stuff (have a product and company I am building and while this is all fun I need to be coding!). The only time I had to take a look at my build YAML was last month my builds were failing for no apparent reason (“Killed” in the logs) and I figured I’d investigate the resource limits of the actual build environment – I was right, specifying machineType of E2_HIGHCPU_32 fixed it.

[ben-neuraforge]

There are a number of references to images in my yaml file and after throwing in the —publish flag I got another error about an image not being found. My guess is something changed on Google’s end since I first setup my application last July. I tried to set everything up again and decided to try and use their Developer Connect method for connecting to my repository - it forces selection of a region (east 4), and then when I try to run my builds and specify the machineType there, it now says my machineType is not available due to quota limits - all of which I’m not even close to hitting? The error directs me to contact support… which I can’t do because I’m not on a support contract with them. Fun times, wish me luck in not losing another day to this one lol.

[ben-neuraforge]

I spent most of my waking hours over the last two days on this with tedious trial and error, waiting 15m for my builds to complete after each change I made, and I’ve finally fixed the problem and isolated what is causing it for me.

My app has been working fine for months after connecting my Github repository and getting Cloud Run setup so I never had to get into the guts of this stuff until now - I am building my app full time and my goal with using Cloud Run and it’s Cloud Build / Buildpack support is to avoid getting sucked into these kind of rabbit holes.

In suspecting a change on Google’s end breaking things, I figured out how to hard-code Google’s build tools to specific hashes for versions from using their “latest” version which is the default.

There were three tools I cycled through changing versions for after finding them on gcr.io’s browser interface – the buildpack ‘builder’, the cloud builder ‘docker’, and the cloudsdktool ‘cloud-sdk’.

The issue is with the builder – if leave it at it’s standard setting to pull “:latest”, my builds fail with the EOF error or the socket error I mentioned above every single time.

But if change it in my YAML from this:

gcr.io/buildpacks/builder:latest

To this (a version from January 21st):

gcr.io/buildpacks/builder@sha256:a1a4bcf6bed942e6771d151ddee17cc679403d31b678b3efab09aa2e7ad60acc

My app builds every time.

A next step in further isolating the problem would see me snagging specific buildpack commits from the Github repository and trying to reproduce the problem locally, but I can’t see how they correlate with the actual releases and I need to get back to coding on my app anyway. I’ll be crossing my fingers this issue goes away when I inevitably have to update my cloud build yaml to run a newer buildpack as this one ages out. This could all be as simple as a caching/edge network issue with the “latest” pack version and this problem never rears it’s head again.

If someone @Google finds this helpful lmk, got a demo to put together for your Google for Startups team soon and having another internal reference would be awesome

[kymfx]

Did you try the latest official version by chance?

@kymfx

This works with both gcr.io/k8s-skaffold/pack and docker.io/buildpacksio/pack (the recommended replacement since the skaffold image is deprecated).

[ben-neuraforge]

If you meant trying the new pack you mentioned to replace k8s-skaffold/pack, I tried swapping in ‘docker.io/buildpacksio/pack’ and it throws this error: “ERROR: failed to build: failed to fetch builder image ‘gcr.io/buildpacks/builder:latest’: Error response from daemon: client version 1.52 is too new. Maximum supported API version is 1.41”

Looks like this can’t really be done if Google’s API for grabbing the images is rejecting the call.

*edit: I really need to get back to launching the app but I couldn’t help myself - this was a known issue in their issue tracker - Google Issue Tracker

FWIW, I created a bug in Google’s Issue tracker. We shall see if the gods at Google deem it worthy of exploring https://issuetracker.google.com/issues/482084655

[kymfx]

Oh yes, you have to downgrade the Docker client version used by the pack tool by adding:

env: ["DOCKER_API_VERSION=1.41"]

Updating my previous example:

steps:
  - name: gcr.io/k8s-skaffold/pack
    entrypoint: pack
    args:
    - build
    - LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME
    - --builder
    - gcr.io/buildpacks/builder:latest
    - --network
    - cloudbuild
    - --publish                                                    # <-- add
    env: ["DOCKER_API_VERSION=1.41"]                               # <-- add
  # images:                                                        # <-- remove
  # - LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME