Gcloud App Deploy Failure

Hi,

I have a problem today and need some help
Suddenly i get deploy fail today when i try to redeploy my app engine application.

Here are the deploy log

starting build "e8375101-7592-43c7-8198-2d633c957529"

FETCHSOURCE
BUILD
Starting Step #0 - "fetch"
Step #0 - "fetch": Pulling image: asia.gcr.io/gae-runtimes/utilities/gcs-fetcher:base_20211020a_18_04_RC00
Step #0 - "fetch": base_20211020a_18_04_RC00: Pulling from gae-runtimes/utilities/gcs-fetcher
Step #0 - "fetch": 0d367dec8778: Pulling fs layer
Step #0 - "fetch": 0d367dec8778: Verifying Checksum
Step #0 - "fetch": 0d367dec8778: Download complete
Step #0 - "fetch": 0d367dec8778: Pull complete
Step #0 - "fetch": Digest: sha256:0f15545637c7e82e4a6464758c6236ce11a52c66c6f192e2f900a8dc32e5a72d
Step #0 - "fetch": Status: Downloaded newer image for asia.gcr.io/gae-runtimes/utilities/gcs-fetcher:base_20211020a_18_04_RC00
Step #0 - "fetch": asia.gcr.io/gae-runtimes/utilities/gcs-fetcher:base_20211020a_18_04_RC00
Step #0 - "fetch": Fetching manifest gs://staging.jktprojects-309309.appspot.com/ae/3f481ede-4835-4a49-87f7-b5b5d78d7b9d/manifest.json.
Step #0 - "fetch": Processing 153 files.
Step #0 - "fetch": ******************************************************
Step #0 - "fetch": Status:                      SUCCESS
Step #0 - "fetch": Started:                     2022-01-06T03:25:35Z
Step #0 - "fetch": Completed:                   2022-01-06T03:25:40Z
Step #0 - "fetch": Requested workers:    200
Step #0 - "fetch": Actual workers:       153
Step #0 - "fetch": Total files:          153
Step #0 - "fetch": Total retries:          7
Step #0 - "fetch": GCS timeouts:           7
Step #0 - "fetch": MiB downloaded:        16.07 MiB
Step #0 - "fetch": MiB/s throughput:       3.97 MiB/s
Step #0 - "fetch": Time for manifest:    773.54 ms
Step #0 - "fetch": Total time:             4.82 s
Step #0 - "fetch": ******************************************************
Finished Step #0 - "fetch"
Starting Step #1 - "pre-buildpack"
Step #1 - "pre-buildpack": Pulling image: asia.gcr.io/gae-runtimes/utilities/buildpack-shim:base_20211020a_18_04_RC00
Step #1 - "pre-buildpack": base_20211020a_18_04_RC00: Pulling from gae-runtimes/utilities/buildpack-shim
Step #1 - "pre-buildpack": 0c4b9d289ba2: Pulling fs layer
Step #1 - "pre-buildpack": 0c4b9d289ba2: Verifying Checksum
Step #1 - "pre-buildpack": 0c4b9d289ba2: Download complete
Step #1 - "pre-buildpack": 0c4b9d289ba2: Pull complete
Step #1 - "pre-buildpack": Digest: sha256:5eace6d5f796863ce9dd73a59919ea8e2208a85449f3f36a9ebec7a113a01c64
Step #1 - "pre-buildpack": Status: Downloaded newer image for asia.gcr.io/gae-runtimes/utilities/buildpack-shim:base_20211020a_18_04_RC00
Step #1 - "pre-buildpack": asia.gcr.io/gae-runtimes/utilities/buildpack-shim:base_20211020a_18_04_RC00
Step #1 - "pre-buildpack": Preparing directory /layers
Step #1 - "pre-buildpack": Preparing directory /workspace
Step #1 - "pre-buildpack": Preparing directory /builder/home
Step #1 - "pre-buildpack": Preparing directory /builder/outputs
Step #1 - "pre-buildpack": Preparing directory /platform/env
Step #1 - "pre-buildpack": Passing build environment variable BUILDER_OUTPUT to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GOOGLE_DEBUG to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GOOGLE_RUNTIME to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GOOGLE_LABEL_BUILDER_VERSION to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GOOGLE_LABEL_BUILDER_IMAGE to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GOOGLE_LABEL_RUN_IMAGE to buildpacks
Step #1 - "pre-buildpack": Passing build environment variable GAE_APP_ENGINE_APIS to buildpacks
Step #1 - "pre-buildpack": Retagging asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest as asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:e8375101-7592-43c7-8198-2d633c957529
Step #1 - "pre-buildpack": Checking if image asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest exists
Step #1 - "pre-buildpack": Image asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest exists
Step #1 - "pre-buildpack": DEBUG: "asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest"(asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest) took 327.079845ms
Step #1 - "pre-buildpack": Checking if image asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest is valid
Step #1 - "pre-buildpack": DEBUG: ...pull("asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest") took 298.050385ms
Step #1 - "pre-buildpack": DEBUG: ...validate("asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest") took 1.069429718s
Step #1 - "pre-buildpack": Image asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest is valid
Step #1 - "pre-buildpack": DEBUG: "asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest"(asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest) took 1.367558874s
Step #1 - "pre-buildpack": Image asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest copied to asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:e8375101-7592-43c7-8198-2d633c957529
Step #1 - "pre-buildpack": DEBUG: "asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest", "asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:e8375101-7592-43c7-8198-2d633c957529"(asia.gcr.io/jktprojects-309309/app-engine-tmp/build-cache/default/ttl-7d:latest) took 2.682900496s
Finished Step #1 - "pre-buildpack"
Starting Step #2 - "build"
Step #2 - "build": Pulling image: asia.gcr.io/gae-runtimes/buildpacks/python39/builder:python39_20211201_3_9_8_RC00
Step #2 - "build": python39_20211201_3_9_8_RC00: Pulling from gae-runtimes/buildpacks/python39/builder
Step #2 - "build": 47d17572643b: Already exists
Step #2 - "build": 00b2adca781c: Already exists
Step #2 - "build": 3c2cba919283: Already exists
Step #2 - "build": e758c727073f: Already exists
Step #2 - "build": e3f5e4049239: Already exists
Step #2 - "build": 18a600dfee2d: Already exists
Step #2 - "build": 8565ddac2d03: Already exists
Step #2 - "build": 0337c6a3249f: Already exists
Step #2 - "build": 83aea568a57e: Already exists
Step #2 - "build": a47e3adca365: Already exists
Step #2 - "build": 404c37cea534: Already exists
Step #2 - "build": f37aca58e2d9: Pulling fs layer
Step #2 - "build": 3005df9cbfa4: Pulling fs layer
Step #2 - "build": f4f341577a3f: Pulling fs layer
Step #2 - "build": 387064007954: Pulling fs layer
Step #2 - "build": d0b36474a58b: Pulling fs layer
Step #2 - "build": 57053df04365: Pulling fs layer
Step #2 - "build": 357ac2b6aba4: Pulling fs layer
Step #2 - "build": 7732bf30b92b: Pulling fs layer
Step #2 - "build": 40a157b24d43: Pulling fs layer
Step #2 - "build": 06f428a8446f: Pulling fs layer
Step #2 - "build": df2a11ebcd38: Pulling fs layer
Step #2 - "build": 92d5d81f16ae: Pulling fs layer
Step #2 - "build": 4b47ce3bc58c: Pulling fs layer
Step #2 - "build": dc382a7e4f50: Pulling fs layer
Step #2 - "build": 6f03a57bc29c: Pulling fs layer
Step #2 - "build": 8cf41ae1599c: Pulling fs layer
Step #2 - "build": a140b44b1f4f: Pulling fs layer
Step #2 - "build": 4f4fb700ef54: Pulling fs layer
Step #2 - "build": 387064007954: Waiting
Step #2 - "build": d0b36474a58b: Waiting
Step #2 - "build": 57053df04365: Waiting
Step #2 - "build": 357ac2b6aba4: Waiting
Step #2 - "build": 7732bf30b92b: Waiting
Step #2 - "build": 40a157b24d43: Waiting
Step #2 - "build": 06f428a8446f: Waiting
Step #2 - "build": df2a11ebcd38: Waiting
Step #2 - "build": 92d5d81f16ae: Waiting
Step #2 - "build": 4b47ce3bc58c: Waiting
Step #2 - "build": dc382a7e4f50: Waiting
Step #2 - "build": 6f03a57bc29c: Waiting
Step #2 - "build": 8cf41ae1599c: Waiting
Step #2 - "build": a140b44b1f4f: Waiting
Step #2 - "build": 4f4fb700ef54: Waiting
Step #2 - "build": f37aca58e2d9: Verifying Checksum
Step #2 - "build": f37aca58e2d9: Download complete
Step #2 - "build": 3005df9cbfa4: Download complete
Step #2 - "build": f37aca58e2d9: Pull complete
Step #2 - "build": d0b36474a58b: Verifying Checksum
Step #2 - "build": d0b36474a58b: Download complete
Step #2 - "build": 3005df9cbfa4: Pull complete
Step #2 - "build": 57053df04365: Verifying Checksum
Step #2 - "build": 57053df04365: Download complete
Step #2 - "build": 357ac2b6aba4: Verifying Checksum
Step #2 - "build": 357ac2b6aba4: Download complete
Step #2 - "build": 387064007954: Verifying Checksum
Step #2 - "build": 387064007954: Download complete
Step #2 - "build": 7732bf30b92b: Verifying Checksum
Step #2 - "build": 7732bf30b92b: Download complete
Step #2 - "build": 40a157b24d43: Verifying Checksum
Step #2 - "build": 40a157b24d43: Download complete
Step #2 - "build": 06f428a8446f: Verifying Checksum
Step #2 - "build": 06f428a8446f: Download complete
Step #2 - "build": df2a11ebcd38: Verifying Checksum
Step #2 - "build": df2a11ebcd38: Download complete
Step #2 - "build": f4f341577a3f: Verifying Checksum
Step #2 - "build": f4f341577a3f: Download complete
Step #2 - "build": 92d5d81f16ae: Verifying Checksum
Step #2 - "build": 92d5d81f16ae: Download complete
Step #2 - "build": 4b47ce3bc58c: Verifying Checksum
Step #2 - "build": 4b47ce3bc58c: Download complete
Step #2 - "build": dc382a7e4f50: Verifying Checksum
Step #2 - "build": dc382a7e4f50: Download complete
Step #2 - "build": 6f03a57bc29c: Verifying Checksum
Step #2 - "build": 6f03a57bc29c: Download complete
Step #2 - "build": 8cf41ae1599c: Verifying Checksum
Step #2 - "build": 8cf41ae1599c: Download complete
Step #2 - "build": a140b44b1f4f: Verifying Checksum
Step #2 - "build": a140b44b1f4f: Download complete
Step #2 - "build": 4f4fb700ef54: Verifying Checksum
Step #2 - "build": 4f4fb700ef54: Download complete
Step #2 - "build": f4f341577a3f: Pull complete
Step #2 - "build": 387064007954: Pull complete
Step #2 - "build": d0b36474a58b: Pull complete
Step #2 - "build": 57053df04365: Pull complete
Step #2 - "build": 357ac2b6aba4: Pull complete
Step #2 - "build": 7732bf30b92b: Pull complete
Step #2 - "build": 40a157b24d43: Pull complete
Step #2 - "build": 06f428a8446f: Pull complete
Step #2 - "build": df2a11ebcd38: Pull complete
Step #2 - "build": 92d5d81f16ae: Pull complete
Step #2 - "build": 4b47ce3bc58c: Pull complete
Step #2 - "build": dc382a7e4f50: Pull complete
Step #2 - "build": 6f03a57bc29c: Pull complete
Step #2 - "build": 8cf41ae1599c: Pull complete
Step #2 - "build": a140b44b1f4f: Pull complete
Step #2 - "build": 4f4fb700ef54: Pull complete
Step #2 - "build": Digest: sha256:c68f54cdd24d22de1bd982630a33fc1f16fb5cee6bccd246bc739542fea05da7
Step #2 - "build": Status: Downloaded newer image for asia.gcr.io/gae-runtimes/buildpacks/python39/builder:python39_20211201_3_9_8_RC00
Step #2 - "build": asia.gcr.io/gae-runtimes/buildpacks/python39/builder:python39_20211201_3_9_8_RC00
Step #2 - "build": ===> DETECTING
Step #2 - "build": google.python.webserver 0.9.1
Step #2 - "build": google.python.pip       0.9.2
Step #2 - "build": google.python.appengine 0.9.0
Step #2 - "build": google.utils.label      0.0.1
Step #2 - "build": ===> ANALYZING
Step #2 - "build": Previous image with name "asia.gcr.io/jktprojects-309309/app-engine-tmp/app/default/ttl-18h:3f481ede-4835-4a49-87f7-b5b5d78d7b9d" not found
Step #2 - "build": Restoring metadata for "google.python.pip:pipcache" from cache
Step #2 - "build": ===> RESTORING
Step #2 - "build": Restoring data for "google.python.pip:pipcache" from cache
Step #2 - "build": ===> BUILDING
Step #2 - "build": === Python - webserver (google.python.webserver@0.9.1) ===
Step #2 - "build": DEBUG: Adding webserver requirements.txt to the list of requirements files to install.
Step #2 - "build": === Python - pip (google.python.pip@0.9.2) ===
Step #2 - "build": DEBUG: Found requirements.txt files provided by other buildpacks: [/cnb/buildpacks/google.python.webserver/0.9.1/requirements.txt]
Step #2 - "build": --------------------------------------------------------------------------------
Step #2 - "build": Running "python3 --version"
Step #2 - "build": Python 3.9.8
Step #2 - "build": Done "python3 --version" (35.362063ms)
Step #2 - "build": DEBUG: Current dependency hash: "4b0ebf8200ad3d998cfb50aef8c1996ffeb83304c25655a0ea48a89c56c67cc3"
Step #2 - "build": DEBUG:   Cache dependency hash: ""
Step #2 - "build": DEBUG: No metadata found from a previous build, skipping cache.
Step #2 - "build": Installing application dependencies.
Step #2 - "build": DEBUG: ***** CACHE MISS: "pip"
Step #2 - "build": DEBUG: Setting environment variable PYTHONUSERBASE=/layers/google.python.pip/pip
Step #2 - "build": --------------------------------------------------------------------------------
Step #2 - "build": Running "python3 -m pip install --requirement /cnb/buildpacks/google.python.webserver/0.9.1/requirements.txt --upgrade --upgrade-strategy only-if-needed --no-warn-script-location --no-warn-conflicts --force-reinstall --no-compile --user (PIP_CACHE_DIR=/layers/google.python.pip/pipcache PIP_DISABLE_PIP_VERSION_CHECK=1)"
Step #2 - "build": Collecting gunicorn==20.0.4
Step #2 - "build":   Using cached gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Step #2 - "build": Collecting setuptools>=3.0
Step #2 - "build":   Downloading setuptools-60.3.0-py3-none-any.whl (953 kB)
Step #2 - "build": Installing collected packages: setuptools, gunicorn
Step #2 - "build": Successfully installed gunicorn-20.0.4 setuptools-60.3.0
Step #2 - "build": Done "python3 -m pip install --requirement /cnb/buildpacks/google...." (1.688326513s)
Step #2 - "build": --------------------------------------------------------------------------------
Step #2 - "build": Running "python3 -m pip install --requirement requirements.txt --upgrade --upgrade-strategy only-if-needed --no-warn-script-location --no-warn-conflicts --force-reinstall --no-compile --user (PIP_CACHE_DIR=/layers/google.python.pip/pipcache PIP_DISABLE_PIP_VERSION_CHECK=1)"
Step #2 - "build": /opt/python3.9/bin/python3: Error while finding module specification for 'pip' (AttributeError: module '__main__' has no attribute '__file__')
Step #2 - "build": Done "python3 -m pip install --requirement requirements.txt --upgr..." (39.212635ms)
Step #2 - "build": Failure: (ID: c84b3231) /opt/python3.9/bin/python3: Error while finding module specification for 'pip' (AttributeError: module '__main__' has no attribute '__file__')
Step #2 - "build": --------------------------------------------------------------------------------
Step #2 - "build": Running "mv -f /builder/outputs/output-5577006791947779410 /builder/outputs/output"
Step #2 - "build": Done "mv -f /builder/outputs/output-5577006791947779410 /builder/o..." (16.326456ms)
Step #2 - "build": ERROR: failed to build: exit status 1
Finished Step #2 - "build"
ERROR
ERROR: build step 2 "asia.gcr.io/gae-runtimes/buildpacks/python39/builder:python39_20211201_3_9_8_RC00" failed: step exited with non-zero status: 145

Thanks

Iwan

2 Likes

Hi,

Same error since this morning, investigating…
Seems to come from setuptools-60.3.0

Execution succeed with downgrading setuptools to 60.2.0
python3 -m pip install --upgrade setuptools==60.2.0

Will be corrected in next version of setuptools : https://setuptools.pypa.io/en/latest/history.html

I’ve solved it by downgrading from python 3.8 to python 3.7, which uses a different version of setuptools.
It also worked by deleting the us.artifacts bucket in the storage cloud menu and restarting the deployment. Probably a conflict in a dependency file.

How do you downgrade setuptools through the gcloud console using the app engine default instance?