Hi everyone,
We’re hitting a recurring issue with a PyPI remote repository in Artifact Registry and would love to hear how others have dealt with it.
Our setup is a virtual repository that aggregates three upstreams: one standard repository for our own private packages, one standard repository for company packages, and one remote repository that proxies and caches public PyPI. The expectation was that once a package version is cached, our CI installs (using uv, region europe-west1) are served entirely from Artifact Registry and never depend on public PyPI again.
In practice, our pipelines regularly fail with 404 Not Found on wheels that should be available.
error: Failed to fetch:
https://europe-west1-python.pkg.dev/<project>/<virtual-repo>/pyasn1/pyasn1-0.6.3-py3-none-any.whl
Caused by: HTTP status client error (404 Not Found) for urlhttps://europe-west1-python.pkg.dev/<project>/<virtual-repo>/pyasn1/pyasn1-0.6.3-py3-none-any.whl
The important part: this is not a package that’s missing or newly published. The wheel has been in the remote repository since July 9, and we run several builds a day that have pulled it successfully many times since then. So the 404 is intermittent on a long-cached artifact and not a first-time
“not mirrored yet” race.
At the same time we regularly breach the “Public upstream host reads per organization, per region, per minute” quota. We opened a support case, and
Google confirmed the underlying mechanism. Quoting their reply:
Why does the mirror call PyPI even for seemingly cached packages?
Your expectation that cached package files like .whl are served directly from the Artifact Registry remote repository is correct. However, the remote repository still interacts with the upstream (PyPI) for a few key reasons:
* Index and Metadata Updates: Tools like pip need to know what package versions are available. They do this by fetching index files from the repository. Artifact Registry caches these index files but refreshes them periodically to ensure users can access the latest packages published to PyPI. As noted in the [1], Python index files are typically refreshed from the upstream approximately every hour. Each of these refresh attempts consumes the Public upstream host reads quota.* New or Uncached Packages: Any request for a package or version not currently in the remote repository’s cache will trigger a fetch from PyPI, consuming the quota.
With multiple CI/CD pipelines running, these periodic index refreshes can add up, especially if the pipelines start around the same time, leading to spikes in requests to PyPI and potentially exceeding the organization-level quota.
On our side we’ve already tried a couple of things. We added uv caching on the GitHub Actions runners, and we built a retry that specifically catches these transient GAR mirror 404s and retries the install.
Neither fully fixed it, the 404s still show up whenever enough pipelines overlap. Honestly, the runner-side uv cache feels like a workaround for the fact that our caching remote repository isn’t actually shielding us from the upstream, which doesn’t sit right with us.
Raising the quota isn’t really an option either, the GCP support has written, increases for this specific quota require a strong business justification, so we’d rather cut down the number of upstream calls than try to lift the ceiling.
So the questions we keep coming back to:
- Why would an already-cached wheel (cached for over a week, pulled successfully many times) intermittently return a 404 from the mirror?
- Is this a side effect of the hourly index refresh or of exceeding the upstream-reads quota mid-refresh?
- Is there any way to control or extend the refresh interval,
or to keep concurrent CI pipelines from all triggering upstream index refreshes at the same moment? - And has anyone landed on a clean architecture staggering pipeline starts, a shared warm cache, pinning/hashing so resolves don’t hit a fresh index, or something else — that keeps installs fully served from Artifact Registry in practice?
Any real-world experiences or pointers would be hugely appreciated. Happy to share more config or logs.
Thanks