Dataflow: mportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

Google Cloud Data Analytics
1

So, I have working on this dataflow project where I have receiving this error especially in a shared VPC network project, I packaged the dataflow code when running it is giving this error and I am unable to understand how to fix it.

Ran into a tricky issue recently while deploying a Python Dataflow Flex Template:

ImportError: cannot import name 'secretmanager' from 'google.cloud'

Even though google-cloud-secretmanager was installed in my Docker image and worked locally, the pipeline failed on Dataflow workers.

Root cause:

  • Dataflow Flex Templates for Python run inside a worker virtual environment (beam-venv-worker-sdk-0-0).

  • Packages installed in the Docker image system Python are not automatically visible inside the worker venv.

  • In a Shared VPC, workers cannot access PyPI at runtime, so Beam cannot install missing dependencies → ImportError.

I have tried to implement similar to this Dataflow: ModuleNotFoundError: No module named ‘src’, but with additional dependencies and no runtime installation.

2

Hi Bharath_Velamala1,

It looks like you are hitting an ImportError because Dataflow workers run in an isolated Python environment and can’t access PyPI in a Shared VPC. Even though google-cloud-secretmanager is in your Docker image, it’s not available inside the worker’s virtual environment.

Here are the potential ways that might help with your use case:

  • setup.py (or pyproject.toml): You may want to declare ‘google-cloud-secretmanager’ and other dependencies in ‘install_requires’ within your ‘setup.py’ to ensure they’re installed inside the Dataflow worker’s virtual environment, especially since Shared VPCs block runtime access to PyPI.
  • Dockerfile: You may want to begin with the official Beam Python base image, copy your full project into the container, and run ‘pip install --no-cache-dir .’ to ensure your code and dependencies are properly installed in the Dataflow worker environment.
  • Build and Push Docker Image: You may want to build your Docker image locally and then push it to Google Container Registry or Artifact Registry for deployment.
  • Deploy Flex Template: You may want to reference the pushed Docker image as the ‘containerImage’ in both your ‘metadata.json’ file and the ‘gcloud’ deployment command to ensure Dataflow uses the correct environment.

For more information, you can check the following documentation below:

1 Like
3

Thanks, Marvin!

I did I all the steps apart from passing the image via gcloud in parameters, I did that it worked now. The below process worked for me thanks.