Impact of Adding PyPi Packages on Cloud Composer Scheduler Stability and DAG Execution

Google Cloud Data Analytics
,
1

Currently we are facing issues in Cloud Composer. While adding new PyPi packages in Composer, it’s updating Composer to add these packages. But the main issue is that it’s restarting a scheduler that is impacting running dags and queuing dags. After restarting a schedule, it goes into a fail state. We have tried increasing spec as suggested some solution in the community. Still it is giving the same issue.

2

Hi @Nikita_G ,

It sounds like you’re facing some challenges with Cloud Composer. When you update PyPi packages or make other changes, Cloud Composer often restarts the scheduler to apply these updates. This can disrupt running DAGs and potentially lead to failed states.

One common issue you might encounter is dependency conflicts. Adding a new custom package could introduce conflicts with existing packages or those pre-installed in your environment, which might cause errors during the environment update process.

Here are the considerations before installing Python dependencies in Cloud Composer:

  • Permissions: Ensure you have the appropriate role to trigger environment updates and that the service account has sufficient permissions. Refer to Access Control for details.
  • VPC Service Controls: For environments protected by VPC Service Controls, grant additional user identities access to protected services and enable support for a private PyPI repository.
  • Package Requirements: Specify package requirements using the PEP-508 format, including lowercase names and optional extras and version specifiers.
  • Docker Images: PyPI dependency updates create Docker images in Artifact Registry. Avoid modifying or deleting these images or the repository. Create and manage your own repository if needed. The repository is deleted with the environment.
  • Handling Conflicts: If an update fails due to dependency conflicts, your environment will continue with existing dependencies. Successful updates will allow you to use the new Python dependencies in your DAGs.

You may also view the Cloud Composer environments’ PyPI package management for reference.

I hope the above information is helpful.

1 Like