Dataproc cluster creation permission issue from Composer Env

Google Cloud Data Analytics
,
1

I’m having a problem with creating dataproc clusters for my spark job using a DAG from the composer environment.
My service account has following permissions

ROLE
roles/composer.worker
roles/compute.instanceAdmin.v1
roles/compute.viewer
roles/dataproc.admin
roles/dataproc.editor
roles/dataproc.hubAgent
roles/dataproc.serviceAgent
roles/dataproc.worker
roles/dataprocrm.admin
roles/iam.serviceAccountUser
roles/metastore.admin
roles/storage.admin
roles/storage.objectAdmin
roles/storage.objectViewer
Yet the error I’m getting is 403 permission

Thanks in advance

google.api_core.exceptions.PermissionDenied: 403 Permission ‘dataproc.clusters.create’ denied on resource '//dataproc.googleapis.com/projects/can**-****n/regions/us-central1/clusters/temp-spark-cluster-20250707’ (or it may not exist). [reason: “IAM_PERMISSION_DENIED”
domain: “dataproc.googleapis.com”
metadata {
key: “resource”
value: "projects/can-******n/regions/us-central1/clusters/temp-spark-cluster-20250707"
}
metadata {
key: “permission”
value: “dataproc.clusters.create”
}
]

Thanks in advance

2

Hi @sarath_ts ,

Welcome to Google Cloud Community!

Have you checked the following possible causes?

  • Incorrect service account: Ensure that the listed roles are granted to the correct service account associated with the Composer environment. For more detailed information, you may refer to this documentation.
  • API not enabled: Confirm if the necessary APIs are enabled for the project.
  • Denial by VPC SC and Organization Policy Restrictions: Even if the correct IAM permissions have been granted, the security perimeter defined around Google resources, along with restrictions set at the organization level can still prevent access to Dataproc. Review your organization policy constraints and control perimeters that might impact Dataproc cluster creation or related services.

If the issue persists, I suggest reaching out to Google Cloud Support with detailed information and relevant screenshots of the errors you’ve encountered. This will assist them in diagnosing and resolving your issue more efficiently.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.