No Such Object ERROR message in dataProc temp bucket

Google Cloud Cloud Foundations & Onboarding
1

Hello.,

In DataProc while GoogleAPIS writing the audit logs using get method, we are facing below ERROR message.

ERROR message : No such object: dataproc-temp-asia-northeast1-xxxxx/YYYYY/spark-job-history

ERROR message : No such object: dataproc-temp-asia-northeast1-xxxxx/YYYYY/mapreduce-job-history /done_intermediate

As per splunk log analysis., below was the details.
Log Name projects/project_id/logs/cloudaudit.googleapis.com%2Fdata_access MethodName : storage.objects.get

Result of Check in GCP:
In the Cloud Storage bucket, i have checked whether existing bucket was available or not, as an result the bucket with

dataproc-temp-asia-northeast1-xxxxx/YYYYY/spark-job-history - Available "dataproc-temp-asia-northeast1-xxxxx/YYYYY/mapreduce-job-history /done_intermediate - Available

Can you please help me to resolve the issue.

2

Hi @boopatma , I ran into the same error, did you find a solution?

3

Hey,

Hope you’re keeping well.

This usually happens when the Dataproc job history server is trying to read specific object paths that haven’t been created yet or have already been cleaned up. The dataproc-temp bucket is used for ephemeral job staging, and objects like spark-job-history or mapreduce-job-history/done_intermediate may not persist after job completion.

Check the Dataproc cluster’s Job history server settings in the Cloud Console under Dataproc → Clusters → your cluster → Configuration, and ensure the history server points to a persistent bucket instead of the default temp bucket. You can set a custom spark.eventLog.dir or mapreduce.jobhistory.done-dir in cluster properties to a bucket you manage. Also confirm that the service account running your jobs has roles/storage.objectViewer on that bucket path.

Thanks and regards,
Taz