I’m attempting to start an ML job through the Cloud ML Engine API and when ever I run I get the following error:
HttpError 429 when requesting https://ml.googleapis.com/v1/projects/{project_id}/jobs?alt=json returned "Quota failure for project {project_id}. The request for 1 K80 accelerators exceeds the allowed maximum of 0 A100, 0 K80, 0 P100, 0 P4, 0 T4, 0 TPU_V2, 0 TPU_V2_POD, 0 TPU_V3, 0 TPU_V3_POD, 0 V100 accelerators.
However, I requested quota and was able to receive it by the images below (this is just for us-central1 where I make the request to, but that I also have quota for other us-x regions):
I did sign up for the free credit, which I know has restrictions on GPU usage however I did also create a billing account which I’m pretty sure means that I’m able to be billed for stuff like this (I also wouldn’t have been able to make the quota increase requests, if I was only on a free credit account):
The code I ran below that generated the error is:
eval_job_id = __get_eval_job_id(model_uuid)
eval_training_input = {
'region': 'us-central1',
'scaleTier': 'BASIC_GPU',
'masterConfig': {
'imageUri': IMAGE_URI,
},
'jobDir': model_dir,
'args': [
'--model_dir', model_dir,
'--pipeline_config_path', pipeline_config_path,
'--checkpoint_dir', model_dir,
],
}
eval_job = {
'jobId': eval_job_id,
'trainingInput': eval_training_input,
}
ml = get_ml_service()
parent = __get_parent()
eval_job_response = ml.projects().jobs().create(parent=parent, body=eval_job).execute()
My account has also been active for more than 48 hours (read somewhere that this my also restrict usage). Any suggestions of how to proceed here? I’m not sure why I don’t have any GPU capacity available to me.