Getting Error Deadline Exceeded when deploying model from Cloud Firestoer functions

Google Cloud Build with AI Custom ML & MLOps
1

Hello,

I currently have a firebase function that is set to deploy my AutoML tables model everyday at 5am. This has been working fine for the past month, up until the last week. I have been getting the following error below when the function attempts to deploy the model.

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

I watched a google tutorial and it recommend to return a promise from my cloud function. That seemed to work for 1 day, but I received the error again this morning.

I am going to try to implement a retry function, but I figured I would ask on here as well. Also, I am thinking that moving from autoML to VertexAI might help alleviate my issues. Any guidance here is helpful.

See below for my deploy model code:

const deploy_model = () => {

const client = new automl.v1beta1.AutoMlClient(config);

// Get the full path of the model.
const modelFullId = client.modelPath(projectId, computeRegion, modelId);

// Deploy a model with the deploy model request.
return client.deployModel({name: modelFullId})

}

1 Like
2

Is deploying the model fails via Cloud Functions only, or does it also fail when the model is deployed manually, or by any other means?

3

Manually (through the cloud console) it usually works. I have been testing using an http call in Firebase functions. Sometimes (not consistantly) that fails if I have recently undeployed the model. Although, I do not get any type of error notification, I just know it fails by checking the cloud console.

The deploy model function only runs once a day though, and the model has typically been undeployed for at least 20 hours before that, so I don’t think I am getting that error because I am calling it too often.

4

So after further investigation it looks like the model is failing when I am requesting predictions through firebase functions. I get the same error “DEADLINE_EXCEEDED”. This is not consistent though, it worked for the previous 2 days before this and today failed again. I haven’t changed anything.

I have 2 questions:

  1. is it possible that congestion on the network is causing these to fail? Would it help if I moved the prediction to a different time? Currently I have it set at 6am PST.

  2. Since autoML is beta, would it help if I moved the model to VertexAI? I can make that move if it helps

1 Like