What is Canary Evaluation in Apigee Instances

Google Cloud Apigee
,
1

Hello,

I recently tried to detach an environment from an Apigee instance using Terraform, and encountered the following error:

Error when reading or editing InstanceAttachment: googleapi: Error 400: the resource is locked by another operation where instance [RUNTIME_NAME] is currently under maintenance by operation: [OPERATION_UUID_1]

Upon checking the long-running operations in Apigee, I found two operations in progress:

$ gcloud alpha apigee operations list --organization="[ORG_NAME]" --filter="STATE=IN_PROGRESS"
UUID: [OPERATION_UUID_1]
ORGANIZATION: [ORG_NAME]
STATE: IN_PROGRESS

UUID: [OPERATION_UUID_2]
ORGANIZATION: [ORG_NAME]
STATE: IN_PROGRESS

$ gcloud alpha apigee operations describe --organization="[ORG_NAME]" [OPERATION_UUID_1]
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  progress:
    description: Evaluating Deployment with Canary Analysis
    percentDone: 10
  state: IN_PROGRESS
  targetResourceName: organizations/[ORG_NAME]/instances/[RUNTIME_NAME]/canaryevaluations/[OPERATION_TARGETED_RESOURCE_UUID_1]
name: organizations/[ORG_NAME]/operations/[OPERATION_UUID_1]
organization: [ORG_NAME]
uuid: [OPERATION_UUID_1]

$ gcloud alpha apigee operations describe --organization="[ORG_NAME]" [OPERATION_UUID_2]
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  progress:
    description: Evaluating Deployment with Canary Analysis
    percentDone: 10
  state: IN_PROGRESS
  targetResourceName: organizations/[ORG_NAME]/instances/[RUNTIME_NAME]/canaryevaluations/[OPERATION_TARGETED_RESOURCE_UUID_2]
name: organizations/[ORG_NAME]/operations/[OPERATION_UUID_2]
organization: [ORG_NAME]
uuid: [OPERATION_UUID_2]

After some research, I found the following document which seems somewhat related:

GoogleCloudApigeeV1CanaryEvaluation: CanaryEvaluation represents the canary analysis between two versions of the runtime that is serving requests.

https://pkg.go.dev/google.golang.org/api/apigee/v1

So, I would like to ask two questions:

  1. Does Canary Evaluation occur randomly on a timeline different from user requests? Is it possible to control this, similar to Cloud SQL maintenance windows?
  2. On average, how long does a Canary Evaluation take to complete?
2

Also, this is just one example. The first log for google.cloud.apigee.v1.RuntimeService.CreateCanaryEvaluation was recorded at 14:11:34 JST, and the last operation completed at 17:21:46 JST.

A total of 48 start and end logs were recorded, meaning that 24 operations occurred during this period. Each operation typically took about an hour to finish.

During this time, no changes could be made to the Runtime Instance, such as attaching or detaching environments.