Unable to connect transfer agents to pool

Google Cloud Compute Infrastructure
1

I wanted to make use of GCP Cloud Transfer Service to sync data from on-premise. For this I had to install agents and connect them to a pool. I was following the steps here, where they provide gcloud/docker commands to install a transfer agent

The agents do start but they aren’t able to connect to the pool. If I see the output of the agent container (using docker attach containerID)

0B/s txSum:      0B taskResps[copy:0 delete:0 list:0] ctrlMsgAge:10m50s (??) |

and agent.INFO logs:

Build target: //cloud/transfer/online/onprem/workers/agent:agent
Build id: <some_id>
I1222 06:47:51.288924       3 log_spam.go:51] Command line arguments:
I1222 06:47:51.288926       3 log_spam.go:53]  argv[0]: './agent'
I1222 06:47:51.288928       3 log_spam.go:53]  argv[1]: '--project-id=<project_id>'
I1222 06:47:51.288930       3 log_spam.go:53]  argv[2]: '--hostname=<hostname>'
I1222 06:47:51.288931       3 log_spam.go:53]  argv[3]: '--agent-pool=source_agent_pool'
I1222 06:47:51.288933       3 log_spam.go:53]  argv[4]: '--container-id=49be0b94bced'
I1222 06:47:51.289408       3 prodlayer.go:217] layer successfully set to NO_LAYER with source DEFAULT
I1222 06:47:53.148699       3 handler.go:45] TaskletHandler initialized to delete at most 1000 objects in parallel: 
I1222 06:47:53.148725       3 handler.go:48] TaskletHandler initialized with delete-files: 1024
I1222 06:47:53.148827       3 copy.go:145] TaskletHandler initialized with copy-files: &{0xc00073d2c0 10800000000000}
I1222 06:47:53.148860       3 handler.go:61] TaskletHandler initialized to process at most 256 list outputs in parallel: 
I1222 06:48:51.291680       3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:49:51.291017       3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:50:51.290721       3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:51:51.291057       3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:52:51.290677       3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:53:51.290445       3 cpuutilization.go:86] Last minute's CPU utilization: 0

I also checked all the troubleshooting steps here, but couldn’t find anything. In GCP logging dashboard, this was the only relevant log I found

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "{my-email-id}",
      "principalSubject": "user:{my-email-id}"
    },
    "requestMetadata": {
      "callerIp": "gce-internal-ip",
      "callerSuppliedUserAgent": "grpc-go/1.52.0-dev,gzip(gfe)",
      "requestAttributes": {
        "time": "2023-01-06T05:49:44.260233615Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "pubsub.googleapis.com",
    "methodName": "google.pubsub.v1.Subscriber.CreateSubscription",
    "authorizationInfo": [
      {
        "resource": "projects/{my-project-id}/topics/destination_agent_pool-cloud-ingest-control",
        "permission": "pubsub.topics.attachSubscription",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceName": "projects/{my-project-id}/topics/destination_agent_pool-cloud-ingest-control",
    "request": {
      "ackDeadlineSeconds": 10,
      "@type": "type.googleapis.com/google.pubsub.v1.Subscription",
      "name": "projects/{my-project-id}/subscriptions/destination_agent_pool-cloud-ingest-control-12910427026409884352",
      "topic": "projects/{my-project-id}/topics/destination_agent_pool-cloud-ingest-control"
    },
    "response": {
      "name": "projects/{my-project-id}/subscriptions/destination_agent_pool-cloud-ingest-control-12910427026409884352",
      "messageRetentionDuration": "604800s",
      "topic": "projects/{my-project-id}/topics/destination_agent_pool-cloud-ingest-control",
      "@type": "type.googleapis.com/google.pubsub.v1.Subscription",
      "pushConfig": {},
      "ackDeadlineSeconds": 10
    }
  },
  "insertId": "uiyrc6b1y",
  "resource": {
    "type": "pubsub_topic",
    "labels": {
      "project_id": "{my-project-id}",
      "topic_id": "projects/{my-project-id}/topics/destination_agent_pool-cloud-ingest-control"
    }
  },
  "timestamp": "2023-01-06T05:49:44.253549825Z",
  "severity": "NOTICE",
  "logName": "projects/{my-project-id}/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2023-01-06T05:49:47.091556587Z"
}
2

Hi @tarun360 ,

It seems that the transfer agents are running but they are not able to connect to the pool. Based on the logs you provided, it looks like the agents are initializing correctly but there is no network activity.

Here are a few things you can check:

  1. Verify that the agents can access the internet by running a simple command such as ping www.google.com from within the agent’s container.

  2. Make sure that the project ID and agent pool name used in the command to run the agent match the correct values in the GCP console.

  3. Check if there is a firewall blocking the connection. You should ensure that the necessary ports are open.

  4. Verify that the DNS resolution is working correctly within the container.

  5. Check if the agent’s version is up to date and if there are any known issues or deprecations related to the version you are using.

  6. If none of the above steps work, you can try creating a new agent pool and agent and see if that resolves the issue.

    It could also be an issue with the service account that the agent is running under, you can check that the service account has the necessary permissions to access the agent pool and the resources it needs to transfer.

You can refer to the official documentation for Cloud Transfer Service on-premises agents to install and troubleshoot issues:

  1. Installation instructions: https://cloud.google.com/storage-transfer/docs/managing-on-prem-agents

  2. Troubleshooting guide: https://cloud.google.com/storage-transfer/docs/troubleshooting-on-prem

  3. Agent pool and agent reference: https://cloud.google.com/storage-transfer/docs/reference/agent-pool-and-agent

  4. Best practices: https://cloud.google.com/storage-transfer/docs/best-practices

    You can also refer to the Cloud Transfer Service troubleshooting forum for additional help and to see if there are any known issues or solutions for the specific error message you are encountering.

Thanks