I am currently investigating latency issue being faced while communicating to Postgres via GKE/VM.
Both Postgres (vCPU: 8 | Memory: 8GB) and GKE/VM (e2-highcpu-2) are hosted under the same region and zone while they connect to each other privately under the same VPC.
When I use psql via GKE/VM and run a query (instructions below to reproduce), it seems to take 8-10 ms and apparently most of the time is taken over the network as Query Insights at Postgres shows the query execution to be taking 0.15 ms.
On my local, it takes around 1 ms or less, should I expect it to be so much higher on GCP?
Does anyone know if this is expected or there is any possible misconfiguration leading to it?
Here are the commands I have tried over a pod at GKE or VM:
VM:
$ sudo apt-get install -y postgresql-client
$ PGPASSWORD=user psql --username=user --host=10.7.2.3 --port=5432
user=# \timing
user=# SELECT count(*) FROM pg_tables;
count
-------
81
(1 row)
Time: 8.771 ms
Pod at GKE:
$ apk add postgresql-client
$ PGPASSWORD=user psql --username=user --host=10.7.2.3 --port=5432
user=# \timing
user=# SELECT count(*) FROM pg_tables;
count
-------
81
(1 row)
Time: 10.055 ms