Vertex AI online prediction throwing deadline_exceeded, internal and unavailable errors

I am using ruby client “google-cloud-ai_platform” for Vertex AI online prediction but seeing intermittent exceptions in multiple regions.
I do not understand the root cause of them and how to fix. Don’t get much in error_debug_string too except the message like “Deadline exceeded”.

I found some threads talking about timeouts, and I have it set to 2 seconds, Will it retry if it doesn’t get answer within 2 seconds? should I increase it?

any pointers? Thank you in advance for suggestions to look after.