Issues with RedisCluster Connection Using ioredis

Google Cloud Database
1

Hi,

I’m experiencing some connection issues with RedisCluster.

  • I’m using Redis version 7.0 and connecting to RedisCluster with the ioredis package in a Node.js environment.
  • In my development environment, I notice frequent connection closures. I suspect this is due to inactivity, so I’ve set keepAlive to 600000.
  • In production, some pods occasionally encounter this error:
    “WRONGPASS invalid username-password pair or user is disabled.”
  • Additionally, in Cloud Functions, some instances report this error:
    “[ioredis] Unhandled error event: ClusterAllFailedError: Failed to refresh slots cache.”

Do you think the issue could be related to the Redis-Cluster configuration? Any suggestions on how to resolve this?

Thanks

code implementation-

 new Redis.Cluster(hosts, {
    scaleReads: 'all', // Send write queries to masters and read queries to masters or slaves randomly.
    redisOptions: {
        password: token,
        keepAlive: 600000, // 10 min in milliseconds
        reconnectOnError: (err) => {
            console.error('Reconnect on error:', err);
            return true;
        },
        maxRetriesPerRequest: null // Infinite retries for requests  let commands wait forever until the connection is alive again.
    },
    slotsRefreshTimeout: 5000,
    clusterRetryStrategy: (times) => this.exponentialBackoffWithJitter(times)
})
1 Like
2

When working with Redis Cluster, connection issues often fall into three main categories: frequent disconnections, authentication failures, and cluster topology errors. Understanding their causes and applying targeted solutions can significantly improve reliability.In development, persistent connection closures may stem from Redis server timeouts, client settings, or network drops.

Even if keepAlive is set to 10 minutes, Redis may still close idle connections if its timeout setting (check with CONFIG GET timeout) is lower. Adjusting this value, setting appropriate client-side timeouts (connectTimeout in ioredis), and sending periodic PINGs to keep the connection active can help prevent unwanted disconnections.Intermittent authentication failures suggest issues with passwords or access control lists (ACLs).

In Redis 7.0, ACLs require consistent user credentials across all nodes. Ensure that the correct password is used, all cluster nodes share the same ACL configuration, and credentials are securely managed (e.g., via Kubernetes Secrets). If passwords are rotated, rolling updates should be applied to avoid pods using outdated credentials.This error typically occurs due to network interruptions, cluster instability, or an outdated slots cache. Increasing slotsRefreshTimeout and implementing a retry strategy with exponential backoff and jitter can help the client recover from temporary failures. In Cloud Functions, where connection pooling can cause stale connections to be reused, it may be better to establish a new connection per invocation or use a pool optimized for serverless environments.

By fine-tuning timeouts, securing authentication, and improving retry logic, you can ensure stable Redis Cluster connections across different environments, reducing downtime and improving performance.

3 Likes
3

Hi,

Thanks for your comment. I am not using ACL; the Redis password is retrieved from the Google Auth token.

However, I still occasionally receive the error:
WRONGPASS: Invalid username-password pair or user is disabled, even though I refresh the Redis authentication every 56 minutes.

How is it possible that the initialization succeeds, and the Redis cluster provides a ready connection, but after a few hours, I start encountering these errors?

Additionally, some errors are not being caught, causing the pod to crash, despite configuring reconnect on error and setting up an error listener.

Do you have any insights on why this might be happening?

4

Intermittent authentication failures and pod crashes in a Redis Cluster setup on Google Cloud can stem from token expiration, stale connections, and unhandled errors. While initialization succeeds, the Redis client may later fail due to outdated credentials or topology changes. Understanding these issues and implementing robust solutions can enhance stability.

The “WRONGPASS: Invalid username-password pair or user is disabled” error often occurs due to expired Google Auth tokens or stale connections retaining old credentials. While the token refreshes every 56 minutes, long-lived connections may continue using outdated authentication details, leading to failures. Additionally, Redis cluster topology changes—such as node failovers—can require re-authentication, which the client may not automatically handle.

To mitigate this, authentication should be refreshed dynamically rather than only at initialization. Instead of setting the password once, each new connection or reconnection should retrieve the latest token. A reconnectOnError function can detect authentication errors and update the credentials before reconnecting. Implementing a periodic heartbeat (e.g., sending PING commands) can also ensure connections remain valid.

Despite configuring reconnectOnError and an error listener, unhandled exceptions may still crash the pod. This suggests that fatal errors, such as unhandled promise rejections or unexpected disconnections, are not properly caught. To prevent crashes, global error handlers for uncaughtException and unhandledRejection should be implemented, ensuring that any Redis-related failures are logged and addressed without terminating the process.

Furthermore, improving error handling in the Redis client is essential. Instead of allowing “WRONGPASS” errors to break the connection, an event listener should dynamically fetch a new authentication token, update the Redis configuration, and reconnect. If persistent failures occur, implementing a circuit breaker pattern can prevent excessive retries from destabilizing the application.

By ensuring active token refresh, dynamically handling authentication failures, and reinforcing error handling mechanisms, Redis connections can remain stable and resilient. These improvements will help prevent downtime, reduce pod crashes, and maintain consistent connectivity across environments.

1 Like