Advice for multi.exec() promise not resolving after some time

[tim-kos]

We run a multi.exec() to BLPOP from a lot of different lists. We do this several times per second. After some (random) time, the multi.exec() promise stops resolving or throwing. It likely got something to do with the redis command queue, but we are unable to reproduce the error in a test environment. It happens regularly in production, though.

Code:

multiBlpop (cb) {
    const multi = this._redisClient.multi()
      for (let i = 0; i < 100; i++) {
        multi.blPop(this._listNames, this._multiBlpopTimeout)
      }

    const callbackifiedMultiExec = callbackify(multi.exec.bind(multi))

    // If after 5s the multi call did not call back we have the node-redis stalling problem
    // and need to call our callback so that the next cycle of blpops can be triggered.
    // 5 seconds is a good one here, because these calls should actually take 100ms at most.
    withTimeout(callbackifiedMultiExec, (err, replies) => {
      if (err) {
        Logger.logEvent({
          event    : 'RedisMultiBlpopErr',
          err,
        })
      }

      if (err && this._abortOnErr) {
        return cb(err)
      }
      ...
   }
}

Some example log output:

So the timeout then fires consistently after some time and no further blpops go through, even if called again. The returned error is most often not about "the queue is full", even though it happens. But why would there be a command queue not emptying out? When we restart the script that makes the redis connection, everything starts working again.

Any advice?

Environment:

• Node.js Version: 16.15.1
• Redis Server Version: 6.2.6
• Node Redis Version: 4.3.1
• Platform: Ubuntu 20.04.5 LTS

[dabroek]

@tim-kos Do you think #2293 would (help) resolve your issue?

[tim-kos]

It should help, but I am not sure if that would be exhaustive. Please notice how in the logs I also logged .isReady and .isOpen and they are both true, which is kind of unexpected with #2293 ?

[leibale]

@tim-kos wanna try [email protected]/@redis/[email protected] to see if it fixes the issue?

[leibale]

@tim-kos if the issue still persists feel free to reopen