Test execution hangs or crashes in e2e

[codiophile]

Is there an existing issue for this?

• I have searched the existing issues

Current behavior

As pointed out in this issue: #2168, the default behaviour when testing an app that uses BullMQ, is that it doesn't complete, due to the Redis connection still being open.

However, awaiting app.close() results in the test execution crashing and exiting with a non-zero exit code, indicating that the tests have failed, even though they succeeded.

Ran all test suites.
node:events:507
    const err = new ERR_UNHANDLED_ERROR(stringifiedEr);
                ^

Error: Unhandled error. ([Error: Connection is closed.])
    at RedisConnection.emit (node:events:507:17)
    at /my-project/node_modules/bullmq/src/classes/redis-connection.ts:133:41
    at processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'ERR_UNHANDLED_ERROR',
  context: [Error: Connection is closed.]
}

Node.js v22.17.0

I've tried every single suggested solution from here: https://stackoverflow.com/questions/62975121/close-redis-connection-when-using-nestjs-queues, and it always fails with the above error. (Some of them add additional errors, but this one seems to be the core of the problem.)

Minimum reproduction code

I'm unfortunately blocked from doing this on my work laptop.

Steps to reproduce

In a project that uses BullMQ:

1. Call await app.close() in the afterAll/afterEach hook.
2. Run npm run test:e2e

Expected behavior

Well, I would expect it to work out of the box, without adding app.close(). It seems like it should automatically close the app when all references to it become out of scope. If not, I don't see why app.close() isn't part of the auto-generated code.

All of that aside, I'd obviously expect that the test execution would finish without throwing an error.

Package version

11.0.2

Bull version

5.56.1 (bullmq)

NestJS version

11.0.1

Node.js version

v22.17.0

In which operating systems have you tested?

• macOS
• Windows
• Linux

Other

I've only reproduced this in our project, but I'll hopefully have time to create the minimum reproduction code when I get home. The project in which it was reproduced, there are multiple queues and workers, so I suppose it is possible that it would work in a simpler project with only one queue and one worker. I'll get back to you when I've been able to produce the minimum reproduction code.

[codiophile]

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.

When I add a second test, I get the above message and it exits with a zero exit code. So at least I am able to run e2e tests now. It seems like the difference is just that when it's only one file it runs directly in the parent jest process, but when you have multiple tests, each file is run in a child-process. The parent process can handle misbehaving child-processes, but when the crash is in the parent process itself, the whole execution crashes.

[codiophile]

Actually, having more than one test file seems to be a workaround, as jest is able to pass the test execution even if individual test processes fail. In a subsequent run, I again got the same error message, once for each test, but apart from printing it, jest ignored it.

[codiophile]

If you have multiple tests in the same file and you are creating a new app in beforeEach and closing it in afterEach, the second test will not be able to run, because it is crashing in-between and then you get a test failure.

As of right now, it seems like the best workaround is to ignore the issue and simply make sure to have more than one test file. The warning, about the worker process failing to exit gracefully, is a lot less spammy, than an error being logged, after each test.

Regardless of the behaviour of jest, the tests shouldn't crash, hang, or print warnings, but until that is fixed, at least there is a viable workaround.

[wodCZ]

I got stuck on this too for few hours.

Looking at issues in the bullmq repo I found this related discussion: taskforcesh/bullmq#2686

Thanks to the comments there I found that it's possible to run into a weird race-condition if you interact with or close the queue before it's ready.

Specifically, I have an upsertJobScheduler call in my onModuleInit lifecycle hook. Once I added await this.queue.waitUntilReady(); before that call, the Connection is closed error went away.

Now as I'm writing this, I figured that I was missing await in front of my upsertJobScheduler call - adding the await there also fixes the issue.

Edit: after fixing this, I no longer need to manually close my queues individually.

[Helveg]

Hi, I'm running into Connection is closed issues during tests as well, and I interact with queues in onApplicationBootstrap logic. The main difference is that I'm using regular Bull, not BullMQ: queues there don't have a waitUntilReady method. Any advice?

[clintonb]

I have finally solve this issue in my codebase! There are two parts.

1. Ensure that all e2e tests actually close the application. You can probably get away with simply calling await app.close(), but I like having a helper function I can adjust in the future.

/**
 * Close a test application.
 *
 * This is a helper to close an application, and squash any errors that may arise.
 * This can be removed after https://github.com/KurtzL/nestjs-temporal/pull/40 is resolved.
 */
export async function closeApp(app: INestApplication) {
  try {
    await app.close();
  } catch (e) {
    console.log(`Test app was not cleanly shutdown: ${e}`);
  }
}

describe('/v1/coupons', () => {
  let app: INestApplication;
  let db: DataSource;
  let transactionalContext: TransactionalTestContext;
  let banner: Banner;
  let user: APIBannerUser | undefined;
  let couponsService: CouponsService;

  beforeAll(async () => {
    mockTemporalSetupAndShutdown();

    // I have helper functions that setup common modules, mock, etc.
    // The end result is a standard `NestApplication`.
    const moduleRef = await createE2ETestingModule({
      imports: [AppModule],
    })
      .mockAuthentication(() => user)
      .overrideProvider(GoogleCloudStorage)
      .useValue(createMock<GoogleCloudStorage>())
      .compile();

    app = moduleRef.createNestApplication();
    app = configureApp(app);
    await app.init();
  });

  // This is needed for every e2e test.
  afterAll(async () => {
    await closeApp(app);
  });

...
}

2. Cleanup the open connections.

We were already closing the Redis connection on shutdown...

@Global()
@Module({
  providers: [...redisProviders],
  exports: [...redisProviders],
})
export class RedisModule implements OnApplicationShutdown {
  private readonly logger = new Logger(RedisModule.name);

  constructor(
    @Inject(Providers.REDIS_CLIENT)
    private readonly redis: Redis
  ) {}

  /**
   * Close the Redis connections when the application shuts down.
   *
   * @note This assumes modules that rely on Redis (e.g., cache, or BullMQ) have
   *  gracefully disconnected via OnModuleDestroy. Failure to do so will result
   *  in error log spam.
   */
  onApplicationShutdown(signal?: string) {
    const options = {
      host: this.redis.options.host,
      port: this.redis.options.port,
      tls: this.redis.options.tls,
      username: this.redis.options.username,
    };

    try {
      this.redis.disconnect();
      this.logger.log({ signal, options }, 'Disconnected from Redis');
    } catch (err) {
      this.logger.error({ err, signal, options }, 'Redis disconnect failed!');
    }
  }
}

We were closing some queues, but not all. We also were not closing the cache manager connection. I've done some experimentation here. You may be able to get away with simply close the cache manager connections. Tests still end when removing the call to closeQueues.

import { getQueueToken } from '@nestjs/bullmq';
import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Inject, Logger, Module, OnModuleDestroy } from '@nestjs/common';
import { ModuleRef } from '@nestjs/core';
import { Queue } from 'bullmq';
import { Cache } from 'cache-manager';
import { RedisStore } from 'cache-manager-ioredis-yet';

import { QueueName } from '@vori/nest/libs/queue/queue-constants';

/**
 * Module that closes all Redis connections on application shutdown.
 *
 * This addresses:
 * - BullMQ queue connections not being closed
 * - CacheManager Redis store not being closed (NestJS #1621, #8020)
 *
 * @see https://github.com/nestjs/nest/issues/1621
 * @see https://github.com/nestjs/nest/issues/8020
 * @see https://github.com/nestjs/bull/issues/2685
 */
@Module({})
export class RedisCleanupModule implements OnModuleDestroy {
  private readonly logger = new Logger(RedisCleanupModule.name);

  constructor(
    private readonly moduleRef: ModuleRef,
    @Inject(CACHE_MANAGER) private readonly cacheManager: Cache
  ) {}

  async onModuleDestroy() {
    await Promise.allSettled([this.closeQueues(), this.closeCacheStore()]);
  }

  private async closeQueues() {
    // NOTE: We keep track of all queues with a enum.
    const closePromises = Object.values(QueueName).map(async (queueName) => {
      try {
        const queue = this.moduleRef.get<Queue>(getQueueToken(queueName), {
          strict: false,
        });
        if (queue) {
          await queue.close();
          this.logger.log({ queue: queueName }, `Closed queue: ${queueName}`);
        }
      } catch (err) {
        // Queue may not be registered in this app - that's OK
        if (!(err instanceof Error && err.message.includes('not found'))) {
          this.logger.warn(
            { err, queue: queueName },
            `Failed to close queue: ${queueName}`
          );
        }
      }
    });

    await Promise.allSettled(closePromises);
  }

  private async closeCacheStore() {
    try {
      const store = this.cacheManager.store as RedisStore;
      if (store?.client) {
        await store.client.quit();
        this.logger.log('Closed CacheManager Redis connection');
      }
    } catch (err) {
      this.logger.warn(
        { err },
        'Failed to close CacheManager Redis connection'
      );
    }
  }
}