Bug: V9 can't test non-exists file in `RuleTester`

[fisker]

Environment

Node version: v18.18.0
npm version: v9.4.2
Local ESLint version: v9.0.0-alpha.0 (Currently used)
Global ESLint version: Not found
Operating System: linux 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

What parser are you using?

Default (Espree)

What did you do?

Create a file to run RuleTester

import { RuleTester } from 'eslint';

const tester = new RuleTester();

const rule = {
  create: (context) => ({
    Program(node) {
      context.report({ node, message: 'It works' });
    },
  }),
};

tester.run('rule-id', rule, {
  valid: [],
  invalid: [
    {
      code: '_',
      filename: '/a-none-exits-file',
      errors: [{ message: 'It works' }],
    },
  ],
});

What did you expect to happen?

Should pass.

What actually happened?

Error: Error rule name should be the same as the name of the rule being tested (false == true)
    at _0x2e26db._evaluate (https://stackblitzstartersdvpau5-bahv.w-credentialless.staticblitz.com/blitz.a2aabdd9.js:352:376700)
    at async ModuleJob.run (https://stackblitzstartersdvpau5-bahv.w-credentialless.staticblitz.com/blitz.a2aabdd9.js:181:2372) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '=='
}

Link to Minimal Reproducible Example

https://stackblitz.com/edit/stackblitz-starters-dvpau5?description=Starter%20project%20for%20Node.js,%20a%20JavaScript%20runtime%20built%20on%20Chrome%27s%20V8%20JavaScript%20engine&file=index.js,package.json&title=node.new%20Starter

Participation

• I am willing to submit a pull request for this issue.

Additional comments

No response

[mdjermanovic]

filename: '/a-none-exits-file',

The problem is that absolute paths are not supported here (unless it happens that the path is in or under the current working directory). It would work with filename: 'a-none-exits-file' regardless of whether the file exists.

Do you need absolute paths in your test cases?

[fisker]

Shouldn't RuleTester allow to test any case?

[mdjermanovic]

I think that absolute paths should be supported because rules are getting absolute paths in production so testing with absolute paths seems more realistic. Perhaps we could pass path.dirname(item.filename) (or path.parse(item.filename).root?) as the basePath when creating a config array instance for the test.

eslint/lib/rule-tester/rule-tester.js

Lines 579 to 580 in 806f708

	function runRuleForItem(item) {
	const configs = new FlatConfigArray(testerConfig, { baseConfig });

@nzakas what do you think?

[nzakas]

I think that absolute paths should be supported because rules are getting absolute paths in production so testing with absolute paths seems more realistic. Perhaps we could pass path.dirname(item.filename) (or path.parse(item.filename).root?) as the basePath when creating a config array instance for the test.

eslint/lib/rule-tester/rule-tester.js

Lines 579 to 580 in 806f708

	function runRuleForItem(item) {
	const configs = new FlatConfigArray(testerConfig, { baseConfig });

@nzakas what do you think?

I think that makes sense. We should be able to support absolute paths in tests. 👍