Custom runner is ignored in browser mode

[andyearnshaw]

Describe the bug

I'm trying to write a custom runner for Svelte tests. However, I'm also running my tests in browser mode, and I noticed that my runner is not instantiated at all. If I comment out the browser configuration, my runner is instantiated correctly.

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-kmmccmjy?file=vite.config.ts

Given a basic runner that throws on construction:

// customrunner.js
import { VitestTestRunner } from 'vitest/runners';

class CustomRunner extends VitestTestRunner {
  constructor(config) {
    throw new Error('Custom runner failed successfully!');
  }
}

export default CustomRunner;

And a basic config:

export default defineConfig({
  test: {
    runner: './customrunner.js',

    // Comment out here to see the custom runner failing tests
    browser: {
      enabled: true,
      provider: preview(), // issue is with other providers also
      instances: [{ browser: 'chromium' }],
    },
  },
});

You can see that the tests run even though the runner should be throwing. If you comment out the browser config and run again, the runner will throw and fail the tests as expected.

System Info

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 20.19.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.8.2 - /usr/local/bin/npm
    pnpm: 8.15.6 - /usr/local/bin/pnpm
  npmPackages:
    @vitest/browser-preview: latest => 4.0.18 
    @vitest/ui: latest => 4.0.18 
    vite: latest => 7.3.1 
    vitest: latest => 4.0.18

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[sheremet-va]

What are you trying to solve with a custom test runner? At the moment it is intentionally disabled because browser integration is implemented as a test runner

[andyearnshaw]

I had a feeling that might be the reason.

It's sort of a long story, but the short of it is that I was curious and toying around with an approach to writing declarative tests for Svelte components (something Svelte is sorely lacking tbh) based on a couple of comments I came across whilst searching for a solution.

It involves writing tests as native Svelte components, like so:

<!-- Popover.test.svelte -->
<script lang="ts">
  import { Test, Check } from '$lib/test/test-components/index';
  import { expect, vi } from 'vitest';
  import { Popover, PopoverTrigger, PopoverMenu, PopoverMenuItem } from './index.ts';
  const actionFn = vi.fn();
</script>

<Test only it="opens the popover menu when the trigger is clicked">
  <!-- It's not possible to do this kind of declarative composition in standard Svelte tests -->
  <Popover>
    <PopoverTrigger>Open Menu</PopoverTrigger>
    <PopoverMenu>
      <PopoverMenuItem href="/login">Item 1</PopoverMenuItem>
      <PopoverMenuItem action={actionFn}>Item 2</PopoverMenuItem>
    </PopoverMenu>
  </Popover>

  {#snippet checks()}
    <Check
      fn={async (screen) => {
        expect(screen.getByRole('menu', { includeHidden: true })).not.toBeVisible();
        await screen.getByRole('button', { name: 'Open Menu' }).click();
        expect(screen.getByRole('menu')).toBeVisible();
      }}
    />
  {/snippet}
</Test>

Writing the Test and Check components was fairly straightforward and I confirmed the tests run and pass if I import them from a plain JS test file and render the exported "component". However, it has a few shortcomings:

• The tests cannot be discovered and run by having an include pattern for the *.test.svelte file names, because the compiler exports a default export that needs to be passed to mount() or render()
• A test error points to the wrong location in the source code
• The Vitest VSCode extension also doesn't pick up the tests for the test explorer, nor are the tests marked in the gutters

I thought a custom runner might be a way to solve these problems. I noticed there were hooks for task discovery and source imports. However, my tests are already running in browser mode. When I started toying around with the custom runner, I noticed it would only be instantiated if I disabled browser mode. I guess there's possibly an alternative way using Vite plugins, but I haven't looked into that yet.

[sheremet-va]

If the goal is to be supported automatically, then using a runner won't help with that. I would instead recommend writing a plugin that transforms Test into test functions. vscode extension can then parse the generated JS file and see tests (it doesn't run files to collect tests, it uses experimental_parseSpecification), the tests could also then run as a normal vitest <filter> command.

[andyearnshaw]

Thanks. I'm guessing it is done that way for performance reasons? The tradeoff is that tests generated at runtime cannot be detected then?

I don't think transforming <Test ...> into test(...) is going to work for this. The whole reason for this little experiment was that there's no way to have this kind of declarative component composition in plain JS tests. Writing a Test component means it runs through the Svelte compiler, so all of the Svelte syntax is possible and it does not need to depend on Svelte internals (of which external usage is strongly discouraged).

[sheremet-va]

Thanks. I'm guessing it is done that way for performance reasons? The tradeoff is that tests generated at runtime cannot be detected then?

Yes, collecting it statically is more beneficial for more people. Dynamic tests are collected after manually running tests, so users do get both.

I don't think transforming <Test ...> into test(...) is going to work for this. The whole reason for this little experiment was that there's no way to have this kind of declarative component composition in plain JS tests. Writing a Test component means it runs through the Svelte compiler, so all of the Svelte syntax is possible and it does not need to depend on Svelte internals (of which external usage is strongly discouraged).

You don't have to transform Test into test, you can transform the file in any way you like, even after svelte compiled it, for example.

[andyearnshaw]

I'm a little confused. Could you possibly give me an example of what you mean? Perhaps a sample input/output?

Edit: I was able to get the tests to run using the following transform code:

    {
      name: 'blah',
      enforce: 'post',
      transform(code, id) {
        if (!id.endsWith('.test.svelte')) {
          return;
        }

        const componentName = id.split('/').pop()?.split('.').slice(0, -1).join('_');
        const mountingCode = `import { mount } from 'svelte'; mount(${componentName}, { target: document.body });`;
        return `${code};\n${mountingCode}`;
      },

[sheremet-va]

I'm a little confused. Could you possibly give me an example of what you mean? Perhaps a sample input/output?

I don't know what code Svelte produces, but let's imagine that it exposes a default export function a component function:

// this is a file processed by svelte plugin
export default component() {
  // your component is rendered somehow
}

Then you can inject your own plugin and based on Test generate test blocks:

export default component() {
  // ...
}

test('opens the popover menu when the trigger is clicked', async ({ task }) => {
  await mount(component, { test: task.name }) // render Test snippet
})

Then any Vitest compiler can safely parse this file. This is all pseudo code, but that's the idea. Storybook, for example, works like this.

So in this case, the Test doesn't actually call test, but just renders itself when inside the test block of the same name.

[andyearnshaw]

Ah, I see now. Thanks for clarifying. I'll keep playing around and see if I can come up with something.

Thanks for all your help.

[andyearnshaw]

@sheremet-va I managed to put something together and it's working in VSCode with the Vitest extension.

Thanks again for your guidance. I'll close this now.

[sheremet-va]

@sheremet-va I managed to put something together and it's working in VSCode with the Vitest extension.

Awesome!