Skip to content

Commit 7729236

Browse files
authored
feat(api): add extensible test artifact API (#8987)
1 parent afd1f3e commit 7729236

File tree

41 files changed

+1181
-206
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1181
-206
lines changed

docs/.vitepress/components.d.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ export {}
77
/* prettier-ignore */
88
declare module 'vue' {
99
export interface GlobalComponents {
10+
Advanced: typeof import('./components/Advanced.vue')['default']
1011
ArrowDown: typeof import('./components/ArrowDown.vue')['default']
1112
BlogIndex: typeof import('./components/BlogIndex.vue')['default']
1213
Box: typeof import('./components/Box.vue')['default']
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<template>
2+
<Badge type="danger" title="This is an advanced API intended for library authors and framework integrations. Most users should not need this." class="cursor-help">
3+
advanced
4+
</Badge>
5+
</template>

docs/.vitepress/config.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -993,6 +993,10 @@ export default ({ mode }: { mode: string }) => {
993993
text: 'TaskMeta',
994994
link: '/api/advanced/metadata',
995995
},
996+
{
997+
text: 'TestArtifact',
998+
link: '/api/advanced/artifacts',
999+
},
9961000
],
9971001
},
9981002
// {

docs/api/advanced/artifacts.md

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
---
2+
outline: deep
3+
title: Test Artifacts
4+
---
5+
6+
# Test Artifacts <Advanced /> <Version type="experimental">4.0.11</Version> <Experimental />
7+
8+
::: warning
9+
This is an advanced API. As a user, you most likely want to use [test annotations](/guide/test-annotations) to add notes or context to your tests instead. This is primarily used internally and by library authors.
10+
:::
11+
12+
Test artifacts allow attaching or recording structured data, files, or metadata during test execution. This is a low-level feature primarily designed for:
13+
14+
- Internal use ([`annotate`](/guide/test-annotations) is built on top of the artifact system)
15+
- Framework authors creating custom testing tools on top of Vitest
16+
17+
Each artifact includes:
18+
19+
- A type discriminator which is a unique identifier for the artifact type
20+
- Custom data, can be any relevant information
21+
- Optional attachments, either files or inline content associated with the artifact
22+
- A source code location indicating where the artifact was created
23+
24+
Vitest automatically manages attachment serialization (files are copied to [`attachmentsDir`](/config/#attachmentsdir)) and injects source location metadata, so you can focus on the data you want to record. All artifacts **must** extend from [`TestArtifactBase`](#testartifactbase) and all attachments from [`TestAttachment`](#testattachment) to be correctly handled internally.
25+
26+
## API
27+
28+
### `recordArtifact` <Experimental /> {#recordartifact}
29+
30+
::: warning
31+
`recordArtifact` is an experimental API. Breaking changes might not follow SemVer, please pin Vitest's version when using it.
32+
33+
The API surface may change based on feedback. We encourage you to try it out and share your experience with the team.
34+
:::
35+
36+
```ts
37+
function recordArtifact<Artifact extends TestArtifact>(task: Test, artifact: Artifact): Promise<Artifact>
38+
```
39+
40+
The `recordArtifact` function records an artifact during test execution and returns it. It expects a [task](/api/advanced/runner#tasks) as the first parameter and an object assignable to [`TestArtifact`](#testartifact) as the second.
41+
42+
This function has to be used within a test, and the test has to still be running. Recording after test completion will throw an error.
43+
44+
When an artifact is recorded on a test, it emits an `onTestArtifactRecord` runner event and a [`onTestCaseArtifactRecord` reporter event](/api/advanced/reporters#ontestcaseartifactrecord).
45+
46+
Note: annotations, [even though they're built on top of this feature](#relationship-with-annotations), won't appear in the `task.artifacts` array for backwards compatibility reasons until the next major version.
47+
48+
### `TestArtifact`
49+
50+
The `TestArtifact` type is a union containing all artifacts Vitest can produce, including custom ones. All artifacts extend from [`TestArtifactBase`](#testartifactbase)
51+
52+
### `TestArtifactBase` <Experimental /> {#testartifactbase}
53+
54+
```ts
55+
export interface TestArtifactBase {
56+
/** File or data attachments associated with this artifact */
57+
attachments?: TestAttachment[]
58+
/** Source location where this artifact was created */
59+
location?: TestArtifactLocation
60+
}
61+
```
62+
63+
The `TestArtifactBase` interface is the base for all test artifacts.
64+
65+
Extend this interface when creating custom test artifacts. Vitest automatically manages the `attachments` array and injects the `location` property to indicate where the artifact was created in your test code.
66+
67+
### `TestAttachment`
68+
69+
```ts
70+
export interface TestAttachment {
71+
/** MIME type of the attachment (e.g., 'image/png', 'text/plain') */
72+
contentType?: string
73+
/** File system path to the attachment */
74+
path?: string
75+
/** Inline attachment content as a string or raw binary data */
76+
body?: string | Uint8Array
77+
}
78+
```
79+
80+
The `TestAttachment` interface represents a file or data attachment associated with a test artifact.
81+
82+
Attachments can be either file-based (via `path`) or inline content (via `body`). The `contentType` helps consumers understand how to interpret the attachment data.
83+
84+
### `TestArtifactLocation`
85+
86+
```ts
87+
export interface TestArtifactLocation {
88+
/** Line number in the source file (1-indexed) */
89+
line: number
90+
/** Column number in the line (1-indexed) */
91+
column: number
92+
/** Path to the source file */
93+
file: string
94+
}
95+
```
96+
97+
The `TestArtifactLocation` interface represents the source code location information for a test artifact. It indicates where in the source code the artifact originated from.
98+
99+
### `TestArtifactRegistry`
100+
101+
The `TestArtifactRegistry` interface is a registry for custom test artifact types.
102+
103+
Augmenting this interface using [TypeScript's module augmentation feature](https://typescriptlang.org/docs/handbook/declaration-merging#module-augmentation) allows registering custom artifact types that tests can produce.
104+
105+
Each custom artifact should extend [`TestArtifactBase`](#testartifactbase) and include a unique `type` discriminator property.
106+
107+
Here are a few guidelines or best practices to follow:
108+
109+
- Try using a `Symbol` as the **registry key** to guarantee uniqueness
110+
- The `type` property should follow the pattern `'package-name:artifact-name'`, **`'internal:'` is a reserved prefix**
111+
- Use `attachments` to include files or data; extend [`TestAttachment`](#testattachment) for custom metadata
112+
- `location` property is automatically injected
113+
114+
## Custom Artifacts
115+
116+
To use and manage artifacts in a type-safe manner, you need to create its type and register it:
117+
118+
```ts
119+
import type { TestArtifactBase, TestAttachment } from 'vitest'
120+
121+
interface A11yReportAttachment extends TestAttachment {
122+
contentType: 'text/html'
123+
path: string
124+
}
125+
126+
interface AccessibilityArtifact extends TestArtifactBase {
127+
type: 'a11y:report'
128+
passed: boolean
129+
wcagLevel: 'A' | 'AA' | 'AAA'
130+
attachments: [A11yReportAttachment]
131+
}
132+
133+
const a11yReportKey = Symbol('report')
134+
135+
declare module 'vitest' {
136+
interface TestArtifactRegistry {
137+
[a11yReportKey]: AccessibilityArtifact
138+
}
139+
}
140+
```
141+
142+
As long as the types are assignable to their bases and don't have errors, everything should work fine and you should be able to record artifacts using [`recordArtifact`](#recordartifact):
143+
144+
```ts
145+
async function toBeAccessible(
146+
this: MatcherState,
147+
actual: Element,
148+
wcagLevel: 'A' | 'AA' | 'AAA' = 'AA'
149+
): AsyncExpectationResult {
150+
const report = await runAccessibilityAudit(actual, wcagLevel)
151+
152+
await recordArtifact(this.task, {
153+
type: 'a11y:report',
154+
passed: report.violations.length === 0,
155+
wcagLevel,
156+
attachments: [{
157+
contentType: 'text/html',
158+
path: report.path,
159+
}],
160+
})
161+
162+
return {
163+
pass: violations.length === 0,
164+
message: () => `Found ${report.violations.length} accessibility violation(s)`
165+
}
166+
}
167+
```
168+
169+
## Relationship with Annotations
170+
171+
Test annotations are built on top of the artifact system. When using annotations in tests, they create `internal:annotation` artifacts under the hood. However, annotations are:
172+
173+
- Simpler to use
174+
- Designed for end-users, not developers
175+
176+
Use annotations if you just want to add notes to your tests. Use artifacts if you need custom data.

docs/api/advanced/reporters.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Vitest has its own test run lifecycle. These are represented by reporter's metho
1616
- [`onHookEnd(beforeAll)`](#onhookend)
1717
- [`onTestCaseReady`](#ontestcaseready)
1818
- [`onTestAnnotate`](#ontestannotate) <Version>3.2.0</Version>
19+
- [`onTestCaseArtifactRecord`](#ontestcaseartifactrecord) <Version type="experimental">4.0.11</Version>
1920
- [`onHookStart(beforeEach)`](#onhookstart)
2021
- [`onHookEnd(beforeEach)`](#onhookend)
2122
- [`onHookStart(afterEach)`](#onhookstart)
@@ -332,3 +333,18 @@ function onTestAnnotate(
332333
The `onTestAnnotate` hook is associated with the [`context.annotate`](/guide/test-context#annotate) method. When `annotate` is invoked, Vitest serialises it and sends the same attachment to the main thread where reporter can interact with it.
333334

334335
If the path is specified, Vitest stores it in a separate directory (configured by [`attachmentsDir`](/config/#attachmentsdir)) and modifies the `path` property to reference it.
336+
337+
## onTestCaseArtifactRecord <Version type="experimental">4.0.11</Version> {#ontestcaseartifactrecord}
338+
339+
```ts
340+
function onTestCaseArtifactRecord(
341+
testCase: TestCase,
342+
artifact: TestArtifact,
343+
): Awaitable<void>
344+
```
345+
346+
The `onTestCaseArtifactRecord` hook is associated with the [`recordArtifact`](/api/advanced/artifacts#recordartifact) utility. When `recordArtifact` is invoked, Vitest serialises it and sends the same attachment to the main thread where reporter can interact with it.
347+
348+
If the path is specified, Vitest stores it in a separate directory (configured by [`attachmentsDir`](/config/#attachmentsdir)) and modifies the `path` property to reference it.
349+
350+
Note: annotations, [even though they're built on top of this feature](/api/advanced/artifacts#relationship-with-annotations), won't hit this hook and won't appear in the `task.artifacts` array for backwards compatibility reasons until the next major version.

packages/browser-playwright/src/commands/trace.ts

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -106,20 +106,25 @@ export const annotateTraces: BrowserCommand<[{ traces: string[]; testId: string
106106
const vitest = project.vitest
107107
await Promise.all(traces.map((trace) => {
108108
const entity = vitest.state.getReportedEntityById(testId)
109-
return vitest._testRun.annotate(testId, {
110-
message: relative(project.config.root, trace),
111-
type: 'traces',
112-
attachment: {
113-
path: trace,
114-
contentType: 'application/octet-stream',
109+
const location = entity?.location
110+
? {
111+
file: entity.module.moduleId,
112+
line: entity.location.line,
113+
column: entity.location.column,
114+
}
115+
: undefined
116+
return vitest._testRun.recordArtifact(testId, {
117+
type: 'internal:annotation',
118+
annotation: {
119+
message: relative(project.config.root, trace),
120+
type: 'traces',
121+
attachment: {
122+
path: trace,
123+
contentType: 'application/octet-stream',
124+
},
125+
location,
115126
},
116-
location: entity?.location
117-
? {
118-
file: entity.module.moduleId,
119-
line: entity.location.line,
120-
column: entity.location.column,
121-
}
122-
: undefined,
127+
location,
123128
})
124129
}))
125130
}

packages/browser/src/client/tester/runner.ts

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import type {
77
TaskResultPack,
88
Test,
99
TestAnnotation,
10+
TestArtifact,
1011
VitestRunner,
1112
} from '@vitest/runner'
1213
import type { SerializedConfig, TestExecutionMethod, WorkerGlobalState } from 'vitest'
@@ -227,28 +228,41 @@ export function createBrowserRunner(
227228
}
228229

229230
onTestAnnotate = (test: Test, annotation: TestAnnotation): Promise<TestAnnotation> => {
230-
if (annotation.location) {
231+
const artifact: TestArtifact = { type: 'internal:annotation', annotation, location: annotation.location }
232+
233+
return this.onTestArtifactRecord(test, artifact).then(({ annotation }) => annotation)
234+
}
235+
236+
onTestArtifactRecord = <Artifact extends TestArtifact>(test: Test, artifact: Artifact): Promise<Artifact> => {
237+
if (artifact.location) {
231238
// the file should be the test file
232239
// tests from other files are not supported
233-
const map = this.sourceMapCache.get(annotation.location.file)
240+
const map = this.sourceMapCache.get(artifact.location.file)
241+
234242
if (!map) {
235-
return rpc().onTaskAnnotate(test.id, annotation)
243+
return rpc().onTaskArtifactRecord(test.id, artifact)
236244
}
237245

238-
const traceMap = new DecodedMap(map as any, annotation.location.file)
239-
const position = getOriginalPosition(traceMap, annotation.location)
246+
const traceMap = new DecodedMap(map as any, artifact.location.file)
247+
const position = getOriginalPosition(traceMap, artifact.location)
248+
240249
if (position) {
241250
const { source, column, line } = position
242-
const file = source || annotation.location.file
243-
annotation.location = {
251+
const file = source || artifact.location.file
252+
artifact.location = {
244253
line,
245254
column: column + 1,
246255
// if the file path is on windows, we need to remove the starting slash
247256
file: file.match(/\/\w:\//) ? file.slice(1) : file,
248257
}
258+
259+
if (artifact.type === 'internal:annotation') {
260+
artifact.annotation.location = artifact.location
261+
}
249262
}
250263
}
251-
return rpc().onTaskAnnotate(test.id, annotation)
264+
265+
return rpc().onTaskArtifactRecord(test.id, artifact)
252266
}
253267

254268
onTaskUpdate = (task: TaskResultPack[], events: TaskEventPack[]): Promise<void> => {

packages/browser/src/node/rpc.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,8 +150,8 @@ export function setupBrowserRpc(globalServer: ParentBrowserProject, defaultMocke
150150
await vitest._testRun.collected(project, files)
151151
}
152152
},
153-
async onTaskAnnotate(id, annotation) {
154-
return vitest._testRun.annotate(id, annotation)
153+
async onTaskArtifactRecord(id, artifact) {
154+
return vitest._testRun.recordArtifact(id, artifact)
155155
},
156156
async onTaskUpdate(method, packs, events) {
157157
if (method === 'collect') {

packages/browser/src/types.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { MockedModuleSerialized, ServerIdResolution, ServerMockResolution } from '@vitest/mocker'
2-
import type { TaskEventPack, TaskResultPack, TestAnnotation } from '@vitest/runner'
2+
import type { TaskEventPack, TaskResultPack, TestArtifact } from '@vitest/runner'
33
import type { BirpcReturn } from 'birpc'
44
import type {
55
AfterSuiteRunMeta,
@@ -18,7 +18,7 @@ export interface WebSocketBrowserHandlers {
1818
onUnhandledError: (error: unknown, type: string) => Promise<void>
1919
onQueued: (method: TestExecutionMethod, file: RunnerTestFile) => void
2020
onCollected: (method: TestExecutionMethod, files: RunnerTestFile[]) => Promise<void>
21-
onTaskAnnotate: (testId: string, annotation: TestAnnotation) => Promise<TestAnnotation>
21+
onTaskArtifactRecord: <Artifact extends TestArtifact>(testId: string, artifact: Artifact) => Promise<Artifact>
2222
onTaskUpdate: (method: TestExecutionMethod, packs: TaskResultPack[], events: TaskEventPack[]) => void
2323
onAfterSuiteRun: (meta: AfterSuiteRunMeta) => void
2424
cancelCurrentRun: (reason: CancelReason) => void

0 commit comments

Comments
 (0)