feat(roll): roll to ToT Playwright (08-02-23) (microsoft#938)

playwrightmachine · github-actions[bot] · web-flow · commit de374be4283d · 2023-02-08T11:20:17.000+01:00
Co-authored-by: github-actions &lt;41898282+github-actions[bot]@users.noreply.github.com&gt;
diff --git a/nodejs/docs/api/class-locatorassertions.mdx b/nodejs/docs/api/class-locatorassertions.mdx
@@ -454,6 +454,9 @@ Note that screenshot assertions only work with Playwright test runner.
   - `caret` "hide"|"initial" *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-1-option-caret"/><a href="#locator-assertions-to-have-screenshot-1-option-caret" class="list-anchor">#</a>
     
     When set to `"hide"`, screenshot will hide text caret. When set to `"initial"`, text caret behavior will not be changed.  Defaults to `"hide"`.
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.31</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-1-option-comparator"/><a href="#locator-assertions-to-have-screenshot-1-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `mask` [Array]<[Locator]> *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-1-option-mask"/><a href="#locator-assertions-to-have-screenshot-1-option-mask" class="list-anchor">#</a>
     
     Specify locators that should be masked when the screenshot is taken. Masked elements will be overlaid with a pink box `#FF00FF` that completely covers its bounding box.
@@ -473,7 +476,7 @@ Note that screenshot assertions only work with Playwright test runner.
     Defaults to `"css"`.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-1-option-threshold"/><a href="#locator-assertions-to-have-screenshot-1-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
   - `timeout` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-1-option-timeout"/><a href="#locator-assertions-to-have-screenshot-1-option-timeout" class="list-anchor">#</a>
     
     Time to retry the assertion for. Defaults to `timeout` in `TestConfig.expect`.
@@ -507,6 +510,9 @@ Note that screenshot assertions only work with Playwright test runner.
   - `caret` "hide"|"initial" *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-2-option-caret"/><a href="#locator-assertions-to-have-screenshot-2-option-caret" class="list-anchor">#</a>
     
     When set to `"hide"`, screenshot will hide text caret. When set to `"initial"`, text caret behavior will not be changed.  Defaults to `"hide"`.
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.31</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-2-option-comparator"/><a href="#locator-assertions-to-have-screenshot-2-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `mask` [Array]<[Locator]> *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-2-option-mask"/><a href="#locator-assertions-to-have-screenshot-2-option-mask" class="list-anchor">#</a>
     
     Specify locators that should be masked when the screenshot is taken. Masked elements will be overlaid with a pink box `#FF00FF` that completely covers its bounding box.
@@ -526,7 +532,7 @@ Note that screenshot assertions only work with Playwright test runner.
     Defaults to `"css"`.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-2-option-threshold"/><a href="#locator-assertions-to-have-screenshot-2-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
   - `timeout` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="locator-assertions-to-have-screenshot-2-option-timeout"/><a href="#locator-assertions-to-have-screenshot-2-option-timeout" class="list-anchor">#</a>
     
     Time to retry the assertion for. Defaults to `timeout` in `TestConfig.expect`.
diff --git a/nodejs/docs/api/class-pageassertions.mdx b/nodejs/docs/api/class-pageassertions.mdx
@@ -68,6 +68,9 @@ Note that screenshot assertions only work with Playwright test runner.
       height of clipping area
     
     An object which specifies clipping of the resulting image. Should have the following fields:
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.29</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-1-option-comparator"/><a href="#page-assertions-to-have-screenshot-1-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `fullPage` [boolean] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-1-option-full-page"/><a href="#page-assertions-to-have-screenshot-1-option-full-page" class="list-anchor">#</a>
     
     When true, takes a screenshot of the full scrollable page, instead of the currently visible viewport. Defaults to `false`.
@@ -90,7 +93,7 @@ Note that screenshot assertions only work with Playwright test runner.
     Defaults to `"css"`.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-1-option-threshold"/><a href="#page-assertions-to-have-screenshot-1-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
   - `timeout` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-1-option-timeout"/><a href="#page-assertions-to-have-screenshot-1-option-timeout" class="list-anchor">#</a>
     
     Time to retry the assertion for. Defaults to `timeout` in `TestConfig.expect`.
@@ -138,6 +141,9 @@ Note that screenshot assertions only work with Playwright test runner.
       height of clipping area
     
     An object which specifies clipping of the resulting image. Should have the following fields:
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.29</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-2-option-comparator"/><a href="#page-assertions-to-have-screenshot-2-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `fullPage` [boolean] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-2-option-full-page"/><a href="#page-assertions-to-have-screenshot-2-option-full-page" class="list-anchor">#</a>
     
     When true, takes a screenshot of the full scrollable page, instead of the currently visible viewport. Defaults to `false`.
@@ -160,7 +166,7 @@ Note that screenshot assertions only work with Playwright test runner.
     Defaults to `"css"`.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-2-option-threshold"/><a href="#page-assertions-to-have-screenshot-2-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
   - `timeout` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="page-assertions-to-have-screenshot-2-option-timeout"/><a href="#page-assertions-to-have-screenshot-2-option-timeout" class="list-anchor">#</a>
     
     Time to retry the assertion for. Defaults to `timeout` in `TestConfig.expect`.
diff --git a/nodejs/docs/api/class-snapshotassertions.mdx b/nodejs/docs/api/class-snapshotassertions.mdx
@@ -52,6 +52,9 @@ Note that matching snapshots only work with Playwright test runner.
   
   Snapshot name.
 - `options` [Object] *(optional)*
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.29</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-1-option-comparator"/><a href="#snapshot-assertions-to-match-snapshot-1-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `maxDiffPixelRatio` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-1-option-max-diff-pixel-ratio"/><a href="#snapshot-assertions-to-match-snapshot-1-option-max-diff-pixel-ratio" class="list-anchor">#</a>
     
     An acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1`. Default is configurable with `TestConfig.expect`. Unset by default.
@@ -60,7 +63,7 @@ Note that matching snapshots only work with Playwright test runner.
     An acceptable amount of pixels that could be different. Default is configurable with `TestConfig.expect`. Unset by default.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-1-option-threshold"/><a href="#snapshot-assertions-to-match-snapshot-1-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
 
 ---
 
@@ -94,6 +97,9 @@ Note that matching snapshots only work with Playwright test runner.
 
 **Arguments**
 - `options` [Object] *(optional)*
+  - `comparator` [string] *(optional)* <font size="2">Added in: v1.29</font><a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-2-option-comparator"/><a href="#snapshot-assertions-to-match-snapshot-2-option-comparator" class="list-anchor">#</a>
+    
+    A comparator function to use when comparing images. Defaults to `"pixelmatch"`.
   - `maxDiffPixelRatio` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-2-option-max-diff-pixel-ratio"/><a href="#snapshot-assertions-to-match-snapshot-2-option-max-diff-pixel-ratio" class="list-anchor">#</a>
     
     An acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1`. Default is configurable with `TestConfig.expect`. Unset by default.
@@ -105,7 +111,7 @@ Note that matching snapshots only work with Playwright test runner.
     Snapshot name. If not passed, the test name and ordinals are used when called multiple times.
   - `threshold` [number] *(optional)*<a aria-hidden="true" tabindex="-1" class="list-anchor-link" id="snapshot-assertions-to-match-snapshot-2-option-threshold"/><a href="#snapshot-assertions-to-match-snapshot-2-option-threshold" class="list-anchor">#</a>
     
-    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`.
+    An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax), default is configurable with `TestConfig.expect`. Defaults to `0.2`. This option is used by "pixelmatch" image comparator.
 
 
 [Accessibility]: /api/class-accessibility.mdx "Accessibility"
diff --git a/nodejs/docs/api/class-testconfig.mdx b/nodejs/docs/api/class-testconfig.mdx
@@ -118,9 +118,12 @@ module.exports = defineConfig({
     
     Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
   - `toHaveScreenshot` [Object] *(optional)*
+    - `comparator` [string] *(optional)*
+      
+      a comparator function to use, either `"pixelmatch"` or `"ssim-cie94"`. Defaults to `"pixelmatch"`.
     - `threshold` [number] *(optional)*
       
-      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`.
+      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`. This option is used by `pixelmatch` image comparator.
     - `maxDiffPixels` [number] *(optional)*
       
       an acceptable amount of pixels that could be different, unset by default.
@@ -139,9 +142,12 @@ module.exports = defineConfig({
     
     Configuration for the [expect(page).toHaveScreenshot()](/api/class-pageassertions.mdx#page-assertions-to-have-screenshot-1) method.
   - `toMatchSnapshot` [Object] *(optional)*
+    - `comparator` [string] *(optional)*
+      
+      a comparator function to use, either `"pixelmatch"` or `"ssim-cie94"`. Defaults to `"pixelmatch"`.
     - `threshold` [number] *(optional)*
       
-      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`.
+      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`. This option is used by `pixelmatch` image comparator.
     - `maxDiffPixels` [number] *(optional)*
       
       an acceptable amount of pixels that could be different, unset by default.
diff --git a/nodejs/docs/api/class-testproject.mdx b/nodejs/docs/api/class-testproject.mdx
@@ -181,9 +181,12 @@ testProject.expect
     
     Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
   - `toHaveScreenshot` [Object] *(optional)*
+    - `comparator` [string] *(optional)*
+      
+      a comparator function to use, either `"pixelmatch"` or `"ssim-cie94"`. Defaults to `"pixelmatch"`.
     - `threshold` [number] *(optional)*
       
-      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`.
+      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`. This option is used by `pixelmatch` image comparator.
     - `maxDiffPixels` [number] *(optional)*
       
       an acceptable amount of pixels that could be different, unset by default.
@@ -202,9 +205,12 @@ testProject.expect
     
     Configuration for the [expect(page).toHaveScreenshot()](/api/class-pageassertions.mdx#page-assertions-to-have-screenshot-1) method.
   - `toMatchSnapshot` [Object] *(optional)*
+    - `comparator` [string] *(optional)*
+      
+      a comparator function to use, either `"pixelmatch"` or `"ssim-cie94"`. Defaults to `"pixelmatch"`.
     - `threshold` [number] *(optional)*
       
-      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`.
+      an acceptable perceived color difference between the same pixel in compared images, ranging from `0` (strict) and `1` (lax). `"pixelmatch"` comparator computes color difference in [YIQ color space](https://en.wikipedia.org/wiki/YIQ) and defaults `threshold` value to `0.2`. This option is used by `pixelmatch` image comparator.
     - `maxDiffPixels` [number] *(optional)*
       
       an acceptable amount of pixels that could be different, unset by default.
diff --git a/nodejs/docs/test-reporters.mdx b/nodejs/docs/test-reporters.mdx
@@ -457,7 +457,53 @@ Or if there is a custom folder name:
 npx playwright show-report my-report
 ```
 
-> The `html` reporter currently does not support merging reports generated across multiple [`--shards`](./test-parallel.mdx#shard-tests-between-multiple-machines) into a single report. See [this](https://github.com/microsoft/playwright/issues/10437) issue for available third party solutions.
+#### Sharded report
+
+When running tests on [multiple shards](./test-parallel.mdx#shard-tests-between-multiple-machines), the `html` reporter can automatically show test results from all shards in one page when configured with `sharded: true`.
+
+<Tabs
+  groupId="js-flavor"
+  defaultValue="ts"
+  values={[
+    {label: 'TypeScript', value: 'ts'},
+    {label: 'JavaScript', value: 'js'}
+  ]
+}>
+<TabItem value="ts">
+
+```js
+// playwright.config.ts
+import { defineConfig } from '@playwright/test';
+
+export default defineConfig({
+  reporter: [['html', { sharded: true }]],
+});
+```
+
+</TabItem>
+<TabItem value="js">
+
+```js
+// playwright.config.js
+// @ts-check
+
+const { defineConfig } = require('@playwright/test');
+
+module.exports = defineConfig({
+  reporter: [['html', { sharded: true }]],
+});
+```
+
+</TabItem>
+</Tabs>
+
+You can use sharded html report combined with a file hosting that allows serving html files.
+
+In your CI recipe, after running tests in each shard, upload all files from `playwright-report` directory to the **same location**. After that you can open `index.html` from the uploaded location directly in the browser.
+
+:::note
+The `html` report for each shard consists of `index.html` and a data file named like `report-003-of-100.zip`. It's ok to overwrite  `index.html` with one another when copying sharded reports to a single directory.
+:::
 
 ### JSON reporter
 
