Visual regression testing using Galata

.github/workflows/tests.yml

@@ -143,3 +143,79 @@ jobs:
       run: |
         python ./packages/schema/generate-spec.py -f markdown > spec.md
         diff -u ./packages/schema/jupyterwidgetmodels.latest.md ./spec.md
+  ui-test:
+    name: Visual Regression
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.7
+    - uses: actions/cache@v1
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}-${{hashFiles('**/requirements.txt')}}
+        restore-keys: |
+          ${{ runner.os }}-pip-
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install jupyterlab==3.0.3
+
+    - name: Use Node.js 12.x
+      uses: actions/setup-node@v1
+      with:
+        node-version: 12.x
+    - name: Get yarn cache
+      id: yarn-cache
+      run: echo "::set-output name=dir::$(yarn cache dir)"
+    - uses: actions/cache@v1
+      with:
+        path: ${{ steps.yarn-cache.outputs.dir }}
+        key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-yarn-
+
+    - name: Build and Install ipywidgets
+      run: |
+        yarn install --frozen-lockfile
+        yarn run build
+    - name: Build the extension
+      run: |
+        pip install .
+        jlpm install
+        jlpm run build
+        cd jupyterlab_widgets
+        pwd
+        pip install -e .
+        jupyter labextension develop . --overwrite
+        jupyter labextension list
+    - name: Install Galata
+      run: |
+        cd ui-tests
+        yarn install --frozen-lockfile
+    - name: Launch JupyterLab
+      run: |
+        cd ui-tests
+        yarn run start-jlab:detached
+    - name: Wait for JupyterLab
+      uses: ifaxity/wait-on-action@v1
+      with:
+        resource: http-get://localhost:8888/api
+        timeout: 20000
+    - name: Run UI Tests
+      run: |
+        cd ui-tests
+        yarn run test
+    - name: Upload UI Test artifacts
+      if: always()
+      uses: actions/upload-artifact@v2
+      with:
+        name: ui-test-output
+        path: |
+          ui-tests/test-output
+
+      env:
+        CI: true

.gitignore

@@ -36,3 +36,5 @@ index.built.js.map
 temp/*
 
 tsconfig.tsbuildinfo
+
+ui-tests/test-output/*

docs/source/dev_testing.md

@@ -1,12 +1,63 @@
-Testing
-=======
+# Unit Tests
 
 To run the Python tests:
 
-    pytest --cov=ipywidgets ipywidgets
+```sh
+pytest --cov=ipywidgets ipywidgets
+```
 
 To run the Javascript tests in each package directory:
-
-    yarn test
+```sh
+yarn test
+```
 
 This will run the test suite using `karma` with 'debug' level logging.
+
+
+# Visual Regression Tests
+
+ipywidgets uses [Galata](https://github.com/jupyterlab/galata) framework for visual regression testing. Galata provides a high level API to interact with JupyterLab UI programmatically and tools for capturing, comparison and report generation.
+
+Galata UI tests are written using typescript and [jest](https://github.com/facebook/jest). ipywidgets UI test suites are located in [ui-tests/tests](./ui-tests/tests) directory.
+
+[ui-tests/tests/widgets.test.ts](./ui-tests/tests/widgets.test.ts) test suite uploads a [notebook](./ui-tests/tests/notebooks/widgets.ipynb) into JupyterLab, runs it cell by cell and takes image captures of cell outputs. Cell outputs of this notebook are widgets of different types. Then, cell output captures are compared to [reference images](./ui-tests/reference-output/) to detect any visual regressions. Test output (diffs, result report) generated by Galata are uploaded to GitHub artifact storage and accessible from GitHub Actions page in `Artifacts` section.
+
+## Running Tests Locally
+
+First install the dependencies:
+
+```sh
+cd ui-tests
+yarn install
+```
+
+Galata needs to connect to a running instance of JupyterLab 3 to run UI tests. First launch JupyterLab and keep it running in a terminal window.
+
+```sh
+# in ui-tests directory
+yarn run start-jlab
+```
+Alternatively, you can use the command below to launch JupyterLab which is equivalent to `start-jlab` script:
+
+```sh
+jupyter lab --expose-app-in-browser --no-browser --ServerApp.token='' --ServerApp.password='' --ServerApp.disable_check_xsrf='True'
+```
+
+Then you can run the `test` script which in turn launches Galata to run UI tests:
+
+```sh
+# in the ui-tests directory
+yarn run test
+```
+
+You can pass additional arguments to Galata by appending parameters to the command as in `yarn run test -- --no-headless`. For a full list of Galata command-line options see [https://github.com/jupyterlab/galata#command-line-options](https://github.com/jupyterlab/galata#command-line-options).
+
+## Adding new UI tests
+
+New test suites can be added into [ui-tests/tests](./ui-tests/tests) directory. Their names need to end with `.test.ts`. You can see some additional example test suites in [Galata repo](https://github.com/jupyterlab/galata/blob/main/packages/galata/tests). If tests in new suites are doing visual regression tests or HTML source regression tests then you also need to add their reference images / HTML files into [ui-tests/tests/reference-output](./ui-tests/tests/reference-output) directory.
+
+## Reference Image Captures
+
+When doing visual regression tests, it is important to use reference images that were generated in the same environment. Otherwise, even though the same browser is used for testing, there might be minor differences in image renderings generated that could cause visual regression tests to fail.
+
+When adding a new visual regression test, first make sure your tests pass locally on your development environment, with a reference image generated in your dev environment. You can use images captured by Galata as reference images. They will be saved as part of test output, under [ui-tests/test-output/test/screenshots](ui-tests/test-output/test/screenshots). However, you shouldn't push these references images generated in your development environment to github. Instead, have the new regression tests run and fail by GitHub Actions first, then download the artifacts from GitHub and use the captures generated in GitHub testing environment as the reference images and push those in a separate commit.

ui-tests/galata-config.json

@@ -0,0 +1,3 @@
+{
+  "testId": "test"
+}

ui-tests/jupyter_server_config.py

@@ -0,0 +1,7 @@
+c.ServerApp.port = 8888
+c.ServerApp.token = ""
+c.ServerApp.password = ""
+c.ServerApp.disable_check_xsrf = True
+c.ServerApp.open_browser = False
+c.LabApp.open_browser = False
+c.LabApp.expose_app_in_browser = True

ui-tests/package.json

@@ -0,0 +1,16 @@
+{
+  "name": "ipywidgets-ui-tests",
+  "version": "1.0.0",
+  "description": "ipywidgets UI Tests",
+  "private": true,
+  "scripts": {
+    "start-jlab": "jupyter lab --config ./jupyter_server_config.py",
+    "start-jlab:detached": "yarn run start-jlab&",
+    "test": "galata"
+  },
+  "author": "Project Jupyter",
+  "license": "BSD-3-Clause",
+  "dependencies": {
+    "@jupyterlab/galata": "3.0.3-3"
+  }
+}