Cypress has gotten much slower

[mccalluc]

John writes:

cypress has gotten slower on ci, taking 20m in recent runs

Should investigate whether this is the docker build, or cypress proper

[john-conroy]

#2725 split the cypress tests for maintenance and main builds cutting time to ~13-15 min.

[mccalluc]

I’ll looking at https://github.com/hubmapconsortium/portal-ui/actions/workflows/cypress-ci.yml?page=2&query=event%3Apush+is%3Asuccess … Every run after # 120 is > 20 minutes; Every run before is < 10 minutes.

I've made a checkout of the earlier state... but it seems to run slow now:

• Just a test of github actions -- Do not merge! #2752

No mid-May complaints on https://github.com/actions/virtual-environments/issues or https://github.com/actions/virtual-environments/discussions, though there are other reports of github actions slowdowns in particular situations, so it seemed like a good place to look.

Next up... did our other actions experience a similar slow down? Or did any other projects?

[mccalluc]

Yes, other tests slowed down too... but not all of them!:

• At that same commit node tests go from ~3 minutes to ~9 minutes
• but the pure python tests show no similar slow down: ~1:30 before and after.

This makes me think the problem is particular to Node (which could also show up in the duration of the Cypress task), but not related to a change in our code, because then a checkout from that point should run as fast as it used to.

Next up: compare timings before and after: What part of the process slowed down?

[mccalluc]

Just looking at the Node action:

• before, it took 3:17 total, 1:07 for dep installs, 2:02 to test
• after, it took 9:07 total, 7:21 for dep installs, 1:37 to test

I don't think I have timing information below that level, but I'll check. If this is a slowdown with NPM, other folks must have noticed that? Investigating...

[mccalluc]

before:

Thu, 12 May 2022 13:00:34 GMT
Run npm ci --prefix context
Thu, 12 May 2022 13:00:34 GMT   npm ci --prefix context
Thu, 12 May 2022 13:00:34 GMT   shell: /usr/bin/bash -e {0}
Thu, 12 May 2022 13:01:35 GMT
Thu, 12 May 2022 13:01:35 GMT > core-js@2.6.11 postinstall /home/runner/work/portal-ui/portal-ui/context/node_modules/core-js
Thu, 12 May 2022 13:01:35 GMT > node -e "try{require('./postinstall')}catch(e){}"

after:

Fri, 13 May 2022 19:53:38 GMT
Run npm ci --prefix context
Fri, 13 May 2022 19:53:38 GMT   npm ci --prefix context
Fri, 13 May 2022 19:53:38 GMT   shell: /usr/bin/bash -e {0}
Fri, 13 May 2022 20:00:55 GMT
Fri, 13 May 2022 20:00:55 GMT > core-js@2.6.11 postinstall /home/runner/work/portal-ui/portal-ui/context/node_modules/core-js
Fri, 13 May 2022 20:00:55 GMT > node -e "try{require('./postinstall')}catch(e){}"

So there's a big slowdown in npm ci between these two, but there's no finer grained information in the logs. (Click on the gear icon in the upper right to get timestamps.)

Next:

• Look at package-lock.json and confirm there aren't any other surprises.
• Look for other repos with CI, package-lock, and commits before and after, and see if they have the same pattern.

[mccalluc]

No diff besides the version bump:

$ git diff f5d5b36e  275bd1a context/package-lock.json
diff --git a/context/package-lock.json b/context/package-lock.json
index b9447877..5505ef0e 100644
--- a/context/package-lock.json
+++ b/context/package-lock.json
@@ -1,6 +1,6 @@
 {
   "name": "portal-ui",
-  "version": "0.43.0",
+  "version": "0.44.0",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {

[mccalluc]

Aha!

• [BUG] GitHub references to dependencies slow/hang npm/cli#4896

As of approximately 10AM EST today [May 13], installing using a GitHub reference on NPM 6 became slow to the point where it often simply hangs. This was not the case previously, and works on NPM 7/8.

Unfortunately there are other performance issues with the v2 lock file format also related to using GitHub references that are currently preventing an upgrade to NPM > 6.

Root cause seems to be dropped support for git protocol by github. Folks suggest upgrading npm... but we're already on npm 8.12.1.

• Confirm that CI has npm 8... maybe it's seeing an older, pre-installed version?
• See if anyone else has the problem, even with npm 8?
• Try the other solutions: changing git references to https?

[mccalluc]

Ah: We are only running npm 6.14.17 in CI... upgrading...