long-term flakiness on install in github actions powered CI

[JamesHenry]

Describe the bug

Hi there 👋

Thank you for verdaccio, it's a fantastic tool.

I have been using it for quite a while now on angular-eslint and it has been hugely useful for me when builing out a realistic suite of integration tests.

However, in (github actions) CI it has been consistently flaky for months now. Errors like SOCKET TIMEOUT are particularly common.

It may well be that I have messed up the setup, and I apologies in advance if that is the case.

The good thing is this is OSS, so you have full access to the project and usage to be able to troubleshoot! (I know how hard it can be without that access). Please let me know if I can help in any way.

To Reproduce

Here is an example tiny PR showing the behaviour:
angular-eslint/angular-eslint#261

And here you can see just how often it fails on master: https://github.com/angular-eslint/angular-eslint/commits/master

Often just a single re-run will resolve, but sometimes it takes a couple of goes.

It's starting to bug my OCD that master is often red just because of this verdaccio flakiness 😄 and it does result in PRs taking longer to merge than would otherwise be required (because of having to frequently rerun everything).

NOTE: I have never been able to reproduce any of the failures locally

Expected behavior

It shouldn't ever fail to install the dependencies from verdaccio when they are valid.

Screenshots

All relevant files, config and usage are in the integration-tests package:
https://github.com/angular-eslint/angular-eslint/tree/master/packages/integration-tests

Environment information

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
• $ npm -ddd prints:
• $ npm config get registry prints:

[juanpicado]

hi, quick reply

I see -> https://github.com/angular-eslint/angular-eslint/blob/11782a2ba94e9f1ac684e0ed6436da9b183ae21f/packages/integration-tests/local-registry/config.yml

You would need check on this,
microsoft/react-native-windows#6313 (comment)
https://twitter.com/jotadeveloper/status/1331494081319624706

I have this setup in my own registry and performance has increased quite a lot recently, also enabled by default not while ago #2014

[JamesHenry]

Sorry to be bothering you again @juanpicado, I really appreciate the help.

Things have been so good...

Latest master commits:

...Until now!

All the installs on all PRs are failing with ENEEDAUTH, which I had never seen before and cannot explain.

Here is one of the open PRs' failing checks as a concrete example, but they are all identical:
https://github.com/angular-eslint/angular-eslint/pull/267/checks?check_run_id=1596939988

[juanpicado]

No idea, AFK now.

I just found npm/npm#3623

But the error does not comes from verdaccio.

Well, Lerna says you are not logged in for some reason.

This command requires you to be logged in.

[JamesHenry]

Hi @juanpicado I got chance to look into this - I have identified the source of the issue, but not yet the root cause.

Node released v15.5.0, which ships with a later version of NPM vs 15.4 and this is where the issue starts. That is why it "suddenly" broke without any changes to the repo itself. I have been able to prove it locally by switching between the versions.

Let you know at this early stage as you may start to get some other folks bumping into this as more people pull down the latest Node version.

I'll let you know when I figure out a proper solution, for now I will be pinning the node version to unblock myself and the community PRs

[juanpicado]

And have you tried npm6 with Node.js 15 ?

[JamesHenry]

You don't even need to go that far back - 15.4 ships with an earlier v7.x release of NPM and that works, it's down to the very latest NPM release that is bundled with 15.5

Likely same underlying cause as this: npm/cli#2411