[CI Instability] Failure to Download Packages Through Verdaccio When Applying RNW Template

[NickGerleman]

This is one of the leading causes of CI instability, though it seems to pop up somewhat sporadically. Our react-native-init tests download some packages while using Verdaccio as an npm registry proxy. Verdaccio seems to cause issues with built-in retry logic which presents as some packages causing 404 errors.

We've previously tried to mitigate in various ways, and have reduced prevalence by doing less downloads through Verdaccio, but this is still an issue affectint maybe 1/20 PRs. Frequency will be reduced if we move more of the init jobs to nightly validation.

[juanpicado]

hey @NickGerleman

I see your configuration https://github.com/microsoft/react-native-windows/blob/master/.ado/verdaccio/config.yaml

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    max_fails: 40
    maxage: 30m
    timeout: 60s

I had the same issues while doing E2E for verdaccio using real package managers and I had to be more aggressive with my configuration

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    max_fails: 30
    fail_timeout: 10m
    timeout: 60s
    cache: false
    maxage: 30m

In my case https://github.com/verdaccio/verdaccio/blob/master/scripts/e2e-config.yaml I did disable the caching of tarballs (<-- in my case I do not need to cache tarballs at all since are one single usage) and increased the fail_timeout to avoid the uplink goes offline. The other values seems to be similar.

Futhermore, https://github.com/verdaccio/verdaccio/blob/master/.github/workflows/e2e-gatsbyjs-cli-workflow.yml#L36 I did increased the settings for npm so the package manager does not give up so easily.

In my case I did manage to reduce failures to 100% in the last 3 weeks, all pretty green while before was 80% failures in one of my scenarios. Hope this helps.

For the future reference, verdaccio/verdaccio#1966 I was actually working on new retry system, still super fresh but just wanna share this hopefully improve the DX in the next major.

[NickGerleman]

@juanpicado thank you for sharing all of that! Changing our config to be more permissive seems like a good next step.

[NickGerleman]

Still seeing this on some PRs. Need to investigate further.