[Bug] Cannot add dependency as tar.gz from URL.

[EliteMasterEric]

• I'd be willing to implement a fix

Describe the bug

GitPKG is a website which allows the installation of NPM packages from a github repo which is a monorepo, i.e. the individual packages to be installed are located in subdirectories of the main repo. This is a situation that typically cannot be resolved by package managers, since the URL of the repo itself cannot be distinguished from a subdirectory of that repo.

In Yarn v1, the following line can be run:

yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

This will add the hello repo to the project's dependencies. Accessing the above URL directly will return a tar.gz file containing the repo data.

In Yarn v2, this command will fail with the following stack trace:

> yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
Internal Error: Invalid descriptor (https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello)
    at Module.x (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:434115)
    at D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:29878
    at Array.map (<anonymous>)
    at M.execute (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:29786)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async M.validateAndExecute (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:659490)
    at async Y.run (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:17:3854)
    at async Y.runExit (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:17:4021)
    at async h (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:284223)
    at async r (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:282834)

I researched the issue here and found #994 (comment), but running yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello provides the same error.

To Reproduce

I have reproduced the error, using Sherlock on my local machine.

Reproduction

This is my reproduction case:

// The following command was valid 
// yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

const installPromise = yarn(`add`, `https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello`);

// Should install successfully but can't.
await expect(installPromise)
  .resolves.toBeTruthy();

/**
 * Specifying the package name doesn't work either.
 * yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
 * @see: https://github.com/yarnpkg/berry/issues/994#issuecomment-628586986
 */

const installPromise = yarn(`add`, `hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello`);

await expect(installPromise)
  .resolves.toBeTruthy();

Environment if relevant (please complete the following information):

• OS: Windows
• Node version: v14.15.4
• Yarn version: 2.4.0

Additional context

I was not able to get Sherlock working properly on the sandbox website due to the following error:

Usage Error: This tool requires a Node version compatible with >=12 <14 || 14.2 - 14.9 || >14.10.0 (got 10.23.0). Upgrade Node, or set `YARN_IGNORE_NODE=1` in your environment.·
Yarn Package Manager - 2.4.0.dev·

I was able to clone the project to execute Sherlock offline and create the above reproduction.

[EliteMasterEric]

The issue appears to be on this line:

berry/packages/yarnpkg-core/sources/structUtils.ts

Line 358 in 14bc23f

const match = strict

[yarnbot]

This issue reproduces on master:

Error: expect(received).resolves.toBeTruthy()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

Usage Error: No project found in /tmp/tmp-20NCgscuoG90y9

$ yarn add [--json] [-E,--exact] [-T,--tilde] [-C,--caret] [-D,--dev] [-P,--peer] [-O,--optional] [--prefer-dev] [-i,--interactive] [--cached] ...
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0ac41999f0.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:10:7)
    at /github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:19
    at executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:22)
    at Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:18)
    at ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:59)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

[arcanis]

This is expected; we currently require the package name to be listed, except for local file paths. ie, this isn't supported:

yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

This is:

yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

I'll clarify the error message in the next release, I agree it's currently quite confusing.

[EliteMasterEric]

This is expected; we currently require the package name to be listed, except for local file paths. ie, this isn't supported:

yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

This is:

yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

I'll clarify the error message in the next release, I agree it's currently quite confusing.

@arcanis Please actually read the issue, I tested the latter (as I researched and found the answer you just gave in #994) and the same error is thrown.

I just got the following error on Ubuntu 20.04/Yarn 2.4.0 (so the issue is cross-platform):

$ yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello isn't supported by any available resolver
    at n.getResolverByDescriptor (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:336683)
    at n.bindDescriptor (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:336048)
    at d (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:357409)
    at async Promise.all (index 0)
    at async ie.resolveEverything (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:358614)
    at async /home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:377271
    at async f.startTimerPromise (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:389740)
    at async ie.install (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:377210)
    at async /home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:33217
    at async Function.start (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:388437)
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 8ms

[merceyz]

You need to add the extension, but that seems to highlight another issue

$ yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello.tgz
➤ YN0001: │ Error: hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello.tgz: ENOENT: no such file or directory, chmod 'C:\Temp\xfs-4d40b2cb\archive.zip'

GitPKG is a website which allows the installation of NPM packages from a github repo which is a monorepo, i.e. the individual packages to be installed are located in subdirectories of the main repo. This is a situation that typically cannot be resolved by package managers, since the URL of the repo itself cannot be distinguished from a subdirectory of that repo.

We can actually https://yarnpkg.com/features/protocols#git

[EliteMasterEric]

We can actually https://yarnpkg.com/features/protocols#git

This is interesting. The dependency I am attempting to import is a clone of an existing project (which uses NPM) but it's good to know Yarn supports this.

This is kind of a tangent but I'd really like to see a page in the docs that provides a breakdown for "Why should you migrate your project from NPM?" Points could include stuff like speed benchmarks and lists of features that NPM does not support.

[refi64]

Just as a quick FYI, if anyone here has a site where adding the .tgz is not an option, it seems you can just add it after a #, e.g.:

yarn add https://the-url.com/package#.tgz

in which case it usually isn't counted as part of the URL, so the actual URL is https://the-url.com/package, but Yarn sees the .tgz. (This trick is often used to rename sources in RPM spec files.)

[hendriku]

We still have this problem in yarn 3. Using https:// and suffixing with #.tgz also is the workaround for us.

[dakom]

that trick does work, but how about when specifying a commit? e.g. this fails:

yarn add recoilize@https://gitpkg.now.sh/open-source-labs/Recoilize/package?524ff2f9650e4a24e89ef00caa638b5148e9e966#.tgz