Pub needs rigorous, testable specification

[peter-ahe-google]

The pub package manager currently use YAML to specify package dependencies. This should be part of the language specification.

[peter-ahe-google]

cc @munificent.

[munificent]

My understanding is that a key design constraint of pub is that it is not part of the language spec. Aside from the tiny "package:" support in #import, everything pub does operates independent of the core Dart language.

This is important because it lets us change pub, or replace it with another package manager, or support multiple package managers.

Is there a reason we'd want it to be baked into the language?

[peter-ahe-google]

If we don't specify it, it isn't tested as part of co19. In my experience unspecified features tend to break when you try to change them in future releases. The team that is writing the co19 tests are particularly good at finding corner cases in specifications and writing tests that break "young" implementations.

[gbracha]

I don't think that pub is part of the language spec. Pub should have its own specification, and if we want co19 to write tests against that spec,we should do so (once such a spec exists).

I'm recategorizing this as pub bug and changing the summary accordingly.

---

Removed the owner.
Removed Area-Language label.
Added Area-Pub label.
Changed the title to: "Pun needs rigorous, testable specification".

[munificent]

YAML already has a specification, and our YAML library has a test suite. I'm sure the co19 guys could give us plenty more tests too but my understanding is that they are focused on the Dart language.

---

Added AsDesigned label.

[peter-ahe-google]

I'll take care of removing any unspecified features from dart2js. We should probably have a bug to remove package: support from the VM as well.

FWIW, saying that YAML is a specification of Pub amounts to saying that the grammar for Dart is the specification of Dart. It isn't, as it doesn't tell you the meaning and semantics.

Feel free to steal this bug away from me if the decision to specify Pub is changed. I certainly don't feel removing package: support from dart2js is moving us in the right direction, but it really isn't my job to specify what Pub does.

---

Set owner to @peter-ahe-google.
Added this to the M1 milestone.
Removed Priority-Medium, Area-Pub labels.
Added Priority-Critical, Area-Dart2JS, Triaged labels.

[dgrove]

Pub is not part of the language spec, IMHO.

However, we do need to get package: added to the spec. So let's identify what needs to happen there and get this into the language meeting.

This bug has morphed from pub needing a spec into deletion of package. I will create a new bug for Gilad on the package quesiton.

---

Set owner to @munificent.
Removed this from the M1 milestone.
Added this to the Later milestone.
Removed Priority-Critical, Area-Dart2JS labels.
Added Priority-Medium, Area-Pub labels.

[peter-ahe-google]

I don't think you can specify what the package: scheme means, without specifying at least some of the tags in the YAML file, in particular dependencies. But I could be wrong. However, it is not only the package scheme that should be specified:

Without a specification of the pubspec (I think that is the name of the YAML files read by Pub), dart2js is never going to be able to read those files directly. I don't think the user should have to run Pub before compiling, long term. So we need to specify what Pub is doing. Pretending it is not part of the language is only helping Bob or Gilad reducing the amount of specification they have to write. For everyone else, in particular users of the Dart SDK, lack of solid testable specification will be a constant source of problems.

Actually, this is very similar to GYP. GYP is using a "standard" format (Python strings, dictionaries, and lists) and describes how they are merged. However, most of the "tags" in GYP files are unspecified or sparsely documented. The result is that GYP is hard to use unless you have experience (trial and error) and read the source code.

Please do not make Pub another GYP.

---

Changed the title to: "Pub needs rigorous, testable specification".

[munificent]

I don't think you can specify what the package: scheme means, without specifying at least some of the tags in the YAML file, in particular dependencies.

You can, actually, which is why "package:" is designed the way it is. If that wasn't the case, it wouldn't have been possible to implement it: the editor, VM, Dartium, and dart2js all support "package:" and they definitely don't know anything about pub, pubspecs, or YAML. The semantics are "package:" are just:

Given a URL "package:<foo>", return the URL "packages/<foo>" resolved relative to the entrypoint's URL.

That's it. I agree that this should definitely be specified somewhere, either in the language spec or somewhere else.

Without a specification of the pubspec (I think that is the name of the YAML files read by Pub), dart2js is never going to be able to read those files directly.

Dart2js should never read those files directly. That's pub's job. If Dart implementations start baking in pub-specific stuff like this, we lose the important abstraction boundary between pub and the language.

I don't think the user should have to run Pub before compiling, long term.

This point has been discussed in unbelievably great detail by a bunch of people and this is the best solution we've come up with. If you'd like to re-open that discussion, let me know and I'll try to get you up to speed.

Pretending it is not part of the language is only helping Bob or Gilad reducing the amount of specification they have to write.

It doesn't save me any effort. I still have to document the pubspec format so that users can use it. Putting it in the language would save me time, because then I could make Gilad do that. :)

The reason it's not in the language is so that the Dart language is not coupled to one particular package manager. It's an abstraction boundary. We made this decision for architectural reasons, not laziness ones.

For everyone else, in particular users of the Dart SDK, lack of solid testable specification will be a constant source of problems.

Yes, like everything in the SDK, pub and the YAML library need solid tests. YAML already has a specification, so we're good there. Pub's tests are getting more solid over time. I don't see what this has to do with the language spec.

However, most of the "tags" in GYP files are unspecified or sparsely documented.

Yes, gyp has awful docs. As someone who has tried to use it for a personal project, I feel the pain here.

Please do not make Pub another GYP.

We won't. Nathan and I care about documentation probably more than anyone else on the Dart team.

[peter-ahe-google]

I don't think a directory is a good abstraction layer. I'm convinced that at least the Editor must know how to read the YAML files for us to ensure fast build times.

I have very concrete experience to back this up. We can talk about this when I return from vacation.

[munificent]

Removed the owner.

[munificent]

Removed Type-Defect label.
Added Type-Enhancement label.

[munificent]

Added NotPlanned label.

[DartBot]

This issue has been moved to dart-lang/pub#50.