cmd/go: add package version support to Go toolchain

[rsc]

proposal: add package version support to Go toolchain

It is long past time to add versions to the working vocabulary of both Go developers and our tools.
The linked proposal describes a way to do that. See especially the Rationale section for a discussion of alternatives.

This GitHub issue is for discussion about the substance of the proposal.

Other references:

• https://research.swtch.com/vgo, the detailed designs
• A Tour of Versioned Go, a walkthrough of what it's like to use
• go get golang.org/x/vgo, the prototype implementation
• https://golang.org/wiki/vgo, links to related posts, videos, etc
• vgo milestone on issue tracker

[rsc]

Frequently Asked Questions

This issue comment answers the most frequently asked questions, whether from the discussion below or from other discussions. Other questions from the discussion are in the next issue comment.

Why is the proposal not “use Dep”?

At the start of the journey that led to this proposal, almost two years ago, we all believed the answer would be to follow the package versioning approach exemplified by Ruby's Bundler and then Rust's Cargo: tagged semantic versions, a hand-edited dependency constraint file known as a manifest, a separate machine-generated transitive dependency description known as a lock file, a version solver to compute a lock file satisfying the manifest, and repositories as the unit of versioning. Dep follows this rough plan almost exactly and was originally intended to serve as the model for go command integration. However, the more I understood the details of the Bundler/Cargo/Dep approach and what they would mean for Go, especially built into the go command, and the more I discussed those details with others on the Go team, a few of the details seemed less and less a good fit for Go. The proposal adjusts those details in the hope of shipping a system that is easier for developers to understand and to use. See the proposal's rationale section for more about the specific details we wanted to change, and also the blog post announcing the proposal.

Why must major version numbers appear in import paths?

To follow the import compatibility rule, which dramatically simplifies the rest of the system. See also the blog post announcing the proposal, which talks more about the motivation and justification for the import compatibility rule.

Why are major versions v0, v1 omitted from import paths?

v1 is omitted from import paths for two reasons. First, many developers will create packages that never make a breaking change once they reach v1, which is something we've encouraged from the start. We don't believe all those developers should be forced to have an explicit v1 when they may have no intention of ever releasing v2. The v1 becomes just noise. If those developers do eventually create a v2, the extra precision kicks in then, to distinguish from the default, v1. There are good arguments about visible stability for putting the v1 everywhere, and if we were designing a system from scratch, maybe that would make it a close call. But the weight of existing code tips the balance strongly in favor of omitting v1.

v0 is omitted from import paths because - according to semver - there are no compatibility guarantees at all for those versions. Requiring an explicit v0 element would do little to ensure compatibility; you'd have to say v0.1.2 to be completely precise, updating all import paths on every update of the library. That seems like overkill. Instead we hope that developers will simply look at the list of modules they depend on and be appropriately wary of any v0.x.y versions they find.

This has the effect of not distinguishing v0 from v1 in import paths, but usually v0 is a sequence of breaking changes leading to v1, so it makes sense to treat v1 as the final step in that breaking sequence, not something that needs distinguishing from v0. As @Merovius put it (#24301 (comment)):

By using v0.x, you are accepting that v0.(x+1) might force you to fix your code. Why is it a problem if v0.(x+1) is called v1.0 instead?

Finally, omitting the major versions v0 and v1 is mandatory - not optional - so that there is a single canonical import path for each package.

Why must I create a new branch for v2 instead of continuing to work on master?

You don't have to create a new branch. The vgo modules post unfortunately gives that impression in its discussion of the "major branch" repository layout. But vgo doesn't care about branches. It only looks up tags and resolves which specific commits they point at. If you develop v1 on master, you decide you are completely done with v1, and you want to start making v2 commits on master, that's fine: start tagging master with v2.x.y tags. But note that some of your users will keep using v1, and you may occasionally want to issue a minor v1 bug fix. You might at least want to fork a new v1 branch for that work at the point where you start using master for v2.

Won't minimal version selection keep developers from getting important updates?

This is a common fear, but I really think if anything the opposite will happen. Quoting the "Upgrade Speed" section of https://research.swtch.com/vgo-mvs:

Given that minimal version selection takes the minimum allowed version of each dependency, it's easy to think that this would lead to use of very old copies of packages, which in turn might lead to unnecessary bugs or security problems. In practice, however, I think the opposite will happen, because the minimum allowed version is the maximum of all the constraints, so the one lever of control made available to all modules in a build is the ability to force the use of a newer version of a dependency than would otherwise be used. I expect that users of minimal version selection will end up with programs that are almost as up-to-date as their friends using more aggressive systems like Cargo.

For example, suppose you are writing a program that depends on a handful of other modules, all of which depend on some very common module, like gopkg.in/yaml.v2. Your program's build will use the newest YAML version among the ones requested by your module and that handful of dependencies. Even just one conscientious dependency can force your build to update many other dependencies. This is the opposite of the Kubernetes Go client problem I mentioned earlier.

If anything, minimal version selection would instead suffer the opposite problem, that this “max of the minimums” answer serves as a ratchet that forces dependencies forward too quickly. But I think in practice dependencies will move forward at just the right speed, which ends up being just the right amount slower than Cargo and friends.

By "right amount slower" I was referring to the key property that upgrades happen only when you ask for them, not when you haven't. That means that code only changes (in potentially unexpected and breaking ways) when you are expecting that to happen and ready to test it, debug it, and so on.

See also the response #24301 (comment) by @Merovius.

If $GOPATH is deprecated, where does downloaded code live?

Code you check out and work on and modify can be stored anywhere in your file system, just like with essentially every other developer tool.

Vgo does need some space to hold downloaded source code and install binaries, and for that it does still use $GOPATH, which as of Go 1.9 defaults to $HOME/go. So developers will never need to set $GOPATH unless they want these files to be in a different directory. To change just the binary install location, they can set $GOBIN (as always).

Why are you introducing the // import comment?

We're not. That was a pre-existing convention. The point of that example in the tour was to show how go.mod can deduce the right module paths from import comments, if they exist. Once all projects use go.mod files, import comments will be completely redundant and probably deprecated.

[rsc]

Discussion Summary (last updated 2017-04-25)

This issue comment holds a summary of the discussion below.

How can we handle migration?

[#24301 (comment) by @ChrisHines.]

Response #24301 (comment) by @rsc. The original proposal assumes the migration is handled by authors moving to subdirectories when compatibility is important to them, but of course that motivation is wrong. Compatibility is most important to users, who have little influence on authors moving. And it doesn't help older versions. The linked comment, now also #25069, proposes a minimal change to old "go build" to be able to consume and build module-aware code.

How can we deal with singleton registrations?

[#24301 (comment) by @jimmyfrasche.]

Response #24301 (comment) by @rsc. Singleton registration collisions (such as http.Handle of the same path) between completely different modules are unaffected by the proposal. For collisions between different major versions of a single module, authors can write the different major versions to expect to coordinate, usually by making v1 call into v2, and then use a requirement cycle to make sure v2 is not used with older v1 that don't know about the coordination.

How should we install a versioned command?

[#24301 (comment) by @leonklingele.]

Response #24301 (comment) by @rsc. In short, use go get. We still use $GOPATH/bin for the install location. Remember that $GOPATH now defaults to $HOME/go, so commands will end up in $HOME/go/bin, and $GOBIN can override that.

Why are v0, v1 omitted in the import paths? Why must the others appear? Why must v0, v1 never appear?

[#24301 (comment) by @justinian.]
[#24301 (comment) by @jayschwa.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @kaikuehne.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @kaikuehne.]

Added to FAQ above.

Why are zip files mentioned in the proposal?

[#24301 (comment) by @nightlyone.]

The ecosystem will benefit from defining a concrete interchange format. That will enable proxies and other tooling. At the same time, we're abandoning direct use of version control (see rationale at top of this post). Both of this motivate describing the specific format. Most developers will not need to think about zip files at all; no developers will need to look inside them, unless they're building something like godoc.org.

See also #24057 about zip vs tar.

Doesn't putting major versions in import paths violate DRY?

[#24301 (comment) by @jayschwa.]

No, because an import's semantics should be understandable without reference to the go.mod file. The go.mod file is only specifying finer detail. See the second half of the semantic import versions section of the proposal, starting at the block quote.

Also, if you DRY too much you end up with fragile systems. Redundancy can be a good thing. So "violat[ing] DRY" - that is to say, limited repeating yourself - is not always bad. For example we put the package clause in every .go file in the directory, not just one. That caught honest mistakes early on and later turned into an easy way to distinguish external test packages (package x vs package x_test). There's a balance to be struck.

Which timezone is used for the timestamp in pseudo-versions?

[#24301 (comment) by @tpng.]

UTC. Note also that you never have to type a pseudo-version yourself. You can type a git commit hash (or hash prefix) and vgo will compute and substitute the appropriate pseudo-version.

Will vgo address non-Go dependencies, like C or protocol buffers? Generated code?

[#24301 (comment) by @AlexRouSg.]
[#24301 (comment) by @stevvooe.]
[#24301 (comment) by @nim-nim.]

Non-Go development continues to be a non-goal of the go command, so there won't be support for managing C libraries and such, nor will there be explicit support for protocol buffers.

That said, we certainly do understand that using protocol buffers with Go is too difficult, and we'd like to see that addressed separately.

As for generated code more generally, a real cross-language build system is the answer, specifically because we don't want every user to need to have the right generators installed. Better for the author to run the generators and check in the result.

Won't minimal version selection keep developers from getting important updates?

[#24301 (comment) by @TocarIP.]
[#24301 (comment) by @nim-nim.]
[#24301 (comment) by @Merovius.]

Added to FAQ.

Can I use master to develop v1 and then reuse it to develop v2?

[#24301 (comment) by @mrkanister.]
[#24301 (comment) by @aarondl.]

Yes. Added to FAQ.

What is the timeline for this?

[#24301 (comment) by @flibustenet.]

Response in #24301 (comment) by @rsc. In short, the goal is to land a "technology preview" in Go 1.11; work may continue a few weeks into the freeze but not further. Probably don't send PRs adding go.mod to every library you can find until the proposal is marked accepted and the development copy of cmd/go has been updated.

How can I make a backwards-incompatible security change?

[#24301 (comment) by @BURO9.]

Response in #24301 (comment) by @rsc. In short, the Go 1 compatibility guidelines do allow breaking changes for security reasons to avoid bumping the major version, but it's always best to do so in a way that keeps existing code working as much as possible. For example, don't remove a function. Instead, make the function panic or log.Fatal only if called improperly.

If one repo holds different modules in subdirectories (say, v2, v3, v4), can vgo mix and match from different commits?

[#24301 (comment) by @jimmyfrasche.]
[#24301 (comment) by @AlexRouSg.]

Yes. It treats each version tag as corresponding only to one subtree of the overall repository, and it can use a different tag (and therefore different commit) for each decision.

What if projects misuse semver? Should we allow minor versions in import paths?

[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]
[#24301 (comment) by @pbx0.]
[#24301 (comment) by @powerman.]

As @powerman notes, we definitely need to provide an API consistency checker so that projects at least can be told when they are about to release an obviously breaking change.

Can you determine if you have more than one package in a build?

[#24301 (comment) by @pbx0.]

The easiest thing to do would be to use goversion -m on the resulting binary. We should make a go option to show the same thing without building the binary.

Concerns about vgo reliance on proxy vs vendor, especially open source vs enterprise.

[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @kardianos.]
[#24301 (comment) by @Merovius.]
[#24301 (comment) by @joeshaw.]
[#24301 (comment) by @jamiethermo.]
[#24301 (comment) by @Merovius.]

Response: [#24301 (comment) by @rsc.] Proxy and vendor will both be supported. Proxy is very important to enterprise, and vendor is very important to open source. We also want to build a reliable mirror network, but only once vgo becomes go.

Concerns about protobuild depending on GOPATH semantics.

[#24301 (comment) by @stevvooe.]

Response [#24301 (comment) by @rsc] asked for more details in a new issue, but that issue does not seem to have been filed.

Suggestion to add special vgo-v1-lock tag.

[#24301 (comment) by @kybin.]

It seems appealing at first but leads to special cases that are probably not worth taking on. Full response in #24301 (comment).

How does one patch a deep dependency without vendoring?

[#24301 (comment) by @chirino.]

Response [#24301 (comment) by @kardianos.] By using a replace directive.

What will we do about module names changing?

[#24301 (comment) by @jimmyfrasche.]

Response [#24301 (comment) by @rsc.]
These are real, pre-existing problems that the vgo proposal does not attempt to address directly, but clearly we should address them eventually. The answer to code disappearing is to have caching proxies (mirrors) along with a reason to trust them; that's future work. (Or use vendoring in top-level project if you prefer.) The answer to code moving is to add an explicit concept of module or package redirects, much like type aliases are type redirects; that's also future work.

What about connecting to local Git servers?

[#24301 (comment) by @korya.]

Direct git access has been added back to the plan. See #24915.

What about binary-only packages?

[#24301 (comment) by @sdwarwick.]

Binary-only packages have only ever been supported in the limited circumstance of some kind of out-of-band installation into GOPATH/pkg. Go get has never supported fetching and installing a binary-only package, and it will continue not to support that. A binary-only package only works with one particular compiler and one particular copy of the dependencies, which severely limits how well it can be supported at all. The right answer is almost always to use source code instead.

Should we use path@version syntax in the go.mod file?

[#24301 (comment) by @sdwarwick.]

This is #24119. It seemed like a good idea at first but, after careful consideration, no.

[rsc]

.

[rsc]

.

[gopherbot]

Change https://golang.org/cl/101678 mentions this issue: design: add 24301-versioned-go

[ChrisHines]

This proposal is impressive and I like most everything about it. However, I posted the following concern on the mailing list, but never received any replies. In the meantime I've seen this issue raised by others in the Gophers slack channel for vgo and haven't seen a satisfactory answer there either.

From: https://groups.google.com/d/msg/golang-dev/Plc42fslQEk/rlfeNlazAgAJ

I am most worried about the migration path between a pre-vgo world and a vgo world going badly. I think we risk inflicting major pain on the Go community if there isn't a smooth migration path. Clearly the migration cannot be atomic across the whole community, but if I've understood all that you've written about vgo so far, there may be some situations where existing widely used packages will not be usable by both pre-vgo tools and post-vgo tools.

Specifically, I believe that existing packages that already have tagged releases with major versions >= 2 will not work with vgo until they have a go.mod file and also are imported with a /vN augmented import path. However, once those changes are made to the repository it will break pre-vgo uses of the package.

This seems to create a different kind of diamond import problem in which the two sibling packages in the middle of the diamond import a common v2+ package. I'm concerned that the sibling packages must adopt vgo import paths atomically to prevent the package at the top of the diamond from being in an unbuildable state whether it's using vgo or pre-vgo tools.

I haven't seen anything yet that explains the migration path in this scenario.

The proposal states:

Module-aware builds can import non-module-aware packages (those outside a tree with a go.mod file) provided they are tagged with a v0 or v1 semantic version. They can also refer to any specific commit using a “pseudo-version” of the form v0.0.0-yyyymmddhhmmss-commit. The pseudo-version form allows referring to untagged commits as well as commits that are tagged with semantic versions at v2 or above but that do not follow the semantic import versioning convention.

But I don't see a way for non-module-aware packages to import module-aware packages with transitive dependencies >= v2. That seems to cause ecosystem fragmentation in a way not yet addressed. Once you have a module-aware dependency that has a package >= v2 somewhere in its transitive dependencies that seems to force all its dependents to also adopt vgo to keep the build working.

Update: see also #24454

[Merovius]

The Go project has encouraged this convention from the start of the project, but this proposal gives it more teeth: upgrades by package users will succeed or fail only to the extent that package authors follow the import compatibility rule.

It is unclear to me what this means and how it changes from the current situation. It would seem to me, that this describes the current situation as well: If I break this rule, upgrades and go-get will fail. AIUI nothing really changes and I'd suggest removing at least the mention of "more teeth". Unless, of course, this paragraph is meant to imply that there are additional mechanisms in place to penalize/prevent breakages?

[jimmyfrasche]

This would also affect things like database drivers and image formats that register themselves with another package during init, since multiple major versions of the same package can end up doing this. It's unclear to me what all the repercussions of that would be.

[justinian]

If the major version is v0 or v1, then the version number element must be omitted; otherwise it must be included.

Why is this? In the linked post, I only see the rationale that this is what developers currently do to create alternate paths when they make breaking changes - but this is a workaround for the fact that they don't initially plan for the tooling not handling versions for them. If we're switching to a new practice, why not allow and encourage (or even mandate) that new vgo-enabled packages include v0 or v1? It seems like paths lacking versions are just opportunities for confusion. (Is this a vgo-style package? Where is the module boundary? etc.)

[jayschwa]

I generally like the proposal, but am hung up on requiring major versions in import paths:

1. It violates the DRY principle when the major version can already be known from the go.mod. Understanding what will happen if there's a mismatch between the two is also hard to intuit.
2. The irregularity of allowing v0 and v1 to be absent is also unintuitive.
3. Changing all the import paths when upgrading a dependency seems potentially tedious.

I understand that scenarios like the moauth example need to be workable, but hopefully not at the expense of keeping things simple for more common scenarios.

[nightlyone]

First of all: Impressive work!

One thing that is totally unclear to me and seems a bit underspecified:

Why there is a zip files in this proposal?

Layout, constraints and multiple use cases like when it is created and how it's life cycle is managed, what tools need support, how tools like linters should interact with it are also unclear, because they are not covered in the proposal.

So I would suggest to either refer to a later, still unwritten, proposal here and remove the word zip or remove the whole part from the proposal text, if you plan not discuss it at all within the scope of this proposal.

Discussing this later also enables a different audiences to contribute better here.