Support for private/enterprise patterns

Today, dep's primary target is Go's public, FLOSS ecosystem. The public ecosystem is the necessary shared baseline use case that must be satisfied.

Necessary...but not sufficient. Supporting private ecosystems and workflows not only makes things easier for Go users, but it can make it easier to convert private code to open source down the line.

There are a number of things enterprise organizations, or really any private organizations, may want:

Private code hosting (related: Support for private repositories e.g. Github Enterprise #174, Central packaging registry #175, Workflow: local cache servers for remote packages #263)
Policies to ban/bless projects, in their entirety or just a subset of their versions - Allow defining project {black,white}list policies #285
Seamless integration with monorepos
Integration with higher-order build systems (bazel, pants, etc.)

It doesn't seem like these changes should be a blocker for initially merging dep into the go toolchain, though they should probably be in good shape by the time the go toolchain ships a version with dep in it.

@pbennett

/cc @pbennett this would be a great place for you to summarize some of the requirements you have that we've discussed :)

Apologies if this is covered elsewhere; I've had a look around and I couldn't find it.

The two big use cases I have as at least a semi-enterprise user that often seem to be poorly covered for me is:

Being able to internally mirror repos and pull from those, even though we still use the "real" URL in the source itself.
Being able to put a patch on top of an external repo for whatever reason (security, integration, etc.) and use this patched repo in preference to the "real" one.

Given the way Go works, I think it would suffice for both these use cases to be able to specify an "alias" or something for a package, i.e., "I want github.com/go/AwesomePackage but please pull source from internalgit.enterprise.org/gits/go/AwesomePackage" or something like that. That would be really useful for me.

I tried to read the source code to see if this is already there, but I don't think it is? I see "overrides" in the manifest but that appears to be related to relaxing version constraints.

@thejerf

@thejerf it is, but no worries :) Have a look around here - #174 (comment). The source field in the manifest will allow you to individually specify an alternate location from which to pull a given dep. Roughly, an alias.

Doing them as more than one-offs is trickier, as described in the linked comment.

#148 is one of the main issues I've been running up against using dep in my current project.

@lancehit

@lancehit #148 is one of those things that we may end up needing to devise an alternate approach for, rather than trying to support it more directly. Not sure yet, but it's possible.

Take this all with a grain of salt, I'm still wrapping my head around the Go import model.

We have a SVN monorepo and are looking at how we should structure Go code within it and how imports will work.

I'm curious what support for a monorepo would look like? Could you elaborate?

For example if we have this example monorepo structure:

trunk/
    pkg1/
    project/
        pkg2/

How do we get to the desired GOPATH structure:

$GOPATH/
    src/
        company/
            pkg1/
            pkg2/

I've already fought with go get and subversion and it seems more hassle than it's worth. Besides we want clean import paths such as company/pkg1 not svn.company.org:8000/trunk/ (which I realize is not a valid import path). Perhaps vanity imports or canonical import paths are of some use but I haven't spent enough time investigating.

Go doesn't really care how the code got into the $GOPATH so my current thought process is to checkout each project/pkg into $GOPATH/src/company as needed. This is obviously slightly annoying though. Each package that has internal dependencies will have to know which packages to check out.

I guess it woud be nice to have the manifest reference a local relative path in the monorepo with a different import name. Something like

{
    "dependencies": {
        "company/pkg1": {
            "source": "../../path/inside/monorepo/pkg1"
        },
    }
}

@sdboyer

@sdboyer hey hey, this isn't exactly about go, but I bet you'd find some of the conversation in rust-lang/rust-roadmap-2017#12 very helpful. We're currently investigating this stuff in rust-land too.

Good luck!

@steveklabnik

@steveklabnik oh man, there's a wealth of useful information there - thanks!

@aajtodd

Grr, GitHub mobile moved the button into where the text field was just as I was tapping, sorry :)

@aajtodd If you have a monorepo, you will do best if you have a single directory that can be checked out into GOPATH. For example, svn.company.com:8000/trunk/go -> $GOPATH/SRC/go.company.com. It is greatly confusing when the directory structure in the repo does not match the package structure. (At that point, you don't have a monorepo, you have a bunch of separate repos that just happen to be stored in the same place.)

@quentinmit

@quentinmit That's what I was leaning towards. And yes you are correct that it is currently more a bunch of separate repos/projects stored in one place. Not my choice, such is life.

To properly work inside an enterprise env, the system needs to allow the insertion of an enterprise repository for internally hosted as well as proxied content. It should be easy for a developer in this env to be able to direct all outbound requests to their internal repository similar to the Maven mirrorOf directive. This is mentioned to contrast other ecosystems like bower and npm where the metadata contains absolute urls and the client follows them faithfully. This means the implementation of an enterprise repository has to get in the business of rewriting all the metadata which potentially involves modifying the packages etc (depending on how the metadata is distributed)

In my experience you're exactly right. Each system comes along these days with a basic tool but companies tend to not to want to run a separate server for each new format/language and gravitate towards a heterogeneous enterprise solution instead.

…

On Mon, Jul 24, 2017 at 1:42 PM, Zev Goldstein ***@***.***> wrote: I have a theory that almost all companies that want an internal registry fall into one of two categories 1. Small enough to be ok with (and maybe prefer) simpler local file system storage 2. Big enough to set up (and maybe prefer) a general purpose repository manager like Nexus or Artifactory As a quick sanity check... *Please thumbs down this comment if your company doesn't fall into one of these two categories* If this comment doesn't get downvoted to oblivion, then maybe we should be thinking about whatever we build here as more of a simple reference implementation than a truly scalable solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#286 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD7CPGK8ne1gXj25kJS4UuSRRfBfQQ-ks5sRNeJgaJpZM4MTRv2> .

@SteelPhase

Big enough to set up (and maybe prefer) a general purpose repository manager like Nexus or Artifactory

I'm mostly trying to avoid putting my toe on the scales by staying out of this thread, but I feel the need to quickly note there that I think it is an error to equate the work @deejross and @SteelPhase are currently doing with the role served by e.g. artifactory or nexus. While there are certainly some possible areas of overlap, I'd say offhand there's at least as much that such a generic system is never likely to be able to provide.

Most importantly, running such a service need not be mutually exclusive with, and in certain configurations could even be nicely complementary to, a generic platform.

? Like what?

Of course, since I have no idea what this repository is supposed to offer as an API, I can't say for sure, but it's hard to see what's being built here as significantly different, abit a customized simplified version of those products.

You mean to say that the repository service being built offers an api that is more than just 'resolve this package' and 'download this package'? ...and perhaps, authenticate me? Like what? What 'go specific' features would it offer that something like nexus wouldn't?

I mean, I'm not really a fan of nexus at all, but I thought that this whole repo thing was about implementing basically a custom version of that to play with?

? Like what?

yeah, i should've known that chiming in would get this question 😄

i'll try to marshal my thoughts and compile a list of the relevant things, here.

but I thought that this whole repo thing was about implementing basically a custom version of that to play with?

FWIW, this has never been the frame from which i've been operating on this topic

S3 may work for some, but we don't use it in our corporate environment. We use an internal High Availability (multiple servers, failover, etc., etc.) Artifactory setup (which seems to be very popular with corporations in general) for binaries, and Git (and still a lot of CVS) for source.

Artifactory even comes with built-in facility to mirror GitHub areas, but ONLY the tarballs of the specific commits.

That's fine for keeping a version around, but it does nothing for actually using it during a Go build... we would have to provide extra tooling to retrieve the tarball from Artifactory, expand it into the 'vendor' subdir, and then run 'go build' or whatever.

I was really hoping for something a bit more transparent. In my playpen, I do 'go get ' and it goes out to the 'net and gets it. In the corporate build, whatever tool is used should have the same effect, but it should only get the package from the internal mirror, without me having to change URLs or anything else.

If dep can make that happen (however it does it... via a database, or whatever), we will be happy campers.

I think that the road to supporting a registry in dep can be quite simple by supporting a configurable, global http(s) URL to resolve from + an optional, separately configurable authentication key.
This can be done without changing the current model: 100% relying on DNS-based go imports as the canonical package identifiers, but not as the resolvable package URLs - these will be HTTP-delegated to the registry. Sources from the original URLs are automatically resolved and cached by the registry upon request.

As already alluded to in this thread, the registry will just have to contain frozen gz-tarballs for versioned dependencies, both for remote external dependencies and for locally deployed packages. Packages are usually stored in the registry in a similar way to how they are name-spaced in the 'vendor' directory, but with an additional tag/commit level. We successfully use the same approach in Artifactory for supporting CocoaPods, Composer, Bower and NPM dependencies - all based on source dependencies rather than binaries (npm has some exceptions, but the general concept still applies).
Happy to further help here!

Hi all, I have started coding the registry! As of now, it is very, very basic and only includes a couple of interfaces, but I hope it gets the idea of what we're going for across. Basically, pluggable binary and metadata storage with a REST API as the primary interface to the service. I haven't yet implemented any of the storage interfaces, but I will at least implement a local filesystem interface of some kind as a proof-of-concept. Additional backends will be easy to add after that.

If you see any potential issues with the skeleton I have out there now, please let me know: https://github.com/deejross/dep-registry

Thank you!

@sdboyer

@deejross @sdboyer Will this (at some point in planned, or at least not ruled out, future) support the following case:

user@company.com:/git/httprouter.git   - a mirror of github.com/.../httprouter
user@company.com:/git/internal-lib.git - internal utils, "company.com/utils"
user@company.com:/git/app.git          - depends on httprouter and internal-lib

app.git will contain some configuration that says that github.com/julienschmidt/httprouter imports in the source code can be found at user@company.com/git/httprouter.git with some commit/tag, and company.com/utils can be found at user@company.com:/git/internal-lib.git with some commit/tag?

Also.. and if I'm doing some development of company.com/utils library at the same time as developing app.git, I can temporarily change the configuration in my checkout of app.git to point it to the local path /path/to/dev/internal-lib?

This question, or at least similar ones, has been raised earlier in the comments, without any clear answer.

app.git will contain some configuration that says that github.com/julienschmidt/httprouter imports in the source code can be found at user@company.com/git/httprouter.git with some commit/tag, and company.com/utils can be found at user@company.com:/git/internal-lib.git with some commit/tag?

yep, these are definitely the sort of cases we're figuring out how to cover.

to point it to the local path /path/to/dev/internal-lib?

local paths end up being a very different question, entirely separate from registries. gotta deal with those separately. this gist has one possible longer-term view, and https://github.com/GetStream/vg may help you in the meantime.

How to use a Github Token using .netrc file

How to use a Github Token in .netrc file

1. Developer is writing code in playpen, and includes something like ‘import “github.com/Jordan-wright/email”’ 2. Code works well, so developer pushes code for review. 3. Review goes well, so code is submitted, and is picked up by nightly builds. 4. Nightly build runs on machine(s) with firewall rules which PREVENT all external access, so ‘github.com’ is not directly accessible. In this case, ‘deps’ would need to be able to resolve the import to an INTERNAL site, specified on the build system, preferably with multiple possibilities (command-line option, env var, etc.). Whether it did this by replacing ‘github.com’, prefixing it, or whatever is of no consequence. There just needs to be a way to make it resolve to an internal system transparently to the code. All transitive dependencies should work transparently as well, with explanatory error messages when something cannot be found/resolved. In other words, the code should work “as-is” with no modifications. This scenario assumes a completely separate process for getting packages approved and populated on the internal system. Stretch goals: Developers given a method to test what will happen when code is submitted for nightly builds. ‘deps’ supports a method of importing from external site, but uploading to internal staging area, independent of builds (like a common internal system separate from the developer’s playpen OR the nightly build machine(s)). From: wade lee [mailto:notifications@github.com] Sent: Wednesday, November 08, 2017 4:25 To: golang/dep <dep@noreply.github.com> Cc: Jamie Cooper <Jamie.Cooper@sas.com>; Mention <mention@noreply.github.com> Subject: Re: [golang/dep] Support for private/enterprise patterns (#286) EXTERNAL who can give me a example ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#286 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQmvXLoLwPmNFrHk2dlUCQkHKhYPGqZMks5s0XNagaJpZM4MTRv2>.

I believe I have a solution for private repos with ssh access, see PR #1717

Navigation Menu

Repository navigation

Support for private/enterprise patterns #286

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support for private/enterprise patterns #286

Description

Activity

sdboyer commented on Mar 6, 2017

thejerf commented on Mar 7, 2017

sdboyer commented on Mar 8, 2017

lancehit commented on Mar 8, 2017

sdboyer commented on Mar 8, 2017

aajtodd commented on Mar 8, 2017

steveklabnik commented on Mar 8, 2017

sdboyer commented on Mar 8, 2017

quentinmit commented on Mar 9, 2017

aajtodd commented on Mar 9, 2017

brianf commented on Mar 20, 2017

24 remaining items

brianf commented on Jul 24, 2017

sdboyer commented on Jul 25, 2017

shadowmint commented on Jul 25, 2017

sdboyer commented on Jul 28, 2017

jamie-at-sas commented on Jul 28, 2017

yoav commented on Jul 30, 2017

rosspeoples commented on Aug 3, 2017

dsvensson commented on Aug 15, 2017

sdboyer commented on Aug 15, 2017

jamie-at-sas commented on Nov 8, 2017

outcoldman commented on Feb 25, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions