Support authenticating pub against 3rd-party servers

[kevmoo]

We've discussed something like pub authenticate example.com

This would allow private pub repos to have authenticated access.

We'd also want authentication to be supported for pub get/update

CC @computmaxer

[donny-dont]

Any idea what authentication schemes would be supported?

I'm moving speakeasy into its own org, https://github.com/speakeasy-pub/speakeasy, and would be very interested in this conversation.

I've had success at work with https://github.com/rlidwka/sinopia for NPM and that has multiple authentication backends. Open to doing something similar with speakeasy.

[computmaxer]

My understanding is that they would extend the same OAuth2 functionality that already exists for the pub publish command.

[nex3]

That's right. The plan is to add a pub authenticate HOSTNAME command that will send the user through the OAuth2 authentication process, eventually providing a token that the given host can use to verify the user's google account. This token will be stored in the user's pub cache, and sent along with any requests to that server in the course of version resolution.

[dynaxis]

For small organizations like us, it's really beneficial to use Google's identity management. And I'll be happy with it. But just of out curiosity and for the larger organizations than us requiring use of their own id management, does it require much effort to implement support for oauth2 id service other than Google's?

[nex3]

There are some technical constraints that make it difficult. OAuth2 requires that the client—in this case pub—have a client ID and secret provided to it out-of-band by the identity provider. We already have such an ID and secret for Google built in to pub, but getting it out-of-band for arbitrary identity servers is harder. Not impossible, but not necessary for what we're doing here either.

[bjornm]

Maybe a first step could be to enable authentication for pub get using the same authentication mechanism that is already used for pub publish? E.g.

export PUB_GET_AUTHENTICATION_ENABLED=true
pub get
# will use ~/.pub-cache/credentials.json just like pub publish

We run our own dart pub package server and we are fine with using google oauth for pub publish. Our server verifies the authorization token against Google. The returned user is checked against a list of allowed uploaders. Adding pub get authentication would give us what we need to protect the repository from unauthorized users.

[nex3]

We can't send users' pub.dartlang.org auth tokens to other servers; it would allow any server that they download packages from to spoof them and upload packages on their behalf.

[bjornm]

What if pub get tokens could be restricted to PUB_HOSTED_URL so it is only on intent that it is used?

Similar to how pub publish behaves right now. It is sending the pub.dartlang.org auto tokens to our server since we override it:

export PUB_HOSTED_URL=https://...

That also opens up for spoofing in a sense. Not that it matters to us because we are completely aware of this and trust our own pub package server.

[nex3]

I'm not comfortable assuming that all users will be completely aware of what they're doing in that case. The only way I'm willing to do this is if each hosted URL has its own credentials.

[bjornm]

Ok, makes sense.

[trentgrover-wf]

any movement on this @nex3 @kevmoo ?

[nex3]

Not yet. I'm hoping to budget time for it next quarter.

[computmaxer]

Any update on this?

[kevmoo]

We still want to do this – but we likely won't get to it until early 2017.

[thosakwe]

I'm willing to help out with this.

[kevmoo]

We'd love help here – but it's critical we have a design that's approved upfront.

If you're interested, @thosakwe ...

[thosakwe]

I have a fork of pub called "re:pub" that supports basic auth, in addition to Google OAuth2.

Perhaps users could store their basic auth credentials in a file, which could be read as follows:

# .pub-auth.yaml
username: jdoe1
password: foobarbaz24

And then, run pub with an argument to point to the file. This is assuming the user has set a "PUB_HOSTED_URL."

pub publish --auth-file .pub-auth.yaml

And then, perhaps users could optionally specify a host in the YAML file, so they wouldn't have to set an environment variable every time.

This design would make it possible to use dependencies from public pub, as well as publishing to a private host.

However, there would need to be some sort of distinction made between private and normal dependencies. I think that in the "pubspec", private dependencies would have an extra attribute added, whereas normal ones would remain the same.

# pubspec.yaml
dependencies:
  string_scanner: ^1.0.0 # normal
  in_house_package:
    hostname: https://pub.my-private-company.com
    version: ^0.3.5

Let me know what you think!

[nex3]

I'm not a big fan of adding a bunch of extra configuration files. We already have .pub-cache/credentials.json as a global config file, so I think any new configuration should go in there. I'd like the user's interaction with it to all go through the pub executable—for example, pub auth add http://example.org could authenticate the user with that domain.

We already have means of depending on hosted packages from arbitrary domains, and of specifying what domain a package should be published to. I don't think we need to add extra functionality for that.

[donny-dont]

Currently speakeasy does caching of packages from pub.dartlang.org.

NPM has scoping which lets you easily target multiple registries if you need to. Not sure how pub would do that at the moment.

[thosakwe]

I'm not sure how to handle scoping, either but...

In response to @nex3: On second thought, adding additional credentials to the .pub-cache/credentials.json is better all-around.

I feel that it might be easier to support Basic Auth for third-party hosts. Otherwise, every third-party host would have to implement the OAuth2 spec, and Pub would have to remember a new Client ID and Client Secret for each host.

Perhaps Pub could support an HTTP header, X-Pub-Auth-Required , or something similar. If the server returns a 403 on a request, and X-Pub-Auth-Required is set to basic, then Pub would prompt the user for a username and password, and retry the request. If another 403 is thrown, then Pub would abort the request. In this case, the server could send another HTTP header to describe the authentication error, X-Pub-Auth-Error: Incorrect password. Otherwise, a default error message could be shown.

Just a thought. I do think this would work, though.

[nex3]

I have usability concerns about supporting basic auth. Either we'd have to make users authenticate for every invocation of pub, which is a very substantial usability burden; or we'd need to store the user's raw credentials in credentials.json, which isn't a good security practice and may not be clear to users, since we don't store pub.dartlang.org credentials that way. One of the major benefits of a system like OAuth2 is that the stored information doesn't make the user's password visible, and I'd be unhappy to give that up.

[thosakwe]

Ah, I understand. So in that case, would it be difficult to store a client ID/secret, as well as an access token for each third-party host?

[SCKelemen]

Is any work currently being done on this? Could someone recommend a solution for private packaging?

[thosakwe]

Currently the best solution is to use private Git repositories.

[travissanderson-wf]

Private git repositories work pretty well, the primary shortcoming is that pub can't do version range resolution with those. Until auth is added, we are running our private pub server with local network access only. We have a huge selection of private libraries so range resolution is a must at this point.

[SCKelemen]

I have a private repo for my components. Is it possible to use pub to retrieve it though? If not, how would I references a directory or repo?

[thosakwe]

dependencies:
  my_package:
    git: git://url

You can also use an HTTPS URL there, i.e. Github

[travissanderson-wf]

Note that this currently only works for one package per repo. I think there is an improvement coming in the next SDK to let you define a sub-directory.

[jonasfj]

Yeah, having an API end-point fetching the client secret, in combination with /.well-known/openid-configuration is tempting... I haven't studied the spec in detail, but I think a client secret is still needed at least for Google.. maybe it's possible to require that servers implement openid-connect without requiring a client secret? And maybe the Google flow could work by having our server redirect to that, so only it knows the client secret...
(These ideas are not fully thought through)

In any event, I worried we're over-engineering this..if we could do the command interface such that we can add support for an automated login flow later that might be nice.

[noahbetzen-wk]

I made dart-archive/pub_server#39 not realizing that this issue had been made. This would be a super useful feature.

[thosakwe]

FWIW, I didn’t realize it before, but it’s possible to include credentials in the PUB_HOSTED_URL, and use that for Basic auth, etc.

https://pub.dartlang.org/packages/mpp

The downside, of course, is including the credentials in an environment variable.

I think I might get started on a PR for patching the credentials.json format...

[jonasfj]

I've discussed this IRL with a few people and consensus is that:

• pub should use an opaque token for authentication with 3rd party servers.
• The opaque token is attached as authorization: bearer <token> in authenticated requests.
• An opaque token is stored for each hostname in $PUB_CACHE/tokens.json (other than pub.dartlang.org and pub.dev, which will use credentials.json).
• pub should prompt users for this token on the command line interface, if not present in $PUB_CACHE/tokens.json.

This means that pub.dartlang.org and pub.dev will have special treatment, where as for all other pub servers the flow will be as follows:

1. User starts command that requires authentication (e.g. pub publish).
2. If $PUB_CACHE/tokens.json doesn't contain a token for the pub server (identified by hostname), then:
   • Prompt user for a token (asking the user to obtain this from the server out of band).
   • Associated token with hostname for server in $PUB_CACHE/tokens.json.
3. Use token associated with hostname for server from $PUB_CACHE/tokens.json.
4. Send request associating token using authorization: bearer <token>.

@thosakwe, if you're interested in working on this let me know, I'll be happy to answer questions around the design, and help with reviews. This is definitely a contributable features and I think we want it :)

[sigurdm]

Consider: attempting without a token first, and when receiving a 401 ask the user for a token, with a message from the server's reply.
This allows a custom server to guide the user to obtain a token.

[thosakwe]

Rolling in this branch: https://github.com/thosakwe/pub-1/tree/external-auth
Related pull request (currently a draft): #2167

[themisir]

Why this is taking so long? Is there any ETA or something planned for this issue?

Just wondering if there's any issue that' blocking this one. If not I'm planning to implement it.

[tusharojha]

Though @jonasfj knows it better, I guess it will be solved by this summer.

[jonasfj]

knows it better, I guess it will be solved by this summer.

We know today that this will not be part of GSoC 2021.

If someone wants to work on it anyways, you are welcome.. maybe drop me a line in hackers-pub-dev-

[themisir]

I've prepared a draft proposal doc for hosted pub authentication. Looking forward for your feedbacks.

https://github.com/TheMisir/pub/blob/master/doc/authentication-proposal.md

[IchordeDionysos]

@themisir nice work, I like it 😍
I have two questions:

1. What should the server send for the realm
2. Do you see an option to replace the "Enter Bearer" dynamically based on the provider?
   For example for Github packages:
   "Enter your Github personal access token:" is much more user friendly :)

Not sure if there is a Standard, that we can utilize for the second issue 🤔

[themisir]

First of all the proposal is based on RFC 7235. Because I think it will be much easier to implement authentication to exists un-protected pub servers just by setting up reverse proxies like NGINX which has built-in support for authentication that uses given RFC specs.

1. What should the server send for the realm

realm parameter is optional (is not required to be defined) and currently is not planned to be used in this proposal. Here's more details about "Protection space (relam)" if you want to read more: https://datatracker.ietf.org/doc/html/rfc7235#section-2.2

2. Do you see an option to replace the "Enter Bearer" dynamically based on the provider?
For example for Github packages:
"Enter your Github personal access token:" is much more user friendly :)

It's possible, but it might add unnecessary additional complexity to the implementation, so I have not included it in proposal doc. We might replace prompts by providing additional parameters in "WWW-Authenticate" header like:

WWW-Authenticate: Bearer prompt="Enter your Github personal access token:"

[jonasfj]

@themisir we'll still need special handling for pub.dartlang.org and such...

I would also like it if it was possible to make requests with a one-time token, like:
dart pub publish --token <token> or maybe using an env var..
This could be useful when publishing from CI systems.

Also we could consider dropping basic auth initially. It might not be necessary, and it certainly would make it harder to do one-time authentication as propose above.