Additional OAuth2 Authentication for Hosted Repositories

[vin-fandemand]

Business Case

As developers develop more packages in dart and Flutter, they might want to keep some of the packages as private for IP purposes. Hence the pub should be able to fetch both from the public 'pub.dartlang.org' as well as any hosted server that is secured by Oauth2 (so that only their engineers can get or publish the packages). I believe this ability is also important to foster open source projects as it provides companies with more flexibility to invest in dart and 'Flutter`.

Proposal

Add additional Authentication configurations for hosted urls.

• So a configuration file needs to be added at the root or in the cache root directory with the name <HOSTURLNAME>_config.json with parameters identifier (pub client's OAuth2 identifier), secret (pub client's OAuth2 secret), authorization and token endpoints, scopes, useIdToken (to decide whether to send id token or access token to send in Authorization header), and a URL to redirect once the authorization is successful. This is exactly same as how it is handled for 'pub.dartlang.org'.

Changes

1. `oauth2.dart' - Changes to implement Oauth2 cycle for hosted URLs and also cache the credentials.
2. 'hosted.dart' - Changes to implement adding access token or identity token in Authorization header.
3. lish.dart' - to pass the hosted URL from the serveroption tohosted.dartandoauth2.dart.`
4. https://github.com/dart-lang/oauth2/pull/83 is required for the changes to work. So the oauth2 library was added in the folder temporarily. (CI is breaking as this is missing in the changes)

Help Required

I really believe in the value this would provide to the community. I have tested scenarios manually. However, I am unable to run the dart test as it says some required files are missing.
I also have issues with versions. It always gets the latest version and not the one in pubspec.
So if all looks good and these changes are acceptable, can you please help me set up the testing correctly and also suggest any new test cases, and also how do I contribute to the documentation of these changes? I am expecting these changes to flow to Flutter SDK so that I can use them with Flutter. Please let me know if there are any queries.

[jonasfj]

Sorry for the late reply, I'm just catching after vacation.

See discussion in:
#1381 (comment)
and
#2167

If you are willing to work on working on this, I'm happy to help designing, reviewing and landing a solution.

I agree, that solving authentication for third-party pub servers is important. I know a few organizations that currently have a custom pub server and rely on network security by having the pub-server behind a VPN. But this hardly ideal.

---

A few notes here:

• I think using oauth2 is way to complicated, most third-party pub-servers would probably prefer to simply have to user login on the website and obtain an opaque secret token.
• For pub publish it is obvious that we need authorization of some sort. For pub.dev / pub.dartlang.org using oauth is neat, but for other servers asking the user for an secret token might be the easiest solution.
• For pub get not all pub-servers are likely to require authorization, hence, it might be preferable to simply make a request and if we get a 401 response with a www-authenticate header, then we prompt the user for a secret token.
• The secret tokens could be stored in PUB_CACHE/secrets.json or similar. Having the hostname in the filename could cause problems if the hostname contains characters that isn't allowed in filenames.
• We would need this integrated with pub logout and ideally also a pub login command (Login command #2479) :D

---

I'm happy to continue the discussion here, but if you're interesting in working on the approach outlined in #1381 please leave a comment at the end of the thread, and file a WIP PR as soon as you're getting started :)

[vin-fandemand]

Sorry for the late reply, I'm just catching after vacation.

See discussion in:
#1381 (comment)
and
#2167

If you are willing to work on working on this, I'm happy to help designing, reviewing and landing a solution.

I agree, that solving authentication for third-party pub servers is important. I know a few organizations that currently have a custom pub server and rely on network security by having the pub-server behind a VPN. But this hardly ideal.

A few notes here:

• I think using oauth2 is way to complicated, most third-party pub-servers would probably prefer to simply have to user login on the website and obtain an opaque secret token.
• For pub publish it is obvious that we need authorization of some sort. For pub.dev / pub.dartlang.org using oauth is neat, but for other servers asking the user for an secret token might be the easiest solution.
• For pub get not all pub-servers are likely to require authorization, hence, it might be preferable to simply make a request and if we get a 401 response with a www-authenticate header, then we prompt the user for a secret token.
• The secret tokens could be stored in PUB_CACHE/secrets.json or similar. Having the hostname in the filename could cause problems if the hostname contains characters that isn't allowed in filenames.
• We would need this integrated with pub logout and ideally also a pub login command (Login command #2479) :D

I'm happy to continue the discussion here, but if you're interesting in working on the approach outlined in #1381 please leave a comment at the end of the thread, and file a WIP PR as soon as you're getting started :)
Thanks for the comments. Hope you had a cool vacation.

Sure. I am happy to help. I started with these changes as I needed Oauth2 for my firm's requirement. I would like to keep Oauth and extend it to tokens. Tokens and other information could then be stored in secrets.json as you recommended. However, I didn't understand the one with login and logout. My understanding was that we would require to log in only if we are publishing or getting anything for a repo that requires authentication. Also, I would like some version of this in production sooner so that I can use it for my firm.

We also need to consider how to sign in in CI/CD environments, silently without needing a sign in the authorization.

I am happy to continue working on this. Can I convert this PR into WIP PR?

[jonasfj]

We also need to consider how to sign in in CI/CD environments, silently without needing a sign in the authorization.

I thinking was that using tokens would solve this problem.

I started with these changes as I needed Oauth2 for my firm's requirement.

Really, you would like to be using oauth2.. Hmm..

[vin-fandemand]

We also need to consider how to sign in in CI/CD environments, silently without needing a sign in the authorization.

I thinking was that using tokens would solve this problem.

I started with these changes as I needed Oauth2 for my firm's requirement.

Really, you would like to be using oauth2.. Hmm..

My team is using GCP. I am hosting our pub server as a service on Cloud run. So I use the oauth2 to make sure a developer in my team who has been given access to our GCP resources is permitted to publish and get packages. This way I figured that I would just reuse what gcloud SDK would have a developer do otherwise. Maybe this is not the right approach.

[jonasfj]

My team is using GCP. I am hosting our pub server as a service on Cloud run. So I use the oauth2 to make sure a developer in my team who has been given access to our GCP resources is permitted to publish and get packages.

hmm, I have considered letting 3rd party pub servers use oauth2, well, openid-connect (OIDC) really..

I'm no OIDC expert, but I suspect that something like OIDC Discovery could be used to find issuer and OIDC metadata.
TL;DR: maybe a flow could look like:

• Try to make an unauthenticated request, upon a 401 response do:
• Look in PUB_CACHE/secrets.json if we have ODIC metadata and credentials for given hostname.
• Prompt the user for a username
• Use issuer discovery to find the issuer URL.
• Use OIDC configuration discovery to find the OIDC server configuration.
• Run an OIDC login flow against the server configuration.
• Authenticate requests with id_token and store server configuration and credentials in secrets.json.

I'm not entirely sure if the above is how OIDC discovery is intended to be used. The "Resource" ("Entity that is the target of a request in WebFinger.") is a bit weird and seems unnecessary to me. But I suppose maybe the spec assumes that users could be on different servers.

In any case, this seems VERY complicated. If someone wants to investigate these specs and figure out the right way to do this I'm very interested. But even with all that, this wouldn't work Google OIDC, because while Google does serve the OIDC configuration metadata, Google still requires your app to be registered, and Google requires there to be a secret token (for the oauth2/oidc flow).
So the pub client can't magically discover the secrets necessary to connect to Google OIDC by fetching /.well-known/openid-configuration from accounts.google.com.

Otherwise, the idea that a pub server would simply point to a server that has a /.well-known/openid-configuration file, and then the pub client simply does a login flow in compliance with the configuration from the /.well-known/openid-configuration file is very attractive.

Disclaimer: I did not read the OIDC specs in detail, the above might be total nonsense.

This way I figured that I would just reuse what gcloud SDK would have a developer do otherwise.

I'm not sure what this means. Can you elaborate?

---

Maybe we could tweak the flow a bit... Imagine a pubspec.yaml as follows:

name: mypkg
version: 1.0.0
# this is the key that makes it publish to a third-party pub server
publish_to: https://mypubserver.com 
dependencies:
  retry: ^3.0.0 # get package retry from pub.dev
  mypkg_helper: # Package from my own pubserver
    version: ^1.0.0
    hosted:
      name: mypkg_helper
      server: https://mypubserver.com 

(this actually works today, it's just the authentication that isn't working right).

Now, imagine I do: pub publish we could imagine 4 scenarios:

(A) PUB_CACHE/secrets.json has a secret for mypubserver.com
In this case pub knows from PUB_CACHE/secrets.json that publishing to mypubserver.com requires a secret, and pub already has the secret :)
So the publishing request is made with the secret from secrets.json as bearer token.

(B) Prompt for secret

• User invokes pub publish
• PUB_CACHE/secrets.json has no secret for mypubserver.com
• pub attempts to publish without authorization token.
• Server responds 401 with www-authenticate: <custom-message> header.
• Pub prints <custom-message> from the www-authenticate header.
• Pub prompts the user for a secret token.
• User enters secret token.
• pub attempts to publish with secret token as bearer authorization token.
• Pub stores secret token in PUB_CACHE/secrets.json if publishing was successful.

(C) Login before publishing

• User invokes pub login mypubserver.com before trying to do pub publish
• User is prompted to enter a secret token.
• Pub stores secret token in PUB_CACHE/secrets.json.
• If users now tries to do pub publish pub will publish with the secret token as bearer authorization token (same state as in (A)).

(D) Publish with PUB_AUTH_PROVIDER

• User sets the env var PUB_AUTH_PROVIDER='/usr/local/bin/my-pub-auth-provider.sh'
• User invokes pub publish
• PUB_CACHE/secrets.json has no secret for mypubserver.com
• pub detects that env var PUB_AUTH_PROVIDER is not-empty.
• pub invokes /usr/local/bin/my-pub-auth-provider.sh mypubserver.com, this script exits zero, and prints a secret token to stdout.
• pub attempts to publish with secret token as bearer authorization token.

In this case the idea is that PUB_AUTH_PROVIDER points to a script that given a hostname as input, either: prints secret tokens and exits zero, or prints nothing and exits non-zero.
This way, advanced systems that have temporary credentials and what not can simply make a script that obtains the secret token and configure PUB_AUTH_PROVIDER to point to said script.

That would allow arbitrarily complex authentication schemes for the paranoid people :D
Maybe your pub server uses a GPG signed timestamp, or a TOTP token from a yubikey, options are endless, hehe :D

I personally think that (A), (B) and (C) will cover most common use-cases, and are rather easy to understand and use for the end-users. I'm less sure about (D), I suppose there are many different ways (D) could be configured too. An environment variable is just one way that could be used to inject an authentication provider.

---

We also need to consider how to sign in in CI/CD environments, silently without needing a sign in the authorization.

This is part of the reason I figured that simple static tokens would be preferable for 3rd party pub servers.

---

However, I didn't understand the one with login and logout. My understanding was that we would require to log in only if we are publishing or getting anything for a repo that requires authentication.

Your understanding is correct. We would only login when a repo requires authentication, and ideally we would detected this by trying and receiving a 401 response with www-autheticate header.

However, once logged in, the pub command should store the secret in a file in the PUB_CACHE... that way you won't need to login the next time.

And with secrets stored in the PUB_CACHE, it might be nice with a pub logout command that just deletes the secrets :)
(in case you want to remove the secrets)

I could also imagine a pub login [server] command, that allows you to perform a login, without first trying to do something that requires authentication. If nothing else this would be useful for debugging the login flow, and/or restoring state after pub logout.

These are more nice to have features that makes the authentication stuff feel more complete. You can also imagine that pub login without any arguments, would display what servers you are logged in with (ie. what servers you have secrets for in PUB_CACHE/secrets.json).

[jonasfj]

Hmm, thinking more about this maybe (D) is way too complicated and unnecessary at this time.
Perhaps it would be better to tweak the server and use option (B) as follows:

(B, example 2) Prompt for secret

• User invokes pub publish
• PUB_CACHE/secrets.json has no secret for mypubserver.com
• pub attempts to publish without authorization token.
• Server responds 401 with www-authenticate: "Open https://mypubserver.com/pub-login to obtain a secret token." header.
• Pub prints Open https://mypubserver.com/pub-login to obtain a secret token. from the www-authenticate header.
• Pub prompts the user for a secret token.
• User clicks on the https://mypubserver.com/pub-login link in their terminal and opens it in their browser.
• User performs a login on https://mypubserver.com/pub-login (using oauth2 or whatever login flow desired).
• Server creates a secret token for the given user, stores the token in a database and displays it to the user.
• User copies secret token, into the pub prompt on the command line and hits enter.
• pub attempts to publish with secret token as bearer authorization token.
• Pub stores secret token in PUB_CACHE/secrets.json if publishing was successful.

Notice that, that:

• The server can make the user login using whatever mechanism it desires, oauth2, cookie, password/username, third-party SSO, anything it can do in a browser. If the user is already logged in on the server, the user may already have an active cookie session.
• The server can store the secret token in it's database, hence,
  • The server can show the user a list of tokens that the user has created,
  • The server can allow users to revoke tokens they previously created,
  • The server can expire tokens as they become old or haven't been used in a long time.
• The server can display the token in an easy to copy command like: pub login mypubserver.com --token <token> (which the user can just copy to CLI)
• If you want a stateless authentication token the server can just return a signed JWT that expires in 31 days, obviously this is not very secure as those tokens can't be revoked.

But by allowing the server to inject a short custom message through the www-authenticate header, and by guiding the user towards authentication through a web browser, the 3rd party pub server can use pretty much whatever authentication scheme is desired.

The only downside, is that the user will have to copy the token from the browser window to the terminal.

This could be automated by letting the www-authenticate be a URL to which the pub client attaches a callback URL like: www-authenticate: https://mypubserver.com/pub-login, then the pub client would direct the user open https://mypubserver.com/pub-login?callback=http://localhost:<random-port>.
And then the pub client listens on localhost:<random-port>. Hence, the server can automate the login flow by just having the user click an "Authorize" button and then redirect the user to localhost:<random-port>?secret=<secret-token>. This isn't overly complicated, but there are many ways to exploit this, which is why oauth2 is complicated and has various callbacks, secrets and random tokens :D

Having to copy a token from the browser window to a terminal prompt isn't pretty, but it works, and it's easy to understand.

[vin-fandemand]

sorry, the latest comment got lost in my mails. Sure, I will look into B) this weekend.

How do the contributions work usually ? should I just keep pushing proposed changes and you would review it?

[jonasfj]

How do the contributions work usually ? should I just keep pushing proposed changes and you would review it?

We can do that... Are we in agreement that we should just stick to (A) and (B), maybe (C), but not (D) or anything complicated with oauth2?

If so I think it would be a great idea to start writing up a PR. Please file WIP PR, as you're making progress, and feel free to ping me with any questions. Ideally we want to do some reviews along the way. I honestly have no idea how complicated this will turn out to be.

[stefandevo]

I really think we need to start with a simple token/secret at first for both get and publish. This token needs to be taken into the api as a header value. The hosted server should pick it up there to continue.
To obtain the token, that is something that pub tool does not need to do at first in my opinion.
Check how nuget is working. They have a file that contains the custom feeds together with their login secrets and uses that.

https://docs.microsoft.com/en-us/nuget/consume-packages/consuming-packages-authenticated-feeds

Important thereby that this just works in ci environments. It is just a file that is placed in the same location as pubspec so it can be under source control.

[jonasfj]

I really think we need to start with a simple token/secret at first for both get and publish. This token needs to be taken into the api as a header value. The hosted server should pick it up there to continue.

Yes, (A) and (B).

To obtain the token, that is something that pub tool does not need to do at first in my opinion.

I think the pub client should manage storage of secrets, and prompt for them when they are necessary.
I suppose that at first (A) the pub tool could just read a $PUB_CACHE/secrets.json file, but that's not very ergonomic.

Important thereby that this just works in ci environments. It is just a file that is placed in the same location as pubspec so it can be under source control.

I agree, working under CI is important. But storing credentials under version control sounds very wrong to me.
One should never check secrets into a git repository (unless they are encrypted, but even that has drawbacks).
(Am I missing something)

For CI, maybe we can solve it, by allowing injection of secrets through environment variables or global config files.
If we implemented (C) you could also solve it by running pub login mypubserver.com $ENV_VAR_WITH_SECRET in CI, and using the mechanism your CI system provides for injecting the environment variable ENV_VAR_WITH_SECRET with a secret for mypubserver.com.

Maybe we should start with (A) and (B), move on to (C) and eventually extend (B) to also automatically detect the need for secrets when running pub get.

[stefandevo]

I think the pub client should manage storage of secrets, and prompt for them when they are necessary.

I think it's a good idea in the future. Prompting will be complicated, because it would depend on the auth mechanism used. With oAuth it is clear, but other mechanisms are not that simple. Basic Authentication, Api Key authentication, Token authentication, etc. How would you prompt? By asking in clear text the values? (see note hereunder).

I agree, working under CI is important. But storing credentials under version control sounds very wrong to me.
One should never check secrets into a git repository (unless they are encrypted, but even that has drawbacks).
(Am I missing something)

Exactly. As with Nuget, you can choose to store your credentials via clear text (not preferred) and by environment variables.

So think we should define a json structure that can be used both global and local (in the repository) based upon the url of the package server.

Is there a way here on the repo to start working on the specs first? Should we create an issue for this, or something else?
I think it's important to define everything before start working on it. I could work on this, but would need consensus on all the steps first.

[jonasfj]

@stefandevo

Is there a way here on the repo to start working on the specs first?

Filing an issue is fine. A PR with a markdown file in doc/ is not a bad choice either..
Perhaps the easiest is to just share a google doc with me ([email protected]) and we can discuss the various topics there and move the solution into an issue for additional feedback from others.

With oAuth it is clear, but other mechanisms are not that simple.

How is oAuth clear? will the user have to supply the oauth2 OIDC configuration for the server? Are do we automatically detect this?

I might be missing some obvious solution here.

(The oauth2 token used to authenticate against pub.dev can't be used to authenticate against other servers, as those servers would then be able to impersonate the user on pub.dev)

Prompting will be complicated... How would you prompt?

I imagine that the simplest option is to ask users to just enter a <secret>, then authenticate requests with Bearer <secret>.
Whether users choose to use a plain password, randomly generated token, temporary tokens or a signed JWT is all up-to the user.

I agree these solution is a bit naive, but it's simple.

---

The only other elegant auth-flow I can think of is generating a random <uuid> in the pub client, then asking the user to open: https://myprivatepubserver.com/api/register?token_hash=<hash(uuid)> in a browser, login using whatever login flow the server has, and click the "authorize token" button (obviously servers must support this).

Then the client could authenticate with https://myprivatepubserver.com using Authorization: Bearer <uuid>, as the server will be able to compare the hash of the <uuid> to the hash previously "authorized" by the user.

But I think this might be a bit too complicated, and I'm not 100% sure about the security of such a scheme.

[stefandevo]

Perhaps the easiest is to just share a google doc with me ([email protected]) and we can discuss the various topics there and move the solution into an issue for additional feedback from others.

I will contact you through email as I already did an implementation and would like to discuss.

[stefandevo]

I did the necessary changes on this PR #2654
I does not support oAuth2 but from the conversation here, this is something that's maybe too difficult, and might be added later.

[sigurdm]

Closing in favor of #3007