[Proposal] Make an API for locale negotiation

[longlho]

Resolving from a list of requested locales (via Accept-Languages header for example) to a list of supported locales for a certain website. See vercel/next.js#18676 for example.

Existing locale matching libraries are fairly rudimentary (exact match only). We already have LookupMatcher/BestAvailableLocale/ResolveLocale abstract operations that take into account extensions as well so IMO we should make a top level API for this.

[zbraniecki]

The proper API for that is very non trivial. The listed APIs are deeply imperfect and the exact strategy especially among multiple locales may deeply depend on the business needs.

That makes me reluctant to "bless" one approach with a standardized API that we'll have to maintain. We already have localeMatcher which is a cautionary tale about how we thought it'll work best.

I'm not opposed to expose improved APIs to help with matching algorithms (parent locales, addLikelySubtags etc.), but I'd like to be cautious about providing a single solution as I don't think we found a golden one yet.

[longlho]

Would exposing top level API for just lookup localeMatcher be possible? IMO that would be a start.

[sffc]

Adding LanguageMatcher was first brought up in #46. So it's not a new issue; it just needs someone to champion it and help shepherd it through the process. As Zibi said, though, we need to be careful in this space, since there are multiple competing implementations of LanguageMatcher with different tradeoffs.

[longlho]

So is the next step to put together a formal proposal repo?

[sffc]

So is the next step to put together a formal proposal repo?

Yes; creating a proposal repo with a README explaining the problem space will enable us to continue exploring this space. The instructions are here:

https://github.com/tc39/ecma262/blob/master/CONTRIBUTING.md#new-feature-proposals

[longlho]

Does this work? https://github.com/longlho/proposal-intl-bestavailablelocale

[zbraniecki]

Does this work? https://github.com/longlho/proposal-intl-bestavailablelocale

Your proposal seems to be going counter to my recommendation. It does introduce an API that makes it look like it is the "canonical" language negotiation. Since I do not believe we have designed such a golden standard in the industry yet, trying to expose one via ECMA-402 will lead to either us having to support inferior algorithm forever, all while likely having to introduce another one one day (Intl.EvenBetterBestAvailableLocale?), or having to change the algorithm internally which may depend on other options or even additional data and make this API hard to advance.

Your proposal also does a negotiation to a single locale, which I think is rarely the right call. I explained my rationale here: https://firefox-source-docs.mozilla.org/intl/locale.html#language-negotiation

To sum it up, I would much rather see us exposing lower level components such as parentlocales, maximize/minimize etc. You have the right to push for standardization of some negotiation that you consider good enough to be added to ECMA-402 and supported forever, but I'd like my votum separatum to be explicit.
I do not believe that at the moment we should have an API with a name such as Intl.bestAvailableLocale or anything similar.

[longlho]

So right now this is already happening with all the Intl APIs that take in a set of locales so I'm not sure I understand the rationale against exposing locale matching as a top level API? It also intentionally expose localeMatcher as an option with best fit to still be an implementation-dependent option, which prevents locking to a specific algorithm.

Would you recommend a different name? I do believe at the moment we need some top level locale negotiation for i18n sites to serve the right translation bundle that is consistent with other formatters like date/time/number. We do have the use cases for it so can you clarify on further requirements?

[zbraniecki]

So right now this is already happening with all the Intl APIs that take in a set of locales so I'm not sure I understand the rationale against exposing locale matching as a top level API?

The BestAvailableLocale is an abstract operation that is internal to our API. It handles CLDR data specifically for a "final" matcher, and doesn't allow any nesting. It also works only because generally we expose the same set of locales for all formaters exposed by ECMA-402. I believe that we made a mistake when we standardized it the way we did, and we can do nothing about it now.

It's going to be a very tricky experience to try to merge this API shape with MessageFormat 2.0, which cannot handle language negotiation the same way as it has to handle a case where we have resources in one locale, but it requires a formatter like PluralRules which may have a different set of locales.
Exposing this algorithm to users as a canonicalized approach means that we recommend them to use it, for their needs. And I don't believe this algorithm to be good for common purpose needs.
In the user-land, the list of available resources will differ and the needs will often nest (list format that also formats its content cells and different cells will have different available locales).

Would you recommend a different name?

No, I just don't think we have a single API that we should expose.

I do believe at the moment we need some top level locale negotiation for i18n sites to serve the right translation bundle that is consistent with other formatters like date/time/number.

What makes it impossible to implement it in user land? We have supportedLocalesOf which allows you to perform any nesting you want with merging filtering against locales available for a given formatter, we have maximize/minimize. What is missing?

We do have the use cases for it so can you clarify on further requirements?

I believe that your questions for me are misguided by the assumption that we should add APIs for everything that we can find "use case" for.

I don't think "have a use case" is a good justification. We have a use case for localization as well, but we don't have a good format to offer, so we don't standardize something just because we can.

I don't have requirements, I have an opinion. There is a large spectrum of possible algorithms for language negotiation, and I believe for now, ECMA-402 should restrain from "blessing" any. Instead, we should make sure that we lower the barrier for user land algorithms to be written.
If there are missing low-level APIs that are data heavy and in result unfit for user-land, then I believe we should consider them.

My recommendation would be to develop what you're aiming for as a user-land library and see if it gains universal adoption. (here's an example of one we use for Fluent)

What prevents you from doing that?

[zbraniecki]

As a possible (long term) path forward:

• @mihnita is currently working on designing a coherent vision for language matching/filtering/negotiation for ICU4X
• Referential issues:
  • Locale of constructed object & language negotiation in data loading unicode-org/icu4x#173
  • Design a cohesive solution for supported locales unicode-org/icu4x#58
• I expect that based on that we'll get a proposal for @sffc call's "horizontal" vs "vertical" language negotiation
• This will be proposed to Unicode and we'll get feedback on it from Unicode experts who manage things like BCP47 and RFC4647
• But I also expect it to be much more comprehensive and flexible in terms of "fallbacks" handling and fallback chains
• Once this settles, and we validate it in ICU4X we can implement a user-land version for JavaScript and recommend it for all major users such as momentjs messageformat-js etc.
• After some time, if we feel that it works well and is battle tested, we could revisit standardizing something like what you're looking for.
• This would then fit well into Adding guidelines for proposals in CONTRIBUTING.md #512 passing all three checks for a good candidate for ECMA-402 inclusion.

[longlho]

I guess I'm basing the requirements off of #512 which this seems to satisfy:

1. There are prior arts for locale negotiation.
2. It is difficult to implement in user land due to CLDR data (e.g alias, language matching, parent locales) and also the LanguageMatching algorithm is certainly tricky.
3. This does have broad appeal, as demo'ed in the next.js GH issue (57K stars). The implementation we're going with is within @formatjs/ecma402-abstract (320K weekly downloads but that number doesn't matter as far as I'm concerned).

I'm not asking for blessing of 1 specific algorithm, but rather a top level API that allows you to pick one. We already have precedent with lookup algorithm which is https://tools.ietf.org/html/rfc4647#section-3.4

Going by the list of requirements I'm not sure what actionable things are missing so I can go figure them out?

[longlho]

@zbraniecki @sffc ok sounds like #512 needs to be updated with something to capture this conversation, since IMO this checks all the reqs but still doesn't seem to qualify.