Broken ParamConverterProvider ordering in 2.26

[chkal]

It looks like there are a few issues regarding provider ordering in 2.26. Although I think older versions are also affected.

I created a minimal sample app to reproduce the issue: https://github.com/chkal/jersey-prio

Basically I'm trying to register a custom ParamConverter which overrides the default converter for the data type java.lang.Integer.

First issue:

The first issue is that whether the custom provider or the default provider is used randomly changes between deployments. That's caused by the use of a HashSet instead of a LinkedHashSet in ParamConverterFactory. My pull request #3669 should fix this.

Second issue

The second problem is more difficult to reason about. After merging #3669 you will see that Jersey will always pick the default provider and ignores the custom provider deployed with the app.

I'll try to explain what I debugged so far. Actually the code in the ParamConverterFactory tries to handle the case of custom providers overriding default providers correctly:

https://github.com/jersey/jersey/blob/4ecf20c7585abfe1bd2c312d890943e884a9eb3b/core-server/src/main/java/org/glassfish/jersey/server/internal/inject/ParamConverterFactory.java#L74-L81

The weird thing is that providers deployed with the app always end up in the providers set in the constructor and NOT in the customProviders set. The ParamConverterFactory is created here:

https://github.com/jersey/jersey/blob/4ecf20c7585abfe1bd2c312d890943e884a9eb3b/core-server/src/main/java/org/glassfish/jersey/server/internal/inject/ParamExtractorConfigurator.java#L68-L71

If you dig deeper into the Provider.getProviders() and Provider.getCustomProviders() code, you will see that getCustomProviders() will look for providers with a qualifier Custom. I'm not sure under which circumstances providers get this qualifier.

Also noteworthy: Only getCustomProviders() seems to apply the ordering algorithm specified in JAX-RS 2.1 (see jax-rs/api#538). As custom providers aren't found by this method, the ordering algorithm isn't applied at all.

I hope this description helps to debug the problem. Let me know if there is anything I can do to help with this.