Integration Quality Scale: Platinum Quality Control

[AJediIAm]

Link the the original discussion
Link to the quality scale documentation
Link to the quality scale developer docs
Link to the platinum integrations

Problem description

The Home Assistant integration quality scale, particularly the Platinum tier, was established to signify the "best of the best" – integrations that exemplify excellence in stability, functionality, user experience, and code quality. Currently, there are several integrations within the Platinum tier which may not consistently meet this "best of the best" standard.
Integration might not be perfect in one or more of the following areas:

• Setup and authentication process
• User experience
• User documentation
• Technical excellence
• Code quality and code annotation
• Reliability and robustness

Not maintaining the platinum quality scale can:

• Diminish the trust and value of the Platinum tier for users and developers.
• Showcase subpar integrations as examples for developers to follow.
• Result in a "drift to the top".

Note: This is a sensitive topic. Every home assistant developer deserves the highest and praise for their hard work, regardless of the quality scale. If integrations need to be mentioned as examples, please do so respectfully.

Note: The scope of this proposal is limited to Platinum tier.

Goals of the Proposal

• Re-establish Platinum as a definitive mark of the highest quality integrations.
• Give proper recognition for all integrations in the platinum tier.

Understanding the Original Intent

The Platinum scale is described as the following:
Platinum is the highest tier an integration can reach, the epitome of quality within Home Assistant. It not only provides the best user experience but also achieves technical excellence by adhering to the highest standards, supreme code quality, and well-optimized performance and efficiency.

The original proposal includes an extensive list of technical criteria an integration needs to adhere to.

The subjective quality criteria such as user experience and code quality are mentioned, but do not have a well-defined test criteria.

Options for improvement

1. Quality scale is set by the maintainer
2. Quality scale requires second opinion (current solution)
3. Platinum tier is granted by quality control board
4. Audit quality scale of integrations

2. Quality scale requires second opinion

Changing the quality scale requires approval from one or more integration developers, other than the integration developer. Quality concerns in relation to the given quality scale can be raised via a bug report. It is up the developer to improve the integration, lower the quality scale or reject the concern.
Pro's:

• Democratic process
  Con's:
• Quality control could be inconsistent
• More administrative overhead

3. Platinum tier is granted by quality control board

A group of community members review integrations (based on nominations). The select few which are chosen to be promoted to platinum are highlighted during the next release party to celebrate the extraordinary achievement and explain to the community what makes it special.
Pro's:

• Best quality control
• Recognition
  Con's
• More administrative overhead

4. Audit quality scale of integrations

The quality scale is audited by a group of community members
Pro's:

• Good quality control
  Con's:
• Downgrading can be disappointing
• Most administrative work

Next Steps

1. Gather feedback on this proposal and update it where needed.
2. Determine if the problem warrants change.
3. Work out the details if the preferred direction requires this.

[mib1185]

Just to clarify - we're already in mode 2 of your options list - not to say, in general no one can merge any PR without a review and approval from a second approved developer 😉

[AJediIAm]

I'll remove option 1 as it provides no added value.

[joostlek]

Interesting.

One thing that I am wondering after reading this is, what is the actual problem we are tackling? You mention that

Currently, there are several integrations within the Platinum tier which may not consistently meet this "best of the best" standard.

And I am not aware of any integration that has this, and I think it's important to keep each other sharp when the integration is indeed lacking.

How the scale currently works is that we approve it once it reached the needed quality, and after that, PRs to that integration are marked with a label to make reviewers aware that they need to take into account the extra rules. A couple of rules have checks for in hassfest, so they are harder to ignore.

In the revamp of the integration quality scale we also described that if we changed a rule, we would remove the quality scale from every integration and they would have to reapply. This has not happened yet. (And I do think this is a rigorous way, so it also depends on the adoption rate of that feature, the ease of implementing that feature and where it places on the scale)

[frenck]

For the example I encountered, I created a bug report. It was rejected by the developer.

Please link it.

This discussion is not about any particular integration, but about the quality control process of platinum tier.

Without showing what is wrong, this will end up being a meta thing with no concrete handles.
IMHO, that is very counterproductive. We are open source, please just be open about it.

[frenck]

I've checked the HA repos, but wasn't able to find a related bug report from you in our organization.

https://github.com/issues?q=is:issue+user:home-assistant+author:AJediIAm+

[AJediIAm]

For the example I encountered, I created a bug report. It was rejected by the developer.

Please link it.

home-assistant/core#143781

[frenck]

Ok... Thanks for that one, and thanks for the suggestion.

I know it can be frustrating that you have a feature request that isn't implemented or expecting a different user experience from this specific integration or Home Assistant as a whole.

I'm going to be real honest, this isn't in scope for this scale or in this architectural repository. If you want to suggest a feature, you should try our Community Forum: Feature Requests.

If you have additional questions, feel free to join our Discord chat server.

Thanks! 👍

../Frenck

[AJediIAm]

As a final request: May I ask for all platinum integrations during a release party?
It's an extraordinary achievement to reach the platinum tier and it deserves a highlight.

[frenck]

As a final request: May I ask for all platinum integrations during a release party?

They have been part of that highlight the past few releases and release parties already.

../Frenck

[AJediIAm]

Yes, but not all if them (as far as I can recall)

[mib1185]

Which one areyou missing? Niote that only the in this release newly adopted quality scales are mentioned, not the overall list

[parautenbach]

Just a small but possibly contentious suggestion: I think the admin overhead is a real challenge – together with consistency – so would employing* an AI for this purpose not be quite helpful? With some excellent prompting it might do quite well to analyse an integration, and one doesn't need to solely rely on it's result, but rather take it as a recommendation.

[frenck]

I don't understand the issue raised. In all honesty, the concrete problem you are trying to address and fix in this proposal is missing, making the proposed solutions basically moot, IMHO, and they suggest changing the process without explaining the issue it solves.

As pointed out above, the current method is the second one you suggested.
The integration author (or one of the contributors) requests setting the IQS scale; the review of it is the secondary opinion.

[AJediIAm]

Quality is something which often requires preventive controls to maintain, so I thought it was worth having the discussion to see if a preventive control is desirable or of we can stick with the current reactive controls. I'm not a home assistant developer, so I'll process the feedback with the goal to have a constructive discussion and outcome.

As stated in the steps, it needs to be determined if there is an actual problem. I didn't know exactly how to approach this. The comments above is already addressing this so we're making progress.