Managing tests through the feature life cycle

[jdesrosiers]

Sorry, I missed the discussion on #803 about moving the propertyDependencies tests to a new directory. I don't disagree with that move, but I realize that it isn't clear when these tests should be moved into the main suite. I think it should be when the feature reaches the "experimentation" phase because that's when we expect implementations to start implementing the keyword, but I imagine some of you are thinking it would be when the features moves to the "stable" stage. I want people's builds to break when a feature gets to the point where they should be implementing it. It's one way we can nudge implementers about new features. I just wanted to make sure we're on the same page on this.

This may be a bit of a tangent, but as an alternative, I was thinking about introducing a flag on tests that says what release or stage a test requires. Then people can easily filter out tests they want to ignore and we don't have to worry about separate directories for proposal tests.

Pinging @gregsdennis and @karenetheridge specifically for your thoughts.

[gregsdennis]

I'm not sure about that. I think while a feature is experimental, it should stay separate. It should only be merged into the main suite when incorporated into the spec and the spec is published.

I guess it depends on how we view the suite. Is it intended to be a representation of the spec, or is it just a tool for implementers? If the former, then features that haven't been merged and published definitely need to be in their own folders. If the latter, then I can see incorporation into the main suite being a message to implementers that this feature is coming.

---

A stage property could do if it's on every test at the schema level (or at least the proposal tests). My reasoning for putting it in a different folder is that there were tests spread throughout different files. Also, if I was the one upgrading a feature from experimental to stable, I'd find the folder approach makes it easier to not miss any.

[jdesrosiers]

I guess it depends on how we view the suite. Is it intended to be a representation of the spec, or is it just a tool for implementers?

I was definitely not thinking of it as a representation of the spec. I think it's more useful as a tool for developers.

A stage property could do if it's on every test at the schema level (or at least the proposal tests).

Actually, I was thinking of using the property for more than just stages. Once it gets added, it would record the release number where it was added. For example, it would go from "proposal" to "experimental" to "2027". That way something like bowtie can filter easily and say what level of support each implementation has by it's test results.

That's another think I'm just thinking about now! Bowtie might need to work a little differently for v1.

My reasoning for putting it in a different folder is that there were tests spread throughout different files. Also, if I was the one upgrading a feature from experimental to stable, I'd find the folder approach makes it easier to not miss any.

That's true, but it would be more maintenance. I'm not excited about having to move files when things are promoted. I honestly think filtering on a release property should be easy enough to use.

[karenetheridge]

My $0.02, which I think matches fairly closely with what Greg was saying:

• add some tests to proposals/ when there is enough detail fleshed out that it is possible to implement (even if some edge cases aren't figured out yet, the act of implementing can help resolve those issues)
• move from proposals/ to the main directory when the feature has been accepted and we do not anticipate any material changes are still to come

except for:

It should only be merged into the main suite when incorporated into the spec and the spec is published

This doesn't make sense because the v1 spec isn't "published" yet -- it's still being written. So by this definition, there would be nothing in the main directory for v1, and all of its tests would be in a side directory.

[jdesrosiers]

move from proposals/ to the main directory when the feature has been accepted and we do not anticipate any material changes are still to come

It's unclear what you mean by "accepted". What you describe sounds like the "experimentation" stage. At that point, no changes are anticipated and implementations need to start implementing them. It only gets added to the spec once enough implementations support it. But, if by "accepted", you mean it's added to the spec, that's different. So, can you clarify what you mean by "accepted"?

[karenetheridge]

Yes, by accepted I mean that there is agreement that it is to be added to the next version of the spec.

It only gets added to the spec once enough implementations support it.

This seems backwards to my understanding. I'm not likely to implement something if it's not in the spec yet (even the tentative version of the next one). I may implement it locally as an experiment to see if it's viable, but I wouldn't publish it for outward consumption, because at that point I'm committing to supporting it forever, and what if it is still subject to change?

[gregsdennis]

@karenetheridge there are two phases to proposal acceptance where you might implement a feature. The first is what you're talking about where just a few of us check viability.

The second is a wider implementation by some (as of yet undefined) subset of the tooling that we thought could be required before formally adding it to the spec. The idea is to get usage of it out there before adding it so that we could get the feedback we need to really refine it. This is to help avoid breaking changes. This:

It only gets added to the spec once enough implementations support it.

I think carries an implied "as an experimental feature" at the end. But it still carries the downsides that we don't have a definition for "enough" and we don't have a way to measure it (maybe Bowtie).

---

That said, I'm with Karen on the point that as an outsider implementer, I probably wouldn't consider supporting a feature if it wasn't part of the test suite.

For example, it would go from "proposal" to "experimental" to "2027". - @jdesrosiers

I'm fine with this idea, but we need to be clear about what each stage means for implementations. I'd say:

• Proposal means that the feature is still being defined - it could break without notice.
• Experimental means that the feature may need minor language tweaks - likely to be in the next release as is
• A year that matches a released spec version means that it is defined and required as is
• A year that is an unreleased spec version means that it is planned for that release as is

I think this would be a good signal to implementors of what features they need to support for the next release.

[jdesrosiers]

This seems backwards to my understanding. I'm not likely to implement something if it's not in the spec yet (even the tentative version of the next one). I may implement it locally as an experiment to see if it's viable, but I wouldn't publish it for outward consumption, because at that point I'm committing to supporting it forever, and what if it is still subject to change?

We're trying to change that mindset for v1. The goal is to emulate the way the ECMAScript TC-39 process works. The spec is supposed to reflect the reality of what's available and can be used, not what we hope vendors will implement.

TC-39 requires that all vendors have the feature available by default before it can go in the spec. When a new version of ECMAScript is released, it means that anything in that spec is already supported by all the major vendors and you can use those features today.

That's part of the vetting process for a new feature. If vendors don't like it and refuse to implement it or don't think it's stable enough to not be behind a feature flag, it doesn't go in the spec. That way the implementations decide as a community what goes in the spec, rather than just us pushing things on everyone.

So, if you refuse to implement a new keyword, you're voting that it shouldn't be included in the spec in any form. If you don't want to include it by default, you're voting that you think there needs to be changes (or at least that you aren't sure there doesn't need to be changes). So, if you're concerned that something is likely to change, by all means, don't make it available and tell us why. That's part of the process. Only, when we're all confident enough to make it available in our implementations does it get added to the spec.

[jdesrosiers]

For example, it would go from "proposal" to "experimental" to "2027".

I think this would be a good signal to implementors of what features they need to support for the next release.

Great! I'll put this on my TODO list.

[imApoorva36]

Hi @jdesrosiers
I noticed this is on your TODO list. If you haven’t started yet, I’d be happy to help by putting together a small proof of concept for the stage metadata

I can draft a PR demonstrating the field on a few sample files (e.g. showing a proposal transitioning to experimental). Before I do so, do you have a preferred “shape” for this, for example, as a property at the test case level (alongside description and schema), or nested within individual tests?

Let me know if there are any other constraints I should keep in mind. Happy to take a first pass to help move this along

[gregsdennis]

@imApoorva36 this repo isn't available for general/newcomer contribution. Please read the contributing guidelines in the Community repo for where you can help. Thanks.

[karenetheridge]

TC-39 requires that all vendors have the feature available by default before it can go in the spec. When a new version of ECMAScript is released, it means that anything in that spec is already supported by all the major vendors and you can use those features today.

I don't see how that could work for us. Who are "all vendors", or even "the major vendors"? And even if we had a defined list, how do we persuade them to implement a feature that isn't specified yet?

[jdesrosiers]

@imApoorva36 this repo isn't available for general/newcomer contribution.

Actually, while contributing tests generally requires some expert knowledge, I think helping with infrastructure is not a bad place for newcomers to contribute. But, let's make sure we agree on what we want first. This change would move the propertyDependencies tests back into the main suite, but add a "compatibility": "proposal:propertyDependencies" to the tests. When it's promoted to "experimental", it would change to "compatibility": "experimental:propertyDependencies". When it's promoted to "stable" it would change to "compatibility": 2027 or whatever year it was moved into the stable spec.

That should give implementations all the information they need filter the tests to what they support. Any objections?

[jdesrosiers]

I don't see how that could work for us. Who are "all vendors", or even "the major vendors"? And even if we had a defined list, how do we persuade them to implement a feature that isn't specified yet? - @karenetheridge

We've discussed some strategies for determining a list of the active major vendors, but we don't have a concrete plan yet. It's not a priority at the moment and I'm not worried about it. With bowtie, we should have no problem determining which implementations support which features. The rest should just be a matter of asking the implementers if they intend to keep up their implementations and educate them about what that means and what their role is in how the spec evolves. It will take some community outreach, but I believe it will work.

I realize this new way of doing things means people have to start thinking a little differently and some people won't want to, but I know this is a better path for JSON Schema and I'm confident people will come around and we'll have a lot more implementers involved and contributing to what happens with the spec.