Skip to content

Add test data versioning policy to README.md #735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 24, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,10 @@ needs to be made in the [exercism/x-common repository](https://github.com/exerci
not directly to the test suite itself.

Find the JSON file for the problem in question. For example, if you want to change
the Clock problem, then look for `exercises/clock/canonical-data.json`.
the Clock problem, then look for `exercises/clock/canonical-data.json`. Each
change should also bump the version of the test data. For more information,
see the [test data versioning](README.md#test-data-versioning) section of the
README.

Submit a pull request with the change.

Expand Down
52 changes: 52 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,58 @@ There are also some convention about `expected` that you must follow:
- The string should explain why the error would occur.
- A particular track's implementation of the exercise **need not** necessarily check that the error includes that exact string as the cause, depending on what is idiomatic in the language (it may not be idiomatic to check strings for error messages).

### Test Data Versioning

Test data should be versioned according to [Semantic Version 2.0](http://semver.org/), which is defined as follows:

> Given a version number MAJOR.MINOR.PATCH, increment the:
>
> MAJOR version when you make incompatible API changes,
> MINOR version when you add functionality in a backwards-compatible manner, and
> PATCH version when you make backwards-compatible bug fixes.
> Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

#### MAJOR version changes

The MAJOR version should be changed when the test suite is modified in a
fundamentally incompatible way.

There are examples of changes requiring a MAJOR version change:

- A new "property" (test type).
- Renaming a "property".
- Insertion, deletion or renaming of keys in the test data object.
- Changing the type of one of the test data keys.

MAJOR changes should be expected to break even well-behaved test generators.

#### MINOR version changes

The MINOR version should change when you add functionality in a backwards-compatible manner, make
non-breaking changes that alter the meaning of the test suite, make previously
passing solutions possibly fail, or failing solutions fail at a different spot.

There are examples of changes requiring a MINOR version change:

- Adding or deleting test cases.
- Changing the test cases inputs and/or outputs.
- Changing the test cases ordering.

MINOR changes would never break well-designed test generators, because the test-generation logic remains exactly the same.

#### PATCH version changes

The PATCH version should change when you make backwards-compatible bug fixes or
whenever the meaning of the tests does not change.

There are examples of changes requiring a PATCH version change:

- Regrouping/"Renesting" test cases without changing test case ordering.
- Changing descriptions or comments.
- Changing keys' ordering or formatting (would result in an equivalent JSON file).

PATCH changes would never break well-designed test generators, because the test data remains exactly the same.

## Automated Tests

`canonical-data.json` for each exercise is checked for compliancy against the [canonical-schema.json](canonical-schema.json).
Expand Down