Add output formatting tests

[gregsdennis]

In light of draft2019-09's output formatting, I think it might be prudent to have a set of tests that validate the output formatting.

---

Maybe this can be included in the draft2019-09 tests themselves (when they're added).

Something like:

Example test from draft-07's *properties.json*

{
  "description": "object properties validation",
  "schema": {
    "properties": {
      "foo": {"type": "integer"},
      "bar": {"type": "string"}
    }
  },
  "tests": [
    {
      ...
    },
    {
      "description": "one property invalid is invalid",
      "data": {"foo": 1, "bar": {}},
      "valid": false,
      "outputs": {
        "flag": {
          "valid": false
        },
        "basic":{
          "valid" : false,
          "errors" : [
            {
              "valid" : false,
              "keywordLocation" : "#/properties",
              "instanceLocation" : "#"
            },
            {
              "valid" : false,
              "keywordLocation" : "#/properties/foo/type",
              "instanceLocation" : "#/foo"
            },
            {
              "valid" : false,
              "keywordLocation" : "#/properties/bar/type",
              "instanceLocation" : "#/bar"
            }
          ]
        },
        "detailed": {
          "valid" : false,
          "keywordLocation" : "#/properties",
          "instanceLocation" : "#",
          "errors" : [
            {
              "valid" : false,
              "keywordLocation" : "#/properties/foo/type",
              "instanceLocation" : "#/foo"
            },
            {
              "valid" : false,
              "keywordLocation" : "#/properties/bar/type",
              "instanceLocation" : "#/bar"
            }
          ]
        },
        "verbose": {
          "valid" : false,
          "keywordLocation" : "#",
          "instanceLocation" : "#",
          "errors" : [
            {
              "valid" : false,
              "keywordLocation" : "#/properties",
              "instanceLocation" : "#",
              "errors" : [
                {
                  "valid" : false,
                  "keywordLocation" : "#/properties/foo",
                  "instanceLocation" : "#/foo",
                  "errors" : [
                    {
                      "valid" : false,
                      "keywordLocation" : "#/properties/foo/type",
                      "instanceLocation" : "#/foo"
                    }
                  ]
                },
                {
                  "valid" : false,
                  "keywordLocation" : "#/properties/bar",
                  "instanceLocation" : "#/bar",
                  "errors" : [
                    {
                      "valid" : false,
                      "keywordLocation" : "#/properties/bar/type",
                      "instanceLocation" : "#/bar"
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    {
      ...
    }
  ]
}

Doing this on all the tests, though, seems like overkill. I think a set of tests specifically for output would be more concise.

[vearutop]

I'd like to have sample outputs for *Of, if, then, else and not.

If we have a reference implementation we can easily create validation results for all test cases, which can be helpful before we have a dedicated output test suite.

I think it make sense to stick to basic test suite instead of dedicated output test suite, as it is already mature and has good overall coverage.

[gregsdennis]

Now that the current format has been out for a while, and we've received a lot of good feedback about it (which was the intent behind not making it a hard requirement), we're looking to update it for the next iteration of the spec. Discussion can be found here.

The question I'd like to look into here is whether this test suite needs to include tests for the 2019-09 / 2020-12 formats and the new formats (whatever that ends up being). Based on the current test structure, I expect so, but it's still worth asking.

[Julian]

I have to review that thread -- it's crossed my eye but I haven't tried to understand the change yet -- regardless I think agree with the point.

whether this test suite needs to include tests for the 2019-09 / 2020-12 formats and the new formats (whatever that ends up being).

I think this is what you're suggesting too, but yeah we should have the 2019-09 and 2020-12 tests include output tests in the form defined "back then", and future drafts in the newer format once it's settled on in that thread.