Error messages

[sorinsarca]

I'm writing this because there is a demand for error messages (see #14 and #15) and I'm not sure
which way is the best. There are multiple ways of handling error messages and I'm pretty sure that they are different enough to require different implementations, and I need some feedback.

1)

One way is the json-guard's approach - a flat array of errors. But this approach is not so verbose, for example the anyOf keyword says that no schema is matched but it doesn't say why every schema failed - because of a minimum keyword? or maybe a type keyword?

Schema

{
  "anyOf": [
      {"type": "string"},
      {"type": "number", "minimum": 10}
  ]
}

Data

5

Errors reported by json-guard

[
   {
       "keyword": "anyOf",
       "message": "The data must match one of the schemas.",
      // ...
   }
]

2)

The second way will be similar to json-guard for simple keywords, but the errors for keywords like anyOf will also contain nested arrays of sub-errors.

Here is an example (consider unlimited errors allowed):

Schema

{
    "allOf": [
        {"type": "number"},
        {"$ref": "#/anyOf/1"},
        {"multipleOf": 2}
    ],
    "anyOf": [
        {"type": "string"},
        {"minimum": 11}
    ]
}

Errors when data is 9

[
    {
        "keyword": "allOf",
        "message": "The data must match all subschemas",
        "data": 9,
        "dataPath": [],
        // ...other optional properties (see below)
        "subErrors": [
            {
                "keyword": "minimum",
                "message": "The number must be greater than or equal to 11",
                "data": 9,
                "dataPath": [],
                // ... other optional properties, but no subErrors
            }
        ]
    },
    {
        "keyword": "anyOf",
        "message": "The data must match at least one subschema",
        "data": 9,
        "dataPath": [],
        // ...
        "subErrors": [
            {
                "keyword": "type",
                "message": "The data type must be string",
                "data": 9,
                "dataPath": [],
                // ...
            },
            {
                "keyword": "minimum",
                "message": "The number must be greater than or equal to 11",
                "data": 9,
                "dataPath": [],
                // ...
            }
        ]
    }
]

Other optional properties can be:

• schema - object/boolean
• schemaPath - path to the schema
• schemaId
• others

3)

In the previous approach (2) the problem is that you cannot assign to a schema an error and treat that schema like a simple keyword (some kind of atomicity). Here is what I mean:

{
    "properties": {
        "d": {"$ref": "#/definitions/digit"}
    },

    "definitions": {
        "digit": {
           "type": "integer",
           "minimum": 0,
           "maximum": 9
        }
    }
}

Data

{"d": 12}

Desired error:

{
    "data": 12,
    "dataPointer": ["d"],
    "message": "The value must be a digit",
    // ...
}

As you can see, I don't care how the "digit" schema validates the data, I'm interested only in the result, therefore I want a custom message.

We can add some metadata keywords to solve the problems of the previous approach.

{
    "properties": {
        "d": {"$ref": "#/definitions/digit"}
    },

    "definitions": {
        "digit": {

           "$error": "The value must be a digit",

           "type": "integer",
           "minimum": 0,
           "maximum": 9
        }
    }
}

Now, when the $ref fails to validate the data, it will check for$error keyword (in the referenced schema) and use the custom message if exists.

4)

There is always room for extending so this approach will tackle the problems for keywords like pattern.

Here is an example of what I mean:

Schema

{
   "maxLength": 20,
   "pattern": "^[A-Z].*"
}

Data

"abc"

Desired error:

{
    "data": "abc",
    "message": "The string must start with a capital letter",
    // ...
}

Current/Default error:

{
    "data": "abc",
    "message": "The string must match pattern ^[A-Z].*",
    // ...
}

Again, some metadata keyword can be used to replace the default message.

{
   "maxLength": 20,
   "pattern": "^[A-Z].*",

   "$errors": {
     "maxLength": "Too long! 20 chars should be enough for everyone",
     "pattern": "The string must start with a capital letter"
   }
}

---

These are the cases that have come to my mind at the moment, probably there are many more.

[ghola]

I think that regardless of option, the goal here should be to produce the most human readable error message and path. With that in mind:

1)

I agree that in the case of anyOf the error can be a ambiguous.

When it comes to usage in API this is partially mitigated by the fact that the schema becomes public, as it's part of the (ideally OpenAPI) spec.

This option is clearly the easiest to implement and could be a good first version.

2)

On complex schemas, this level of array nesting can become significant and hard to follow, but I think it's the most complete solution. There's no way to avoid nesting if you want to get the errors for each subschema.

3)

Initially I was put off by this option, because it requires the schema designer to follow a convention (which means existing schemas need to be refactored just so they can make use of this type of validation). But given the difficulty in expressing an "ideal" message at point 2 in the case of complex subschemas, using the reference as an alias to refer to the entirety of the subschema doesn't seem that bad.

The's still the issue of actually knowing what failed for each of the subschemas (in case of single field schemas that's easy, but when they are complex, knowing the path of the troublesome field and it's own error would be helpful).

4)

Without the metadata $error field, this would be too complex to implement and I'm not sure it's worth the effort. Gathering meaning from arbitrary regular expressions is no small feat (if at all possible).

5)

Another option would be to take option 2) and flatten it. There's a similar discussion about this option in this redux form issue. In the json schema case, the key of the error array would probably be the path. Personally I'm not really sold on this option, but I'm mentioning it for completeness.

[vearutop]

Please check json-schema-org/json-schema-spec#679 and json-schema-org/json-schema-spec#643 to meet standardized validation result format.

[PetrToman]

Please consider refactoring the code so that localization of error messages is possible.

[m1x0n]

Hi there. I made an attempt to write some sort of error-presenting library. Currently it has several presenting approaches and localization support.
Just in case you're interested in https://github.com/m1x0n/opis-error-presenter.

[sorinsarca]

In v2.0 the class Opis\JsonSchema\Errors\ErrorFormatter contains helpers to get and format the error messages.