Add validation suite checker and schema for the new validation/ directory#912
Add validation suite checker and schema for the new validation/ directory#912codefromlani wants to merge 2 commits into
Conversation
jdesrosiers
left a comment
jdesrosiers
left a comment
There was a problem hiding this comment.
Here are a few things I found since our call earlier.
You need to update tox.ini to use check_validation_suite.py instead of the old script. Then the ci check will happen properly.
This doesn't take into account the v1 tests. The v1 tests cover future releases that we don't necessarily have a validator for. Currently there are several case where the checks will fail if we don't have a validator for them. I think if we don't have a validator needed for a test, then we should just skip that check.
jdesrosiers
left a comment
jdesrosiers
left a comment
There was a problem hiding this comment.
I left a few comments based on your last push. You still need a way to handle future releases.
| "additionalProperties": false | ||
| }, | ||
|
|
||
| "$defs": { |
There was a problem hiding this comment.
In draft-07, it should be definitions, not $defs.
| "description": { | ||
| "description": "The test description, briefly explaining which behavior it exercises", | ||
| "type": "string", | ||
| "maxLength": 69 |
There was a problem hiding this comment.
| "maxLength": 69 | |
| "maxLength": 70 |
| "description": { | ||
| "description": "The test case description", | ||
| "type": "string", | ||
| "maxLength": 149 |
There was a problem hiding this comment.
| "maxLength": 149 | |
| "maxLength": 150 |
| Validator.check_schema(schema) | ||
| except jsonschema.SchemaError as e: | ||
| self.fail( | ||
| f"Schema is invalid under draft {dialect} " |
There was a problem hiding this comment.
This message won't make sense for future releases because we won't be calling them "draft" anymore.
| f"Schema is invalid under draft {dialect} " | |
| f"Schema is invalid under release {dialect} " |
I have guard against missing validators with Or, is there another approach to handle it? |
2 participants
Following your suggestion to use a new
validation/directory to avoid conflict with the existingtests/, this adds:validation-test-schema.json— schema for files invalidation/,extending
test-schema.jsonwith thecompatibilityfieldcheck_validation_suite.py— sanity checker forvalidation/,similar to
bin/jsonschema_suite