On validation error: add cause and context. by JulienPalard · Pull Request #129 · python-openapi/openapi-spec-validator

JulienPalard · 2021-09-02T10:06:34Z

I often miss the context provided by jsonschema library, so I'm trying to add it.

Given the following erroneous schema (it misses a description):

---
openapi: 3.0.0

info:
  title: test
  description: test
  version: 0.0.1

paths:
  "/":
    get:
      description: Get the API root
      responses:
        200:
          content:
            application/json:
              schema:
                type: string

without this PR, openapi-spec-validator displays:

{'content': {'application/json': {'schema': {'type': 'string'}}}} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$'
]['properties']['responses']['patternProperties']['^[1-5](?:\\d{2}|XX)$']:
    {'oneOf': [{'$ref': '#/definitions/Response'},
               {'$ref': '#/definitions/Reference'}]}

On instance['paths']['/']['get']['responses']['200']:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}

Basically telling « The path./.get.responses.200 is invalid: it should be a Response or a Reference. » without telling me about the missing description.

With this PR it gives:

{'content': {'application/json': {'schema': {'type': 'string'}}}} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$']['properties']['responses']['patternProperties']['^[1-5](?:\\d{2}|XX)$']:
    {'oneOf': [{'$ref': '#/definitions/Response'},
               {'$ref': '#/definitions/Reference'}]}

On instance['paths']['/']['get']['responses']['200']:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


Due to those subschema errors:

'description' is a required property

Failed validating 'required' in schema[0]:
    {'additionalProperties': False,
     'patternProperties': {'^x-': {}},
     'properties': {'content': {'additionalProperties': {'$ref': '#/definitions/MediaType'},
                                'type': 'object'},
                    'description': {'type': 'string'},
                    'headers': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Header'},
                                                                   {'$ref': '#/definitions/Reference'}]},
                                'type': 'object'},
                    'links': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Link'},
                                                                 {'$ref': '#/definitions/Reference'}]},
                              'type': 'object'}},
     'required': ['description'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


'$ref' is a required property

Failed validating 'required' in schema[1]:
    {'patternProperties': {'^\\$ref$': {'format': 'uri-reference',
                                        'type': 'string'}},
     'required': ['$ref'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}

This is probably a bit verbose, but I like a lot seeing 'description' is a required property.

JulienPalard · 2021-09-02T10:15:13Z

Oh I just remembered jsonschema validator had a best_match function!

I'm using it now so it's less floody / more readable:

{'content': {'application/json': {'schema': {'type': 'string'}}}} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$']['properties']['responses']['patternProperties']['^[1-5](?:\\d{2}|XX)$']:
    {'oneOf': [{'$ref': '#/definitions/Response'},
               {'$ref': '#/definitions/Reference'}]}

On instance['paths']['/']['get']['responses']['200']:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


Due to this subschema error:

'description' is a required property

Failed validating 'required' in schema[0]:
    {'additionalProperties': False,
     'patternProperties': {'^x-': {}},
     'properties': {'content': {'additionalProperties': {'$ref': '#/definitions/MediaType'},
                                'type': 'object'},
                    'description': {'type': 'string'},
                    'headers': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Header'},
                                                                   {'$ref': '#/definitions/Reference'}]},
                                'type': 'object'},
                    'links': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Link'},
                                                                 {'$ref': '#/definitions/Reference'}]},
                              'type': 'object'}},
     'required': ['description'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}

JulienPalard · 2021-09-02T13:23:36Z

I'm not happy with the --all flag which could be misinterpreted as giving all errors, not just the first error with all causes.

Any idea?

paulclarkaranz · 2021-12-21T06:04:19Z

This made error messages slightly more useful when trying to deal with upgrading past 0.3.0 and running into some minor adaptive changes required.

p1c2u · 2022-01-11T06:28:18Z

@JulienPalard

I'm not happy with the --all flag which could be misinterpreted as giving all errors, not just the first error with all causes.

Any idea?

Maybe --errors=all with default value best-match ?

JulienPalard · 2022-01-11T07:54:28Z

Maybe --errors=all with default value best-match ?

Updated my PR with this, it would work for me.

p1c2u · 2022-01-11T10:32:29Z

@JulienPalard thanks looks good, can you check failing test?

JulienPalard · 2022-01-11T12:44:21Z

I'm having a look at the failing test.

JulienPalard · 2022-01-11T13:35:55Z

I added a test and tried to make the message a bit readable (having all the subschema dumped in a serie was a bit hard to digest). It currently looks like:

$ python openapi_spec_validator tests/integration/data/v3.0/missing-description.yaml 
# Validation Error

{'content': {'application/json': {'schema': {'type': 'string'}}}} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$']['properties']['responses']['patternProperties']['^[1-5](?:\\d{2}|XX)$']:
    {'oneOf': [{'$ref': '#/definitions/Response'},
               {'$ref': '#/definitions/Reference'}]}

On instance['paths']['/']['get']['responses']['200']:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


# Probably due to this subschema error

## 'description' is a required property

Failed validating 'required' in schema[0]:
    {'additionalProperties': False,
     'patternProperties': {'^x-': {}},
     'properties': {'content': {'additionalProperties': {'$ref': '#/definitions/MediaType'},
                                'type': 'object'},
                    'description': {'type': 'string'},
                    'headers': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Header'},
                                                                   {'$ref': '#/definitions/Reference'}]},
                                'type': 'object'},
                    'links': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Link'},
                                                                 {'$ref': '#/definitions/Reference'}]},
                              'type': 'object'}},
     'required': ['description'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}

(1 more subschemas errors, use --errors=all to see them.)

$ python openapi_spec_validator tests/integration/data/v3.0/missing-description.yaml --errors=all
# Validation Error

{'content': {'application/json': {'schema': {'type': 'string'}}}} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$']['properties']['responses']['patternProperties']['^[1-5](?:\\d{2}|XX)$']:
    {'oneOf': [{'$ref': '#/definitions/Response'},
               {'$ref': '#/definitions/Reference'}]}

On instance['paths']['/']['get']['responses']['200']:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


# Due to one of those errors

## 'description' is a required property

Failed validating 'required' in schema[0]:
    {'additionalProperties': False,
     'patternProperties': {'^x-': {}},
     'properties': {'content': {'additionalProperties': {'$ref': '#/definitions/MediaType'},
                                'type': 'object'},
                    'description': {'type': 'string'},
                    'headers': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Header'},
                                                                   {'$ref': '#/definitions/Reference'}]},
                                'type': 'object'},
                    'links': {'additionalProperties': {'oneOf': [{'$ref': '#/definitions/Link'},
                                                                 {'$ref': '#/definitions/Reference'}]},
                              'type': 'object'}},
     'required': ['description'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}


## '$ref' is a required property

Failed validating 'required' in schema[1]:
    {'patternProperties': {'^\\$ref$': {'format': 'uri-reference',
                                        'type': 'string'}},
     'required': ['$ref'],
     'type': 'object'}

On instance:
    {'content': {'application/json': {'schema': {'type': 'string'}}}}

I find the "markdown-y" ## a good help to visually iterate the list of errors.

codecov · 2022-01-11T14:45:18Z

Codecov Report

Merging #129 (2dcf445) into master (afb6024) will decrease coverage by 0.32%.
The diff coverage is 89.47%.

@@            Coverage Diff             @@
##           master     #129      +/-   ##
==========================================
- Coverage   98.46%   98.14%   -0.33%     
==========================================
  Files          19       19              
  Lines         520      538      +18     
==========================================
+ Hits          512      528      +16     
- Misses          8       10       +2

Impacted Files	Coverage Δ
openapi_spec_validator/__main__.py	`94.23% <89.47%> (-2.83%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afb6024...2dcf445. Read the comment docs.

p1c2u

LGTM

p1c2u · 2022-01-11T15:01:59Z

@JulienPalard thank you for the contribution

This was referenced Sep 2, 2021

Failures in oneOf, anyOf, etc, are actually in the referenced schema #126

Open

Failed validating 'oneOf' in schema #113

Closed

Add a flag to display either 'best-match' or 'all' errors.

2dcf445

p1c2u approved these changes Jan 11, 2022

View reviewed changes

p1c2u merged commit e23cb90 into python-openapi:master Jan 11, 2022

JulienPalard deleted the show-context branch January 11, 2022 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

On validation error: add cause and context.#129

On validation error: add cause and context.#129
p1c2u merged 1 commit intopython-openapi:masterfrom
JulienPalard:show-context

JulienPalard commented Sep 2, 2021

Uh oh!

JulienPalard commented Sep 2, 2021

Uh oh!

JulienPalard commented Sep 2, 2021

Uh oh!

paulclarkaranz commented Dec 21, 2021

Uh oh!

p1c2u commented Jan 11, 2022 •

edited

Loading

Uh oh!

JulienPalard commented Jan 11, 2022

Uh oh!

p1c2u commented Jan 11, 2022

Uh oh!

JulienPalard commented Jan 11, 2022

Uh oh!

JulienPalard commented Jan 11, 2022 •

edited

Loading

Uh oh!

codecov bot commented Jan 11, 2022 •

edited

Loading

Uh oh!

p1c2u left a comment

Uh oh!

p1c2u commented Jan 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

JulienPalard commented Sep 2, 2021

Uh oh!

JulienPalard commented Sep 2, 2021

Uh oh!

JulienPalard commented Sep 2, 2021

Uh oh!

paulclarkaranz commented Dec 21, 2021

Uh oh!

p1c2u commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JulienPalard commented Jan 11, 2022

Uh oh!

p1c2u commented Jan 11, 2022

Uh oh!

JulienPalard commented Jan 11, 2022

Uh oh!

JulienPalard commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

p1c2u left a comment

Choose a reason for hiding this comment

Uh oh!

p1c2u commented Jan 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

p1c2u commented Jan 11, 2022 •

edited

Loading

JulienPalard commented Jan 11, 2022 •

edited

Loading

codecov bot commented Jan 11, 2022 •

edited

Loading