Skip to content

Widened type interface #374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Dec 24, 2017
17 changes: 13 additions & 4 deletions docs/creating.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,16 @@ Creating or Extending Validator Classes
will have :func:`validates` automatically called for the given
version.

:argument dict default_types: a default mapping to use for instances
of the validator class when mapping between JSON types to Python
types. The default for this argument is probably fine. Instances
can still have their types customized on a per-instance basis.
:argument dict default_types:
.. deprecated:: 2.7.0 Please use the type_checker argument instead.

If set, it provides mappings of JSON types to Python types that will
be converted to functions and redefined in this object's
:class:`jsonschema.TypeChecker`.

:argument jsonschema.TypeChecker type_checker: a type checker. If
unprovided, a :class:`jsonschema.TypeChecker` will created with no
supported types.

:returns: a new :class:`jsonschema.IValidator` class

Expand All @@ -59,6 +65,9 @@ Creating or Extending Validator Classes

:argument str version: a version for the new validator class

:argument jsonschema.TypeChecker type_checker: a type checker. If
unprovided, the existing :class:`jsonschema.TypeChecker` will be used.

:returns: a new :class:`jsonschema.IValidator` class

.. note:: Meta Schemas
Expand Down
71 changes: 51 additions & 20 deletions docs/validate.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,14 @@ classes should adhere to.
will validate with. It is assumed to be valid, and providing
an invalid schema can lead to undefined behavior. See
:meth:`IValidator.check_schema` to validate a schema first.
:argument types: Override or extend the list of known types when
:argument types:
.. deprecated:: 2.7.0
Instead, create a custom type checker and extend the validator. See
:ref:`validating-types` for details.

If used, this overrides or extends the list of known type when
validating the :validator:`type` property. Should map strings (type
names) to class objects that will be checked via :func:`isinstance`.
See :ref:`validating-types` for details.
:type types: dict or iterable of 2-tuples
:argument resolver: an instance of :class:`RefResolver` that will be
used to resolve :validator:`$ref` properties (JSON references). If
Expand All @@ -48,8 +52,13 @@ classes should adhere to.

.. attribute:: DEFAULT_TYPES

The default mapping of JSON types to Python types used when validating
:validator:`type` properties in JSON schemas.
.. deprecated:: 2.7.0
Use of this attribute is deprecated in favour of the the new type
checkers.

It provides mappings of JSON types to Python types that will
be converted to functions and redefined in this object's type checker
if one is not provided.

.. attribute:: META_SCHEMA

Expand All @@ -62,6 +71,10 @@ classes should adhere to.
that validate the validator property with that name. For more
information see :ref:`creating-validators`.

.. attribute:: TYPE_CHECKER
A :class:`TypeChecker` that can be used validating :validator:`type`
properties in JSON schemas.

.. attribute:: schema

The schema that was passed in when initializing the object.
Expand Down Expand Up @@ -127,17 +140,35 @@ implementors of validator classes that extend or complement the
ones included should adhere to it as well. For more information see
:ref:`creating-validators`.

Type Checking
-------------

To handle JSON Schema's :validator:`type` property, a :class:`IValidator` uses
an associated :class:`TypeChecker`. The type checker provides an immutable
mapping between names of types and functions that can test if an instance is
of that type. The defaults are suitable for most users - each of the
predefined Validators (Draft3, Draft4) has a :class:`TypeChecker` that can
correctly handle that draft.

See :ref:`validating-types` for an example of providing a custom type check.

.. autoclass:: TypeChecker
:members:

.. autoexception:: jsonschema.exceptions.UndefinedTypeCheck

Raised when trying to remove a type check that is not known to this
TypeChecker. Internally this is also raised when calling
:meth:`TypeChecker.is_type`, but is caught and re-raised as a
:class:`jsonschema.exceptions.UnknownType` exception.

.. _validating-types:

Validating With Additional Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Occasionally it can be useful to provide additional or alternate types when
validating the JSON Schema's :validator:`type` property. Validators allow this
by taking a ``types`` argument on construction that specifies additional types,
or which can be used to specify a different set of Python types to map to a
given JSON type.
validating the JSON Schema's :validator:`type` property.

:mod:`jsonschema` tries to strike a balance between performance in the common
case and generality. For instance, JSON Schema defines a ``number`` type, which
Expand All @@ -152,24 +183,24 @@ more general instance checks can introduce significant slowdown, especially
given how common validating these types are.

If you *do* want the generality, or just want to add a few specific additional
types as being acceptable for a validator object, :class:`IValidator`\s have a
``types`` argument that can be used to provide additional or new types.
types as being acceptable for a validator object, then you should update an
existing :class:`TypeChecker` or create a new one. You may then create a new
:class:`IValidator` via :meth:`extend`.

.. code-block:: python

class MyInteger(object):
...
pass

def is_my_int(checker, instance):
return (Draft3Validator.TYPE_CHECKER.is_type(instance, "number") or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ech, so I didn't know whether this was intentional or not, but now I see that there's some infinite recursion nonsense here when you upcall...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there? In order to use the is_my_int, a user must generate a new TypeChecker instance (via, e.g. redefine), which is then given to a new validator (via create or extend). In other words:

Draft3Validator.TYPE_CHECKER.is_type(instance, "number")

Is not changed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This was a half-baked comment that I just wrote to make sure I didn't forget to address it, sorry for the missing context -- it's the following:)

Not in the way you wrote it, but in the way I thought you mean to write it, which was:

return checker.is_type(instance, "number") or isinstance(instance, MyInteger)

That will do nasty infinite things because it calls itself, but I think the expectation there is clearly to upcall to the original definition of number.

And the above case was basically my motivation for suggesting we include the checker as one of the two arguments to the function (because it allows for forward extension).

The infinite recursion case though makes that a bit more complicated, it means there's a different thing you do if you're overriding the same type and want to upcall as if you're trying to access another type checker.

I think this needs a bit more thought -- my first reaction when I realized the above was "maybe we should pass in a third argument too, the original function from the checker that was redefined", but I think we just need to think through the 2 use cases again and make sure this API makes sense (the use case where you're accessing another checker, and the one where you're trying to upcall to the original one).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the above case was basically my motivation for suggesting we include the checker as one of the two arguments to the function (because it allows for forward extension)

Apologies as I think we could have had this conversation earlier. I was concerned about attempted infinite recursion too but liked the ability to reference another type via checker.

The third argument doesn't immediately feel right to me - mostly because of the awkward basecase of a null function (in fact its not a null function, it's one that would raise an exception for an unknown type).

Will have a think about it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for going AWOL on this - turns out changing jobs and moving takes up a fair bit of time.

As far as I remember, this point was the main remaining issue. I think I am in favour of the current API as a balance of simplicity and power. To me, the biggest benefit for passing in the type checker object (i.e. the current API) is for the ease of custom types, e.g. "string_or_int". In addition, it feels OK to be pretty explicit about which type checking function you are upcalling to in the case of a type redefinition.

Passing in both the object and function that is being overwritten feels like we are providing a somewhat bizarre OO inheritance mechanism, which I dislike.

What do you think? Perhaps we simply add a warning to the documentation on the potential for infinite recursion? If you're happy with that I'll sort a PR for this and the other minor comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely no need to apologise, trust me staying at the same job and not moving is also pretty killer :(

I think the warning might be the thing, but I wanna reread my own comment again and see if a months time has given me any subconscious enlightenment (I doubt it...)

isinstance(instance, MyInteger))

type_checker = Draft3Validator.TYPE_CHECKER.redefine("number", is_my_int)

Draft3Validator(
schema={"type" : "number"},
types={"number" : (numbers.Number, MyInteger)},
)
CustomValidator = extend(Draft3Validator, type_checker=type_checker)
validator = CustomValidator(schema={"type" : "number"})

The list of default Python types for each JSON type is available on each
validator object in the :attr:`IValidator.DEFAULT_TYPES` attribute. Note
that you need to specify all types to match if you override one of the
existing JSON types, so you may want to access the set of default types
when specifying your additional type.

.. _versioned-validators:

Expand Down
5 changes: 5 additions & 0 deletions jsonschema/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@
from jsonschema._format import (
FormatChecker, draft3_format_checker, draft4_format_checker,
)
from jsonschema._types import (
TypeChecker,
draft3_type_checker,
draft4_type_checker,
)
from jsonschema.validators import (
Draft3Validator, Draft4Validator, RefResolver, validate
)
Expand Down
205 changes: 205 additions & 0 deletions jsonschema/_types.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
import numbers

import attr
import pyrsistent

from jsonschema.compat import str_types, int_types, iteritems
from jsonschema.exceptions import UndefinedTypeCheck


def is_array(checker, instance):
return isinstance(instance, list)


def is_bool(checker, instance):
return isinstance(instance, bool)


def is_integer(checker, instance):
# bool inherits from int, so ensure bools aren't reported as ints
if isinstance(instance, bool):
return False
return isinstance(instance, int_types)


def is_null(checker, instance):
return instance is None


def is_number(checker, instance):
# bool inherits from int, so ensure bools aren't reported as ints
if isinstance(instance, bool):
return False
return isinstance(instance, numbers.Number)


def is_object(checker, instance):
return isinstance(instance, dict)


def is_string(checker, instance):
return isinstance(instance, str_types)


def is_any(checker, instance):
return True


@attr.s(frozen=True)
class TypeChecker(object):
"""
A ``type`` property checker.

A :class:`TypeChecker` performs type checking for an instance of
:class:`Validator`. Type checks to perform are updated using
:meth:`TypeChecker.redefine` or :meth:`TypeChecker.redefine_many` and
removed via :meth:`TypeChecker.remove` or
:meth:`TypeChecker.remove_many`. Each of these return a new
:class:`TypeChecker` object.

Arguments:

type_checkers (dict):

The initial mapping of types to their checking functions.
"""
_type_checkers = attr.ib(default={}, convert=pyrsistent.pmap)

def is_type(self, instance, type):
"""
Check if the instance is of the appropriate type.

Arguments:

instance (object):

The instance to check

type (str):

The name of the type that is expected.

Returns:

bool: Whether it conformed.


Raises:

:exc:`jsonschema.exceptions.UndefinedTypeCheck`:
if type is unknown to this object.
"""
try:
fn = self._type_checkers[type]
except KeyError:
raise UndefinedTypeCheck(type)

return fn(self, instance)

def redefine(self, type, fn):
"""
Redefine the checker for type to the function fn.

Arguments:

type (str):

The name of the type to check.

fn (callable):

A function taking exactly two parameters - the type checker
calling the function and the instance to check. The function
should return true if instance is of this type and false
otherwise.

Returns:

A new :class:`TypeChecker` instance.

"""
return self.redefine_many({type:fn})

def redefine_many(self, definitions=()):
"""
Redefine multiple type checkers.

Arguments:

definitions (dict):

A dictionary mapping types to their checking functions.

Returns:

A new :class:`TypeChecker` instance.

"""
definitions = dict(definitions)
type_checkers = self._type_checkers.update(definitions)
return attr.evolve(self, type_checkers=type_checkers)

def remove(self, type):
"""
Remove the type from the checkers that this object understands.

Arguments:

type (str):

The name of the type to remove.

Returns:

A new :class:`TypeChecker` instance

Raises:

:exc:`jsonschema.exceptions.UndefinedTypeCheck`:
if type is unknown to this object

"""
return self.remove_many((type,))

def remove_many(self, types):
"""
Remove multiple types from the checkers that this object understands.

Arguments:

types (iterable):

An iterable of types to remove.

Returns:

A new :class:`TypeChecker` instance

Raises:

:exc:`jsonschema.exceptions.UndefinedTypeCheck`:
if any of the types are unknown to this object
"""
evolver = self._type_checkers.evolver()

for type_ in types:
try:
del evolver[type_]
except KeyError:
raise UndefinedTypeCheck(type_)

return attr.evolve(self, type_checkers=evolver.persistent())


draft3_type_checker = TypeChecker({
u"any": is_any,
u"array": is_array,
u"boolean": is_bool,
u"integer": is_integer,
u"object": is_object,
u"null": is_null,
u"number": is_number,
u"string": is_string
})

draft4_type_checker = draft3_type_checker.remove(u"any")
14 changes: 14 additions & 0 deletions jsonschema/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,20 @@ class RefResolutionError(Exception):
pass


class UndefinedTypeCheck(Exception):
def __init__(self, type):
self.type = type

def __unicode__(self):
return "Type %r is unknown to this type checker" % self.type

if PY3:
__str__ = __unicode__
else:
def __str__(self):
return unicode(self).encode("utf-8")


class UnknownType(Exception):
def __init__(self, type, instance, schema):
self.type = type
Expand Down
Loading