Nested references not supported

[lmichel]

The following schema containing a self reference:
{
"type": "object",
"properties": {
"name": { "type": "string" },
"tree": { "$ref": "#/definitions/tree" }
},
"definitions": {
"tree": {
"type": "object",
"properties": {
"value": { "type": "string" },
"branches": {
"type": "array",
"items": { "$ref": "#/definitions/tree" },
"minItems": 1
}
},
"required": ["value"]
}
}
}

makes a stack overflow from com.networknt.schema.JsonSchema.read().
Not clear to me whether nested elements are supported by the standard, but some validators such as http://jeremydorn.com/json-editor/ support it.

regards
LM

[stevehu]

@lmichel I have added a test case to load your schema and I don't see any issue with it. Please take a look at the following link. Thanks.

https://github.com/networknt/json-schema-validator/blob/master/src/test/java/com/networknt/schema/SelfRefTest.java

[lmichel]

Dear Steve,

Thanks for your reply. Your test class reads my schema as a Node, not as a schema.
The following code does generate a stackoverflow.

Thanks

public class SelfRefTest extends BaseJsonSchemaValidatorTest {
    @Test
    public void testSelfRef() throws Exception {
        JsonSchema node = getJsonSchemaFromClasspath("selfref.json");
        System.out.println("node = " + node);
    }
}       

[stevehu]

Yes. I saw what you mean. When loading schema, it tries to resolve the $ref and goes to dead loop. I don't have an idea to resolve this at the moment. I am wondering if you know how the validator you mentioned resolves this issue.

there are two different approaches to build json schema validators: codegen vs dynamic interpreting. I think the later won't have the problem but overall performance will be slower.

Thanks

[lmichel]

Steve,

To be able to continue playing with your tool, I patched the JsonSchema class in order to limit the recursively depth.
Following attributes added:

    protected static Map<String, Integer> RECURSIVE_DEPTH = new LinkedHashMap<String, Integer>();
    protected static final int MAX_DEPTH = 10;

patch in method read

               String key = getSchemaPath() + "/" + pname;
                Integer d;
                if( (d = RECURSIVE_DEPTH.get(key)) == null || d < MAX_DEPTH) {
                	Integer nd = (d == null)? 1 : (d+1);
                	RECURSIVE_DEPTH.put(key, nd);
                	c = clazz.getConstructor(new Class[]{String.class,
                        JsonNode.class, JsonSchema.class, ObjectMapper.class});
                	validators.put(key, c.newInstance(key, n, this, mapper));
                } 

I'm not sure this fix is consistent with you internal data model, but the schema is created.
I believe that others tools do the same way (http://jeremydorn.com/json-editor/ e.g.)
Laurent

[stevehu]

I have tried your code and there are several test cases failed. I haven't checked each individual test cases yet as I am working on another project almost 14 hours per day including weekends. Once I have time, I will comeback to this issue. Meanwhile, could you please explain your use case so that I can understand the scope of the issue? Thanks.

[lmichel]

Hello, Let me introduce a little bit about what I'm doing. I'm involved in the IVOA (http://ivoa.net) which is an international alliance developing and promoting interoperability standards for the Astronomy. When the project started in 2004, everyone was talking XML/XSD. In this frame, we developed an XML schema suited to the transport of tabular data in XML files (namely VOTable). Using schemas is essential for interoperability. The trend now is to use JSON everywhere and we are considering at add a schema support to JSON data files. Those JSON schemas should be as much compliant as possible with XML schemas and none could do hand-made translations from scratch. Basically we need 2 sorts of tools: 1) An helper for converting XSD to JSon schemas 2) A validator to validate data files against a schema. I'm using the online one I mentioned in my last mail: http://www.jsonschemavalidator.net/. It is written in JS and can be run with NodeJS. I'm also having a look for solution based on either Java or Python. That is why I ran some test with your tool. By the way, I understand you have higher priority tasks and fixing this would a big job. So I propose to postpone the issue. Regards LM Le 07/01/2017 à 18:24, Steve Hu a écrit :

…

I have tried your code and there are several test cases failed. I haven't checked each individual test cases yet as I am working on another project almost 14 hours per day including weekends. Once I have time, I will comeback to this issue. Meanwhile, could you please explain your use case so that I can understand the scope of the issue? Thanks. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK7n-OyDNB7XtSn-5wKNrI7I8tiVYpyfks5rP8pXgaJpZM4LSEcX>.

-- jesuischarlie/Tunis/Paris/Bruxelles/Berlin Laurent Michel SSC XMM-Newton Tél : +33 (0)3 68 85 24 37 Fax : +33 (0)3 )3 68 85 24 32 Université de Strasbourg <http://www.unistra.fr> Observatoire Astronomique 11 Rue de l'Université F - 67200 Strasbourg

[stevehu]

I like what you guys are doing and I really want to help. You use case is a looped reference and I cannot image what is the real scenario that need it. Are you trying to build a menu? or some navigation tree? Thanks.

[lmichel]

Building a menu? You are almost right!
In the schema I'm working with there is a parameter description using a nested definition for the options.
These parameters could be used to describe the way to invoke a service. In a real life, I never seen such nested definitions, but they are supported by our schema.
Below is the XSD snippet:

 <xs:complexType name="Option">
    <xs:sequence>
      <xs:element name="OPTION" type="Option" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute name="name" type="xs:token"/>
    <xs:attribute name="value" type="xs:string" use="required"/>
  </xs:complexType>

Starting from this, I made is very simple schema selfref.json focused on this feature.

[abayer]

I'm hitting the same problem - validating against https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/e4aae36cca402d8612e0c69110f9e7a9128f0ffc/pipeline-model-api/src/main/resources/ast-schema.json just goes into stack overflow hell. I'd really love to be able to use this.

[stevehu]

@abayer Do you know if other Java library supports nested references?

[slinkydeveloper]

I think that I'm affected of this bug. Try to load this schema:

{
  "required": [
    "id_card",
    "name",
    "surname"
  ],
  "properties": {
    "name": {
      "type": "string"
    },
    "surname": {
      "type": "string"
    },
    "id_card": {
      "type": "string"
    },
    "mother": {
      "$ref": "#/definitions/PersonRecursive"
    },
    "father": {
      "$ref": "#/definitions/PersonRecursive"
    }
  },
  "definitions": {
    "PersonRecursive": {
      "allOf": [
        {
          "$ref": "#/definitions/Person"
        },
        {
          "$ref": "#/definitions/Citizen"
        },
        {
          "type": "object",
          "properties": {
            "mother": {
              "$ref": "#/definitions/PersonRecursive"
            },
            "father": {
              "$ref": "#/definitions/PersonRecursive"
            }
          }
        }
      ]
    },
    "Person": {
      "required": [
        "name",
        "surname"
      ],
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "surname": {
          "type": "string"
        }
      }
    },
    "Citizen": {
      "required": [
        "id_card"
      ],
      "type": "object",
      "properties": {
        "id_card": {
          "type": "string"
        }
      }
    }
  }
}

If I try to load this schema, it get stucks during JsonSchemaFactory().getSchema(schema)

[stevehu]

This issue has been bugging us for a long time. I am planning to refactor this library during the holiday season if I can find one or two free days. The major reason this is not fixed is due to lack of ideas on how to get it resolved. Let's brainstorm on how to get it fixed. I don't think setting a limit on the loop is a good idea.

[slinkydeveloper]

@stevehu If you need an help we can work on it together. We can also work to extend the library to support OAS 3 things like discriminator #35

[stevehu]

That would be great. I also have a library that parse and validate OAS 3 and some of the ideas can be borrowed from there. What I was thinking is the use service module in light-4j so that anyone can expend the library to replace existing validator and adding new validators. The openapi-parser does that.

https://github.com/networknt/openapi-parser

[janakosar]

@stevehu Are you planning to resolve issue with nested references?

[stevehu]

@janakosar This issue is not resolved yet as it is a very specific use case and we haven't encountered the same issue from other users since then. It slipped from our attention because it was closed by the originator. I have reopened it and see if someone can help. Thanks for bringing it up.

[stevehu]

resolved by #141