Unable to generate anything with multiple files yaml spec #5753

JFCote · 2017-05-31T19:16:03Z

Description

I'm trying to generate the JavaPlayFramework with a very simple multiple files yaml spec but it doesn't work. I also tried with other server and I receive the same error. I was advised to try to validate the spec using the brand new "validate" function but I still receive the same error.

What is strange is that I am able to look at this spec using swagger-editor in local. There is not error at all. Just put both file in the swagger-editor folder and see for yourself.

There are 2 files (find the content of them below):

swagger.yaml
definitions.yaml

Here is the error I get every time

[main] INFO io.swagger.parser.Swagger20Parser - reading from swagger-editor/swagger.yaml
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -7
        at java.lang.String.substring(Unknown Source)
        at io.swagger.models.refs.GenericRef.computeSimpleRef(GenericRef.java:86)
        at io.swagger.models.refs.GenericRef.<init>(GenericRef.java:29)
        at io.swagger.models.properties.RefProperty.set$ref(RefProperty.java:56)
        at io.swagger.models.properties.RefProperty.<init>(RefProperty.java:18)
        at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:214)
        at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:286)
        at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:137)
        at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:39)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3584)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3508)
        at io.swagger.parser.util.SwaggerDeserializer.property(SwaggerDeserializer.java:974)
        at io.swagger.parser.util.SwaggerDeserializer.definition(SwaggerDeserializer.java:789)
        at io.swagger.parser.ResolverCache.loadRef(ResolverCache.java:135)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:34)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefProperty(ExternalRefProcessor.java:119)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:83)
        at io.swagger.parser.processors.PropertyProcessor.processRefProperty(PropertyProcessor.java:34)
        at io.swagger.parser.processors.PropertyProcessor.processProperty(PropertyProcessor.java:21)
        at io.swagger.parser.processors.ResponseProcessor.processResponse(ResponseProcessor.java:21)
        at io.swagger.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:45)
        at io.swagger.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:101)
        at io.swagger.parser.SwaggerResolver.resolve(SwaggerResolver.java:50)
        at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:67)
        at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:420)
        at io.swagger.codegen.cmd.Generate.run(Generate.java:232)
        at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:43)

Swagger-codegen version

Latest master

Swagger declaration file content or url

swagger.yaml

swagger: '2.0'

info:
  version: "0.1.0"
  title: Title
  description: ""
  contact:
    name: me
    email: [email protected]
    url: http://www.foo.com
schemes:
  - http
  - https
basePath: /api
host: "localhost:9000"
consumes:
  - application/json
produces:
  - application/json
paths:
  /actors:
    delete:
      tags:
        - Actor
      summary: Kill an actor in the actor system (MAW)
      description: ""
      operationId: killActor
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/DefaultMetaOnlyResponse"
    get:
      tags:
        - Actor
      summary: Request the status of an actor
      description: ""
      operationId: requestPrintStatus
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/DefaultMetaOnlyResponse"

definitions.yaml

EmptyObject:
  type: object
  description: "Sample Description"
Error:
  type: object
  description: "Sample Description"
  required:
    - type
  properties:
    type:
      type: string
      description: "Sample Description"
      default: "GenericException"
      example: "GenericException"
    itemInfo:
      type: object
      description: "Sample Description"
      items:
        $ref: '#/ItemId'
    details:
      type: string
      description: "Sample Description"
      example: "Could not update that field"
    exception:
      type: string
      description: "Sample Description"
      example: "DBException + stack trace"
ItemId:
  type: object
  description: "Sample Description"
  required:
    - id
    - type
  properties:
    id:
      type: string
      description: "Sample Description"
      example: "45"
    type:
      type: string
      description: "playlist"
      example: "5667"
ResponseMeta:
  type: object
  description: "Sample Description"
  required:
    - code
  properties:
    code:
      type: string
      description: "Sample Description"
      default: Ok
      enum:
      - Ok
      example: Ok
    detail:
      type: string
      description: "Sample Description"
      example: "this is some detail about the error or the success"
    exception:
      type: string
      description: "Sample Description"
      example: "IOException + stack trace"
    errors:
      type: array
      description: "Sample Description"
      items:
        $ref: '#/Error'
ActorInfo:
  type: object
  description: "Sample Description"
  required:
    - actorPath
  properties:
    actorPath:
      type: string
      example: "/user/synchronizer"
DefaultMetaOnlyResponse:
  type: object
  required:
    - meta
  properties:
    meta:
      $ref: '#/ResponseMeta'

Command line used for generation

java -jar swagger-codegen.jar generate -i swagger-editor/swagger.yaml -l java-play-framework -o generatedServer -DhideGenerationTimestamp=true

Steps to reproduce

Just run the command line above

The text was updated successfully, but these errors were encountered:

jimschubert · 2017-06-01T02:33:23Z

I'd never seen external refs defined outside of a definitions key. I checked swagger-parser, and it should be supported.

I ran the following against the csharp generator with a parent definitions key in your definitions.yaml, and it generated for me.

swagger.yaml:

swagger: '2.0'

info:
  version: "0.1.0"
  title: Title
  description: ""
  contact:
    name: me
    email: [email protected]
    url: http://www.foo.com
schemes:
  - http
  - https
basePath: /api
host: "localhost:9000"
consumes:
  - application/json
produces:
  - application/json
paths:
  /actors:
    delete:
      tags:
        - Actor
      summary: Kill an actor in the actor system (MAW)
      description: ""
      operationId: killActor
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/definitions/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/definitions/DefaultMetaOnlyResponse"
    get:
      tags:
        - Actor
      summary: Request the status of an actor
      description: ""
      operationId: requestPrintStatus
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/definitions/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/definitions/DefaultMetaOnlyResponse"

definitions.yaml:

definitions:
  EmptyObject:
    type: object
    description: "Sample Description"
  Error:
    type: object
    description: "Sample Description"
    required:
      - type
    properties:
      type:
        type: string
        description: "Sample Description"
        default: "GenericException"
        example: "GenericException"
      itemInfo:
        type: object
        description: "Sample Description"
        items:
          $ref: '#/definitions/ItemId'
      details:
        type: string
        description: "Sample Description"
        example: "Could not update that field"
      exception:
        type: string
        description: "Sample Description"
        example: "DBException + stack trace"
  ItemId:
    type: object
    description: "Sample Description"
    required:
      - id
      - type
    properties:
      id:
        type: string
        description: "Sample Description"
        example: "45"
      type:
        type: string
        description: "playlist"
        example: "5667"
  ResponseMeta:
    type: object
    description: "Sample Description"
    required:
      - code
    properties:
      code:
        type: string
        description: "Sample Description"
        default: Ok
        enum:
        - Ok
        example: Ok
      detail:
        type: string
        description: "Sample Description"
        example: "this is some detail about the error or the success"
      exception:
        type: string
        description: "Sample Description"
        example: "IOException + stack trace"
      errors:
        type: array
        description: "Sample Description"
        items:
          $ref: '#/definitions/Error'
  ActorInfo:
    type: object
    description: "Sample Description"
    required:
      - actorPath
    properties:
      actorPath:
        type: string
        example: "/user/synchronizer"
  DefaultMetaOnlyResponse:
    type: object
    required:
      - meta
    properties:
      meta:
        $ref: '#/definitions/ResponseMeta'

When I run without the parent definitions key, I also receive an error:

$ ./bin/csharp-petstore.sh
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[main] INFO io.swagger.parser.Swagger20Parser - reading from /Users/jim/temp/issues/swagger-codegen/5753/swagger.yaml
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -7
	at java.lang.String.substring(String.java:1919)
	at io.swagger.models.refs.GenericRef.computeSimpleRef(GenericRef.java:86)
	at io.swagger.models.refs.GenericRef.<init>(GenericRef.java:29)
	at io.swagger.models.properties.RefProperty.set$ref(RefProperty.java:56)
	at io.swagger.models.properties.RefProperty.<init>(RefProperty.java:18)
	at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:214)
	at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:286)
	at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:137)
	at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:39)
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3584)
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3508)
	at io.swagger.parser.util.SwaggerDeserializer.property(SwaggerDeserializer.java:974)
	at io.swagger.parser.util.SwaggerDeserializer.definition(SwaggerDeserializer.java:789)
	at io.swagger.parser.ResolverCache.loadRef(ResolverCache.java:135)
	at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:34)
	at io.swagger.parser.processors.ExternalRefProcessor.processRefProperty(ExternalRefProcessor.java:119)
	at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:83)
	at io.swagger.parser.processors.PropertyProcessor.processRefProperty(PropertyProcessor.java:34)
	at io.swagger.parser.processors.PropertyProcessor.processProperty(PropertyProcessor.java:21)
	at io.swagger.parser.processors.ResponseProcessor.processResponse(ResponseProcessor.java:21)
	at io.swagger.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:45)
	at io.swagger.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:101)
	at io.swagger.parser.SwaggerResolver.resolve(SwaggerResolver.java:50)
	at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:67)
	at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:431)
	at io.swagger.codegen.cmd.Generate.run(Generate.java:239)
	at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:43)

It seems to me like this would be covered by the tests hitting relative-file-references in swagger-parser, although those are json-only and this could be a yaml issue.

You may want to track swagger-api/swagger-parser#342, as it looks like it's missing functionality (or perhaps a bug) in the YAML processing.

jimschubert · 2017-06-01T02:41:41Z

I should mention, current master was updated today (9 hours ago, issue reported 7 hours ago) to use swagger-parser 1.0.29. I tested this both with 1.0.28 and 1.0.29 and received the same error when there's no parent definitions key in the externally referenced file.

fehguy · 2017-06-01T03:37:36Z

Could be the case that we need to add a test to the parser for this. It should be handled...

JFCote · 2017-06-01T12:02:07Z

@jimschubert and @fehguy : I'm pretty sure it should be supported by swagger-codegen because it seems to be supported by the swagger-editor and swagger-ui (and probably by the OpenApi spec itself).

I followed this article on how to split a single yaml file into multiple files and as you can see, there is no more "definitions" key once it is split and it make sense.

https://apihandyman.io/writing-openapi-swagger-specification-tutorial-part-8-splitting-specification-file/

I'm pretty sure more and more people will start to split their files and will follow this blog post and swagger-codegen must be compliant.

Is this an easy fix? I don't know anything about the parser unfortunately so I don't think I can contribute myself.

jimschubert · 2017-06-01T13:13:41Z

@JFCote I don't think it's a swagger-codegen issue. The third line from the bottom of my stack trace is in CodegenConfigurator, which calls

Swagger swagger = new SwaggerParser().read(inputSpec, authorizationValues, true);

The exception that's thrown is in swagger-core, and looks like it has to do with getting the name of a reference. As @fehguy said, it should be covered.

I'm heading out on vacation today, but I'll try to look later.

jimschubert · 2017-06-02T02:56:13Z

@fehguy I tracked down the problem. This is an issue with swagger-parser. I'm on vacation for a week, so I was wondering if you could look into a fix?

The issue is that the code is getting into GenericRef#computeSimpleRef with a RefFormat.INTERNAL and RefType.DEFINITION for which the prefix is defined as #/definitions/ by the RefType enum. If an internal type definition is a shorter length than #/definitions/ count, line 82 of GenericRef:

result = ref.substring(prefix.length());

Throws an error. For example, n the sample case of this issue, it's doing "#/Error".substring("#/definitions/".length()).

It's weird to me that swagger-parser loosely allows external refs to be defined as #/Error rather than #/definitions/Error, and after stepping through the code I can understand why I've only ever seen it as #/definitions/Error.

If I rename #/Error in the sample above to #/ErrorErrorError, swagger-codegen generates successfully.

I haven't worked in the swagger-parser codebase, so I wouldn't really know where to start with the fix. Intuitively, I'd think the fix would be to just strictly enforce the prefix. However, I'm not too familiar with the spec on external refs and it looks like the test files aren't prefixed with #/definitions/. Maybe the issue is that RefFormat needs an EXTERNAL option which can be special cased in computeSimpleRef?

2ricecrackerfolder · 2017-06-13T23:36:03Z

Hi Jim,

I added the "definitions:" in the parent hierarchy as mentioned above. In the UI I was able to display my api, but when I ran the code generator in java, everything ran and the response models were not created. They were referenced in the other models but not created. They would have this name: SnapAdYamldefinitionssnapAd. Is this related? Does the code-gen not support multiple files?

Greg

jimschubert · 2017-06-13T23:46:13Z

@2ricecrackerfolder the external yaml generated for me in the local filesystem with the examples above. I didn't test it with hosted specs and external definitions Is that how you're generating?

2ricecrackerfolder · 2017-06-14T00:09:47Z

@jimschubert Thanks for your quick reply! I also modified my configuration to what they have above. The yaml did generate with no errors but was missing the response model files when i dug through the generated src.

2ricecrackerfolder · 2017-06-14T00:11:06Z

I have a large api with a lot of definitions. I am splitting the files, and it seems like the response models would not generate.

jimschubert · 2017-06-14T01:39:21Z

@2ricecrackerfolder my assumption would be that you may have a validation error somewhere.

You can check with the online validator by appending your url:
https://online.swagger.io/validator/debug?url=

For example, here's a swagger definition with a validation error: https://online.swagger.io/validator/debug?url=https://quay.io/api/v1/discovery

I don't think the above definition would have a problem when generating client code, but if you have lots of errors it's more likely.

fehguy · 2017-06-14T04:42:34Z

@jimschubert looks like a bug in swagger-core is triggering this. Bummer, we just did a release today! But please see here:

swagger-api/swagger-core#2255

fehguy · 2017-06-14T04:55:02Z

Updating to swagger-parser 1.0.30-SNAPSHOT to see if this fixes the issue

JFCote · 2017-06-14T12:11:36Z

@fehguy : Let me know when it's in the master, I will do some testing.

fehguy · 2017-06-14T15:38:16Z

You should be able to update this:

https://github.com/swagger-api/swagger-codegen/blob/master/pom.xml#L890

to 1.0.30-SNAPSHOT. Can you try that and report back? Thanks a bunch! The logic was a little weird in the current release so I'm pretty sure this will fix it.

JFCote · 2017-06-15T13:29:22Z

@fehguy : I have tested it with the example I give above and the generation was successful! So I'm pretty sure we have a good fix with the new swagger-parser version. Please update this bug when you will update the parser in the master! Thanks!

fehguy · 2017-06-16T14:23:04Z

OK Great! I'll do the release today. We have to do the whole gamut including swagger-core too...

francescobianco · 2018-03-08T20:10:35Z

@JFCote @jimschubert @fehguy I sugget to build file using command line toolkit that link multiple yaml files with $include tag . Look this https://github.com/javanile/yamlinc add in build task on gulp or grunt

JFCote · 2018-03-08T20:28:12Z

@francescobianco It's already fixed

rajkash · 2018-08-08T21:23:50Z

Hi @JFCote ,

Which version of swagger-codegen is this fixed in ? I am still getting the same error for swagger-codegen-maven-plugin 2.2.2 version

francescobianco · 2018-08-09T09:56:06Z

Hi @rajkash please use https://github.com/javanile/yamlinc to build your multiple Yaml files documentation
in this way your files are quite clean and arranged as you want.
I ask @JFCote @jimschubert @fehguy: why swagger parser need $ref but as this limitations:

A common misconception is that $ref is allowed anywhere in an OpenAPI specification file. Actually $ref is only allowed in places where the OpenAPI 3.0 Specification explicitly states that the value may be a reference. For example, $ref cannot be used in the info section and directly under paths:

https://swagger.io/docs/specification/using-ref/

jimschubert mentioned this issue Jun 1, 2017

$ref in additionalProperties in external yaml ignored swagger-api/swagger-parser#342

Closed

fehguy mentioned this issue Jun 14, 2017

GenericRef gives OOB exception swagger-api/swagger-core#2254

Closed

fehguy self-assigned this Jun 16, 2017

fehguy mentioned this issue Jun 16, 2017

Update Swagger Codegen to v1.5.15 #5858

Merged

fehguy closed this as completed in #5858 Jun 26, 2017

m-y-mo mentioned this issue Jun 29, 2017

[CPP] Fixing some more ordering of checks #5951

Merged

francescobianco unassigned fehguy Mar 8, 2018

Unable to generate anything with multiple files yaml spec #5753

Unable to generate anything with multiple files yaml spec #5753

Comments

JFCote commented May 31, 2017

Description

Swagger-codegen version

Swagger declaration file content or url

Command line used for generation

Steps to reproduce

jimschubert commented Jun 1, 2017

Uh oh!

jimschubert commented Jun 1, 2017

Uh oh!

fehguy commented Jun 1, 2017

Uh oh!

JFCote commented Jun 1, 2017

Uh oh!

jimschubert commented Jun 1, 2017

Uh oh!

jimschubert commented Jun 2, 2017

Uh oh!

2ricecrackerfolder commented Jun 13, 2017

Uh oh!

jimschubert commented Jun 13, 2017

Uh oh!

2ricecrackerfolder commented Jun 14, 2017

Uh oh!

2ricecrackerfolder commented Jun 14, 2017

Uh oh!

jimschubert commented Jun 14, 2017

Uh oh!

fehguy commented Jun 14, 2017

Uh oh!

fehguy commented Jun 14, 2017

Uh oh!

JFCote commented Jun 14, 2017

Uh oh!

fehguy commented Jun 14, 2017

Uh oh!

JFCote commented Jun 15, 2017

Uh oh!

fehguy commented Jun 16, 2017

Uh oh!

francescobianco commented Mar 8, 2018

Uh oh!

JFCote commented Mar 8, 2018

Uh oh!

rajkash commented Aug 8, 2018

Uh oh!

francescobianco commented Aug 9, 2018

Uh oh!