Add serDes setting : serialize and deserialize (#506)

pilerou · web-flow · commit 1ca87c7f3b1a · 2021-02-14T18:58:10.000-05:00
* `serDes` setting allows to `serialize` response objects and `deserialize` request parameters or fields. It could resolve : #353 #465 #288 #246 Unit tests validate Date and MongoDb ObjectID. Developers have choice to : - only serialize response contents - also deserialize request strings to custom objects Frequent SerDes are defined in base.serdes.ts (date and date-time). Documentation updated with this setting * I don't know why there was a cloneDeep but it seems to be necessary in all cases. * Fix validation problems with oneOf and force default SerDes when no SerDes are defined (force DateTime and Date serialization). * Delete old code comments * New test : If I answer with an object which serialize fails (here because I answer with an ObjectID instead of Date and no toISOString function exists), a 500 error is thrown. * Add Date and date-time serialization by default in addition to other serDes settings. Custom settings can also override date and date-time formats in order to also deserialize date and/or date-time on requests * Add Date and date-time serialization by default in addition to other serDes settings. Custom settings can also override date and date-time formats in order to also deserialize date and/or date-time on requests * `serDes` option adaptation to be more user friendly Test OK Documentation is modified I also changed my https://github.com/pilerou/mongo-serdes-js with a 0.0.3 version which is compliant to the design : ```javascript serDes: [ OpenApiValidator.baseSerDes.date.serializer, OpenApiValidator.baseSerDes.dateTime, MongoSerDes.objectid, // this configuration if we want to deserialize objectid in request and serialize it in response MongoSerDes.uuid.serializer, // this configuration if we only want to serialize on response ], ``` * When we add custom formats in serDes, they are automatically added to unknownFormats * Rename OpenApiValidator.baseSerDes to OpenApiValidator.serdes
diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@
 - ✔️ request validation
 - ✔️ response validation (json only)
 - 👮 security validation / custom security functions
-- 👽 3rd party / custom formats
+- 👽 3rd party / custom formats / custom data serialization-deserialization
 - 🧵 optionally auto-map OpenAPI endpoints to Express handler functions
 - ✂️ **\$ref** support; split specs over multiple files
 - 🎈 file upload
@@ -493,12 +493,21 @@ OpenApiValidator.middleware({
     validate: (value: any) => boolean,
   }],
   unknownFormats: ['phone-number', 'uuid'],
+  serDes: [{
+    OpenApiValidator.serdes.dateTime,
+    OpenApiValidator.serdes.date,
+    {
+      format: 'mongo-objectid',
+      deserialize: (s) => new ObjectID(s),
+      serialize: (o) => o.toString(),
+    },
+  }],
   operationHandlers: false | 'operations/base/path' | { ... },
   ignorePaths: /.*\/pets$/,
   fileUploader: { ... } | true | false,
   $refParser: {
     mode: 'bundle'
-  }
+  },
 });
 ```
 
@@ -731,6 +740,53 @@ Defines how the validator should behave if an unknown or custom format is encoun
 
 - `"ignore"` - to log warning during schema compilation and always pass validation. This option is not recommended, as it allows to mistype format name and it won't be validated without any error message.
 
+### ▪️ serDes (optional)
+
+Default behaviour convert `Date` objects to `string` when a field, in OpenAPI configuration, has a `format` setting set to `date` or `date-time`.
+This Date conversion only occurs before sending the response.
+
+You can use `serDes` option to add custom mecanism that :
+- `deserialize` string to custom object (Date...) on request
+  - Deserialization is made after other schema validation (`pattern`...)
+- `serialize` object before sending the response 
+  - Serialization is made instead of other validation. No `pattern` or other rule is checked.
+
+The goal of `serDes` option is to focus route functions on feature and without having to cast data on request or before sending response.
+
+To both `deserialize` on request and `serialize` on response, both functions must be defined and are launched when schema `format` fields match. 
+```javascript
+serDes: [{
+  OpenApiValidator.serdes.dateTime, // used when 'format: date-time' 
+  OpenApiValidator.serdes.date, // used when 'format: date' 
+  {
+    format: 'mongo-objectid',
+    deserialize: (s) => new ObjectID(s),
+    serialize: (o) => o.toString(),
+  }
+}],
+```
+
+If you ONLY want to `serialize` response data (and avoid to deserialize on request), the configuration must not define `deserialize` function.
+```javascript
+serDes: [{
+  // No need to declare date and dateTime. Those types deserialization is already done by default.
+  //OpenApiValidator.serdes.dateTime.serializer, 
+  //OpenApiValidator.serdes.date.serializer, 
+  {
+    format: 'mongo-objectid',
+    serialize: (o) => o.toString(),
+  }
+}],
+```
+So, in conclusion, you can use `OpenApiValidator.serdes.dateTime` if you can to serialize and deserialize dateTime.
+You can also use `OpenApiValidator.serdes.dateTime.serializer` if you only want to serialize or `OpenApiValidator.serdes.dateTime.deserializer` if you only want to deserialize.
+
+NOTE : If you add custom formats in serDes, they are automatically added as accepted custom formats in [unknownFormats](#unknownFormats-(optional)) option setting. 
+You don't need to add them in unknownFormats.
+ 
+You may want to use `serDes` option for MongoDB types (ObjectID, UUID...). 
+Then you can use the package [mongo-serdes-js](https://github.com/pilerou/mongo-serdes-js). It is designed to be a good addition to this package.
+
 ### ▪️ operationHandlers (optional)
 
 Defines the base directory for operation handlers. This is used in conjunction with express-openapi-validator's OpenAPI vendor extensions, `x-eov-operation-id`, `x-eov-operation-handler` and OpenAPI's `operationId`. See [example](https://github.com/cdimascio/express-openapi-validator/tree/master/examples/3-eov-operations).
diff --git a/src/framework/ajv/index.ts b/src/framework/ajv/index.ts
@@ -36,6 +36,23 @@ function createAjv(
   ajv.removeKeyword('const');
 
   if (request) {
+    if (options.serDesMap) {
+      ajv.addKeyword('x-eov-serdes', {
+        modifying: true,
+        compile: (sch) => {
+          if (sch) {
+            return function validate(data, path, obj, propName) {
+              if (typeof data === 'object') return true;
+              if(!!sch.deserialize) {
+                obj[propName] = sch.deserialize(data);
+              }
+              return true;
+            };
+          }
+          return () => true;
+        },
+      });
+    }
     ajv.removeKeyword('readOnly');
     ajv.addKeyword('readOnly', {
       modifying: true,
@@ -62,20 +79,23 @@ function createAjv(
     });
   } else {
     // response
-    ajv.addKeyword('x-eov-serializer', {
-      modifying: true,
-      compile: (sch) => {
-        if (sch) {
-          const isDate = ['date', 'date-time'].includes(sch.format);
-          return function validate(data, path, obj, propName) {
-            if (typeof data === 'string' && isDate) return true
-            obj[propName] = sch.serialize(data);
-            return true;
-          };
-        }
-        return () => true;
-      },
-    });
+    if (options.serDesMap) {
+      ajv.addKeyword('x-eov-serdes', {
+        modifying: true,
+        compile: (sch) => {
+          if (sch) {
+            return function validate(data, path, obj, propName) {
+              if (typeof data === 'string') return true;
+              if(!!sch.serialize) {
+                obj[propName] = sch.serialize(data);
+              }
+              return true;
+            };
+          }
+          return () => true;
+        },
+      });
+    }
     ajv.removeKeyword('writeOnly');
     ajv.addKeyword('writeOnly', {
       modifying: true,
diff --git a/src/framework/base.serdes.ts b/src/framework/base.serdes.ts
@@ -0,0 +1,26 @@
+import { SerDes, SerDesSingleton } from './types';
+
+export const dateTime : SerDesSingleton = new SerDesSingleton({
+  format : 'date-time',
+  serialize: (d: Date) => {
+    return d && d.toISOString();
+  },
+  deserialize: (s: string) => {
+    return new Date(s);
+  }
+});
+
+export const date : SerDesSingleton = new SerDesSingleton({
+  format : 'date',
+  serialize: (d: Date) => {
+    return d && d.toISOString().split('T')[0];
+  },
+  deserialize: (s: string) => {
+    return new Date(s);
+  }
+});
+
+export const defaultSerDes : SerDes[] = [
+  date.serializer,
+  dateTime.serializer
+];
diff --git a/src/framework/types.ts b/src/framework/types.ts
@@ -38,7 +38,7 @@ export interface MultipartOpts {
 
 export interface Options extends ajv.Options {
   // Specific options
-  schemaObjectMapper?: object;
+  serDesMap?: SerDesMap;
 }
 
 export interface RequestValidatorOptions extends Options, ValidateRequestOpts {}
@@ -70,9 +70,40 @@ export type Format = {
   validate: (v: any) => boolean;
 };
 
-export type Serializer = {
+export type SerDes = {
   format: string;
-  serialize: (o: unknown) => string;
+  serialize?: (o: unknown) => string;
+  deserialize?: (s: string) => unknown;
+};
+
+export class SerDesSingleton implements SerDes {
+  serializer: SerDes;
+  deserializer: SerDes;
+  format: string;
+  serialize?: (o: unknown) => string;
+  deserialize?: (s: string) => unknown;
+
+  constructor(param: {
+    format: string;
+    serialize: (o: unknown) => string;
+    deserialize: (s: string) => unknown;
+  }) {
+    this.format = param.format;
+    this.serialize = param.serialize;
+    this.deserialize = param.deserialize;
+    this.deserializer = {
+      format : param.format,
+      deserialize : param.deserialize
+    }
+    this.serializer = {
+      format : param.format,
+      serialize : param.serialize
+    }
+  }
+};
+
+export type SerDesMap = {
+  [format: string]: SerDes
 };
 
 export interface OpenApiValidatorOpts {
@@ -85,6 +116,7 @@ export interface OpenApiValidatorOpts {
   securityHandlers?: SecurityHandlers;
   coerceTypes?: boolean | 'array';
   unknownFormats?: true | string[] | 'ignore';
+  serDes?: SerDes[];
   formats?: Format[];
   fileUploader?: boolean | multer.Options;
   multerOpts?: multer.Options;
diff --git a/src/index.ts b/src/index.ts
@@ -29,6 +29,8 @@ export const error = {
   Forbidden,
 };
 
+export * as serdes from './framework/base.serdes';
+
 function openapiValidator(options: OpenApiValidatorOpts) {
   const oav = new OpenApiValidator(options);
   exports.middleware._oav = oav;
diff --git a/src/middlewares/parsers/schema.preprocessor.ts b/src/middlewares/parsers/schema.preprocessor.ts
@@ -5,7 +5,8 @@ import * as _get from 'lodash.get';
 import { createRequestAjv } from '../../framework/ajv';
 import {
   OpenAPIV3,
-  Serializer,
+  SerDesMap,
+  Options,
   ValidateResponseOpts,
 } from '../../framework/types';
 
@@ -48,20 +49,6 @@ class Root<T> extends Node<T, T> {
   }
 }
 
-const dateTime: Serializer = {
-  format: 'date-time',
-  serialize: (d: Date) => {
-    return d && d.toISOString();
-  },
-};
-
-const date: Serializer = {
-  format: 'date',
-  serialize: (d: Date) => {
-    return d && d.toISOString().split('T')[0];
-  },
-};
-
 type SchemaObject = OpenAPIV3.SchemaObject;
 type ReferenceObject = OpenAPIV3.ReferenceObject;
 type Schema = ReferenceObject | SchemaObject;
@@ -87,14 +74,16 @@ export class SchemaPreprocessor {
   private ajv: Ajv;
   private apiDoc: OpenAPIV3.Document;
   private apiDocRes: OpenAPIV3.Document;
+  private serDesMap: SerDesMap;
   private responseOpts: ValidateResponseOpts;
   constructor(
     apiDoc: OpenAPIV3.Document,
-    ajvOptions: ajv.Options,
+    ajvOptions: Options,
     validateResponsesOpts: ValidateResponseOpts,
   ) {
     this.ajv = createRequestAjv(apiDoc, ajvOptions);
     this.apiDoc = apiDoc;
+    this.serDesMap = ajvOptions.serDesMap;
     this.responseOpts = validateResponsesOpts;
   }
 
@@ -356,19 +345,9 @@ export class SchemaPreprocessor {
     schema: SchemaObject,
     state: TraversalState,
   ) {
-    if (state.kind === 'res') {
-      if (schema.type === 'string' && !!schema.format) {
-        switch (schema.format) {
-          case 'date-time':
-            (<any>schema).type = ['object', 'string'];
-            schema['x-eov-serializer'] = dateTime;
-            break;
-          case 'date':
-            (<any>schema).type = ['object', 'string'];
-            schema['x-eov-serializer'] = date;
-            break;
-        }
-      }
+    if (schema.type === 'string' && !!schema.format && this.serDesMap[schema.format]) {
+      (<any>schema).type = ['object', 'string'];
+      schema['x-eov-serdes'] = this.serDesMap[schema.format];
     }
   }
 
diff --git a/src/openapi.validator.ts b/src/openapi.validator.ts
@@ -20,8 +20,10 @@ import {
 } from './framework/types';
 import { defaultResolver } from './resolvers';
 import { OperationHandlerOptions } from './framework/types';
+import { defaultSerDes } from './framework/base.serdes';
 import { SchemaPreprocessor } from './middlewares/parsers/schema.preprocessor';
 
+
 export {
   OpenApiValidatorOpts,
   InternalServerError,
@@ -341,7 +343,27 @@ export class OpenApiValidator {
   }
 
   private normalizeOptions(options: OpenApiValidatorOpts): void {
-    // Modify the request
+    if(!options.serDes) {
+      options.serDes = defaultSerDes;
+    }
+    else {
+      if(!Array.isArray(options.unknownFormats)) {
+        options.unknownFormats = Array<string>();
+      }
+      options.serDes.forEach(currentSerDes => {
+        if((options.unknownFormats as string[]).indexOf(currentSerDes.format) === -1) {
+          (options.unknownFormats as string[]).push(currentSerDes.format)
+        }
+      });
+      defaultSerDes.forEach(currentDefaultSerDes => {
+        let defautSerDesOverride = options.serDes.find(currentOptionSerDes => {
+          return currentDefaultSerDes.format === currentOptionSerDes.format;
+        });
+        if(!defautSerDesOverride) {
+          options.serDes.push(currentDefaultSerDes);
+        }
+      });
+    }
   }
 
   private isOperationHandlerOptions(
@@ -393,7 +415,22 @@ class AjvOptions {
   }
 
   private baseOptions(): Options {
-    const { coerceTypes, unknownFormats, validateFormats } = this.options;
+    const { coerceTypes, unknownFormats, validateFormats, serDes } = this.options;
+    const serDesMap = {};
+    for (const serDesObject of serDes) {
+      if(!serDesMap[serDesObject.format]) {
+        serDesMap[serDesObject.format] = serDesObject;
+      }
+      else {
+        if (serDesObject.serialize) {
+          serDesMap[serDesObject.format].serialize = serDesObject.serialize;
+        }
+        if (serDesObject.deserialize) {
+          serDesMap[serDesObject.format].deserialize = serDesObject.deserialize;
+        }
+      }
+    }
+
     return {
       nullable: true,
       coerceTypes,
@@ -408,6 +445,7 @@ class AjvOptions {
         };
         return acc;
       }, {}),
+      serDesMap : serDesMap,
     };
   }
 }
diff --git a/test/resources/serdes.yaml b/test/resources/serdes.yaml
diff --git a/test/serdes.spec.ts b/test/serdes.spec.ts
