Documentation for batched inference with torchserve

[msaroufim]

No description provided.

[KrzysztofPrzygodzki]

Hi! I didn't know where to put it, but I believe there is a problem with example of batch processing: resnet_152_batch.

There is an exemplary config.properties which has section models where properties like batchsize etc. are defined - README.md#L67 Unfortunately it is ignored by server (Torchserve version: 0.4.0). Model is registered, but parameters are not set.
I had to use model_snapshot={\ "name": "startup.cfg",\ "modelCount": 1,\ "models": {\ ...

Regards!

[HamidShojanazeri]

@KrzysztofPrzygodzki Sorry for the confusion, this feature would be available from 0.4.1 as mentioned in the doc which is coming in couple of weeks, for now you can use this feature by installing from src.

[KrzysztofPrzygodzki]

@HamidShojanazeri
I see. I had to missed it, sorry. Thank you for clarifications.

[HamidShojanazeri]

@KrzysztofPrzygodzki Pls feel free to open a new issue if you ran into any problem with setting them through config.properties.