[Feature] Support RTMDet-Ins. #1662

RangiLyu · 2023-01-16T11:53:00Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

RTMDet-Ins ONNX and TensorRT support.

Modification

Add rewriter for RTMDetInsHead
Fix the rescale of the object detection model class(need to check other models)

Inference Results(TRT)

Please use configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py

mask AP is aligned with pytorch RTMDet-tiny

Evaluate annotation type *segm*
DONE (t=38.74s).
Accumulating evaluation results...
DONE (t=7.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.354
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.550
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.376
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.131
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.383
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.567
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.302
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.475
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.502
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.252
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.569
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.718
01/17 12:42:14 - mmengine - INFO - segm_mAP_copypaste: 0.354 0.550 0.376 0.131 0.383 0.567

Notice

Currently only support single batch inference.

CLAassistant · 2023-01-16T11:53:05Z

All committers have signed the CLA.

mmdeploy/codebase/mmdet/models/dense_heads/rtmdet_ins_head.py

1095788063 · 2023-01-18T07:25:50Z

转换后推理无结果，还有转换时的推理图片报错和位置也不对。

1095788063 · 2023-01-18T07:26:29Z

1095788063 · 2023-01-18T07:27:07Z

1095788063 · 2023-01-18T07:28:03Z

模型已经加载成功，但是推理每个结果返回

1095788063 · 2023-01-18T07:29:50Z

没有结果返回，但是却有耗时

1095788063 · 2023-01-18T07:30:15Z

D:\Python\Python38\python.exe D:\ayjdata\Code\Deep_learning\OpenMMLab\mmdeploy\mmdeploy-rtmdet_ins_2023-01-18\tools\check_env.py
01/18 15:30:06 - mmengine - INFO -

01/18 15:30:06 - mmengine - INFO - Environmental information
01/18 15:30:09 - mmengine - INFO - sys.platform: win32
01/18 15:30:09 - mmengine - INFO - Python: 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
01/18 15:30:09 - mmengine - INFO - CUDA available: True
01/18 15:30:09 - mmengine - INFO - numpy_random_seed: 2147483648
01/18 15:30:09 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 3070
01/18 15:30:09 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3
01/18 15:30:09 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.58
01/18 15:30:09 - mmengine - INFO - MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30147 版
01/18 15:30:09 - mmengine - INFO - GCC: n/a
01/18 15:30:09 - mmengine - INFO - PyTorch: 1.11.0+cu113
01/18 15:30:09 - mmengine - INFO - PyTorch compiling details: PyTorch built with:

C++ Version: 199711
MSVC 192829337
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
OpenMP 2019
LAPACK is enabled (usually provided by MKL)
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.2
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/actions-runner/_work/pytorch/pytorch/builder/windows/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

01/18 15:30:09 - mmengine - INFO - TorchVision: 0.12.0+cu113
01/18 15:30:09 - mmengine - INFO - OpenCV: 4.5.5
01/18 15:30:09 - mmengine - INFO - MMEngine: 0.4.0
01/18 15:30:09 - mmengine - INFO - MMCV: 2.0.0rc3
01/18 15:30:09 - mmengine - INFO - MMCV Compiler: MSVC 192829924
01/18 15:30:09 - mmengine - INFO - MMCV CUDA Compiler: 11.3
01/18 15:30:09 - mmengine - INFO - MMDeploy: 1.0.0rc1+
01/18 15:30:09 - mmengine - INFO -

01/18 15:30:09 - mmengine - INFO - Backend information
01/18 15:30:09 - mmengine - INFO - tensorrt: 8.2.5.1
01/18 15:30:09 - mmengine - INFO - tensorrt custom ops: Available
01/18 15:30:10 - mmengine - INFO - ONNXRuntime: 1.8.1
01/18 15:30:10 - mmengine - INFO - ONNXRuntime-gpu: None
01/18 15:30:10 - mmengine - INFO - ONNXRuntime custom ops: NotAvailable
01/18 15:30:10 - mmengine - INFO - pplnn: None
01/18 15:30:10 - mmengine - INFO - ncnn: None
01/18 15:30:10 - mmengine - INFO - snpe: None
01/18 15:30:10 - mmengine - INFO - openvino: None
01/18 15:30:10 - mmengine - INFO - torchscript: 1.11.0+cu113
01/18 15:30:10 - mmengine - INFO - torchscript custom ops: Available
01/18 15:30:10 - mmengine - INFO - rknn-toolkit: None
01/18 15:30:10 - mmengine - INFO - rknn-toolkit2: None
01/18 15:30:10 - mmengine - INFO - ascend: None
01/18 15:30:10 - mmengine - INFO - coreml: None
01/18 15:30:10 - mmengine - INFO - tvm: None
01/18 15:30:10 - mmengine - INFO -

01/18 15:30:10 - mmengine - INFO - Codebase information
01/18 15:30:10 - mmengine - INFO - mmdet: 3.0.0rc5
01/18 15:30:10 - mmengine - INFO - mmseg: None
01/18 15:30:10 - mmengine - INFO - mmcls: None
01/18 15:30:10 - mmengine - INFO - mmocr: None
01/18 15:30:10 - mmengine - INFO - mmedit: None
01/18 15:30:10 - mmengine - INFO - mmdet3d: None
01/18 15:30:10 - mmengine - INFO - mmpose: None
01/18 15:30:10 - mmengine - INFO - mmrotate: None
01/18 15:30:10 - mmengine - INFO - mmaction: None

进程已结束,退出代码0

mattiasbax · 2023-01-18T09:14:59Z

I also see weird output converting to onnx using the demo input with provided pretrained checkpoints:

i.e:
python ./tools/deploy.py configs/mmdet/instance-seg/instance-seg_onnxruntime_static.py ../mmdetection/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py ../mmdetection/checkpoints/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth demo/resources/det.jpg --work-dir work_dir/RTMInSeg --device cuda:0

produces:

hanrui1sensetime · 2023-01-31T07:14:34Z

Seems get a wrong result.

python ./tools/deploy.py configs/mmdet/instance-seg/instance-seg_tensorrt_dynamic-320x320-1344x1344.py ../../mmdetection/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_s_8xb32-300e_coco/rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth demo/resources/det.jpg --work-dir rtmdet-ins-trt --device cuda:0

RangiLyu · 2023-02-02T05:41:45Z

@mattiasbax @hanrui1sensetime @1095788063 Try to use configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py and configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py

RangiLyu · 2023-02-02T05:46:57Z

@lvhan028 @hanrui1sensetime request review

hanrui1sensetime

May update UT, regression tests, and docs.

hanrui1sensetime · 2023-02-02T12:00:04Z

Using configs you have mentioned above, the visualize is still failed:
Error log is that

  File "/opt/conda/lib/python3.8/site-packages/mmengine/visualization/visualizer.py", line 828, in draw_binary_masks
    assert img.shape[:2] == binary_masks.shape[
AssertionError: `binary_marks` must have the same shape with image

My script is:

python tools/deploy.py configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py ../mmdetection/configs/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco.py https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco/rtmdet-ins_tiny_8xb32-300e_coco_20221130_151727-ec670f7e.pth ../mmdetection/demo/demo.jpg --work-dir ./rtmdet-ins-ort --device cuda:0 --dump-info

In this script: img.shape[:2] is (427, 640), while binary_mask.shape[1:] is (427, 427), FYI.

RangiLyu · 2023-02-03T04:02:43Z

@AllentDan Please review mmdeploy/codebase/mmdet/deploy/object_detection_model.py

RangiLyu · 2023-02-03T04:15:30Z

May update UT, regression tests, and docs.

Unit tests and docs will be added in another PR later.

RangiLyu · 2023-02-03T04:16:53Z

I also see weird output converting to onnx using the demo input with provided pretrained checkpoints:

i.e: python ./tools/deploy.py configs/mmdet/instance-seg/instance-seg_onnxruntime_static.py ../mmdetection/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py ../mmdetection/checkpoints/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth demo/resources/det.jpg --work-dir work_dir/RTMInSeg --device cuda:0

produces:

We have fixed a hard code bug in MMDeploy and now the visualized result should be correct.

hanrui1sensetime

LGTM.

mmdeploy/codebase/mmdet/deploy/object_detection_model.py

AllentDan

RTMDet always creates inputs with static shapes. So there is no need to specify a shape in deploy_cfg. Please set input_shape=None keep pipeline[i].keep_ratio = False.

mmdeploy/codebase/mmdet/deploy/object_detection.py

configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py

AllentDan

LGTM

* [Feature] Support RTMDet-Ins. * fix visualize * fix rewrite trt * add config * support torch 1.13 * fix keep ratio resize * resolve scale factor bug * set to None

[Feature] Support RTMDet-Ins.

f73c3e2

fix visualize

4c38aae

RangiLyu changed the title ~~[WIP][Feature] Support RTMDet-Ins.~~ [Feature] Support RTMDet-Ins. Jan 17, 2023

hanrui1sensetime mentioned this pull request Jan 17, 2023

[Bug] fail to export rtmdet and rtmdet-ins to onnx #1586

Closed

3 tasks

RangiLyu mentioned this pull request Jan 17, 2023

Does RTMDet instance segmentation support onnx conversion? open-mmlab/mmdetection#9615

Closed

fix rewrite trt

3002058

mattiasbax reviewed Jan 17, 2023

View reviewed changes

mmdeploy/codebase/mmdet/models/dense_heads/rtmdet_ins_head.py Outdated Show resolved Hide resolved

RangiLyu mentioned this pull request Jan 18, 2023

[Bug] mmdetection rtmdet 转 tensorrt 或者 onnx 报错 #1633

Open

3 tasks

lvhan028 requested review from hanrui1sensetime and grimoire January 30, 2023 02:23

add config

a4ec132

Merge branch 'dev-1.x' of github.com:open-mmlab/mmdeploy into rtmdet_ins

f88a329

support torch 1.13

eaa0562

hanrui1sensetime reviewed Feb 2, 2023

View reviewed changes

fix keep ratio resize

14c3037

hanrui1sensetime approved these changes Feb 3, 2023

View reviewed changes

AllentDan reviewed Feb 3, 2023

View reviewed changes

mmdeploy/codebase/mmdet/deploy/object_detection_model.py Show resolved Hide resolved

AllentDan reviewed Feb 3, 2023

View reviewed changes

mmdeploy/codebase/mmdet/deploy/object_detection.py Outdated Show resolved Hide resolved

configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py Outdated Show resolved Hide resolved

RangiLyu added 2 commits February 3, 2023 14:40

resolve scale factor bug

040ef84

set to None

5c903da

RangiLyu requested review from AllentDan and removed request for grimoire February 3, 2023 06:56

AllentDan approved these changes Feb 3, 2023

View reviewed changes

lvhan028 merged commit 87e4814 into open-mmlab:dev-1.x Feb 6, 2023

lvhan028 pushed a commit that referenced this pull request Mar 1, 2023

[Feature] Support RTMDet-Ins. (#1662)

3e9e6cd

* [Feature] Support RTMDet-Ins. * fix visualize * fix rewrite trt * add config * support torch 1.13 * fix keep ratio resize * resolve scale factor bug * set to None

irexyc mentioned this pull request Apr 6, 2023

[Feature] RTMDet Instance model export onnx support #1967

Closed

cozeybozey mentioned this pull request May 4, 2023

Error when loading in RTMDet ONNX file in OpenCV [Bug] #2048

Closed

3 tasks

cozeybozey mentioned this pull request May 12, 2023

Is it possible to add batch inference for a deployed RTMDet model? [Feature] #2075

Closed

Lewis-Lu mentioned this pull request Sep 4, 2024

[Export][ONNX] LiDARInstance3DBoxes OpenDriveLab/UniAD#198

Open

[Feature] Support RTMDet-Ins. #1662

[Feature] Support RTMDet-Ins. #1662

Uh oh!

Conversation

RangiLyu commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Inference Results(TRT)

Notice

Uh oh!

CLAassistant commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

1095788063 commented Jan 18, 2023

Uh oh!

mattiasbax commented Jan 18, 2023

Uh oh!

hanrui1sensetime commented Jan 31, 2023

Uh oh!

RangiLyu commented Feb 2, 2023

Uh oh!

RangiLyu commented Feb 2, 2023

Uh oh!

hanrui1sensetime left a comment

Choose a reason for hiding this comment

Uh oh!

hanrui1sensetime commented Feb 2, 2023

Uh oh!

RangiLyu commented Feb 3, 2023

Uh oh!

RangiLyu commented Feb 3, 2023

Uh oh!

RangiLyu commented Feb 3, 2023

Uh oh!

hanrui1sensetime left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AllentDan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AllentDan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RangiLyu commented Jan 16, 2023 •

edited

Loading

CLAassistant commented Jan 16, 2023 •

edited

Loading