Unable to run evaluation on ONNX model due to unregistered `MMCVDeformConv2d`

My goal is to convert a Torch model trained via mmdetection to ONNX. I first followed the guide to build toolchains, dependencies, and MMDeploy in my docker environment. Following this, I’ve successfully convert from a Torch model to an ONNX model through deploy/tools/test.py. Then when I ran evaluation by following this page, it throws an error that the operation MMCVDeformConv2d is not registered.

According to here, my understanding is that MMCVModulatedDeformConv2d is supported as a custom ONNX operation in MMDeploy. So I want to post the issue here to see if there’s any gap in my implementation with the instruction.

Here’s the script I ran to convert torch2onnx:

MMDEPLOY_DIR="/workspace/mmdeploy"
DEPLOY_CFG=${MMDEPLOY_DIR}/configs/mmdet/detection/detection_onnxruntime_dynamic.py
MODEL_CFG=/workspace/experiments/deploy_test_0505/resnet50_reppoints_my_dataset.py
MODEL_FILES=/workspace/experiments/deploy_test_0505/best_instance_mAP_epoch_18.pth
DEVICE=cuda:0
python ${MMDEPLOY_DIR}/tools/deploy.py \
    ${DEPLOY_CFG} \
    ${MODEL_CFG} \
    ${MODEL_FILES} \
    PATH_TO_TEST_IMAGE.png \
    --work-dir ./ \
    --device {$DEVICE}} \
    --dump-info

Here’s the script I ran to evaluate the onnx model:

MMDEPLOY_DIR="/workspace/mmdeploy"
DEPLOY_CFG=${MMDEPLOY_DIR}/configs/mmdet/detection/detection_onnxruntime_dynamic.py
MODEL_CFG=/workspace/experiments/deploy_test_0505/resnet50_reppoints_my_dataset.py
MODEL_FILES=/workspace/experiments/deploy_test_0505/best_instance_mAP_epoch_18.pth
BACKEND_MODEL_FILES=/workspace/experiments/end2end.onnx
METRICS=mAP
DATA_DIR=/workspace/experiments/
OUTPUT_PKL_FILE=$DATA_DIR/output.pkl
OUTPUT_IMAGE_DIR=$DATA_DIR/output_images/
DEVICE=cuda:0

python ${MMDEPLOY_DIR}/tools/test.py \
    ${DEPLOY_CFG} \
    ${MODEL_CFG} \
    --model ${BACKEND_MODEL_FILES} \
    --metrics ${METRICS} \
    --out ${OUTPUT_PKL_FILE} \
    --show-dir ${OUTPUT_IMAGE_DIR} \
    --device ${DEVICE}

Here’s my environment

(llens) ➜  mmdeploy git:(master) ✗ python tools/check_env.py
2022-05-06 02:47:16,814 - mmdeploy - INFO - 

2022-05-06 02:47:16,814 - mmdeploy - INFO - **********Environmental information**********
2022-05-06 02:47:17,083 - mmdeploy - INFO - sys.platform: linux
2022-05-06 02:47:17,083 - mmdeploy - INFO - Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
2022-05-06 02:47:17,084 - mmdeploy - INFO - CUDA available: True
2022-05-06 02:47:17,084 - mmdeploy - INFO - GPU 0: Tesla V100-SXM2-16GB
2022-05-06 02:47:17,084 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-05-06 02:47:17,084 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.2, V11.2.152
2022-05-06 02:47:17,084 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-05-06 02:47:17,084 - mmdeploy - INFO - PyTorch: 1.10.2+cu113
2022-05-06 02:47:17,084 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

2022-05-06 02:47:17,084 - mmdeploy - INFO - TorchVision: 0.11.3+cu113
2022-05-06 02:47:17,084 - mmdeploy - INFO - OpenCV: 4.5.5
2022-05-06 02:47:17,084 - mmdeploy - INFO - MMCV: 1.5.0
2022-05-06 02:47:17,084 - mmdeploy - INFO - MMCV Compiler: GCC 7.3
2022-05-06 02:47:17,084 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-05-06 02:47:17,084 - mmdeploy - INFO - MMDeploy: 0.4.0+16ee9c7
2022-05-06 02:47:17,084 - mmdeploy - INFO - 

2022-05-06 02:47:17,084 - mmdeploy - INFO - **********Backend information**********
[2022-05-06 02:47:17.234] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
2022-05-06 02:47:17,241 - mmdeploy - INFO - onnxruntime: 1.8.1  ops_is_avaliable : True
2022-05-06 02:47:17,242 - mmdeploy - INFO - tensorrt: None      ops_is_avaliable : False
2022-05-06 02:47:17,243 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-05-06 02:47:17,244 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-05-06 02:47:17,245 - mmdeploy - INFO - openvino_is_avaliable: False
2022-05-06 02:47:17,245 - mmdeploy - INFO - 

2022-05-06 02:47:17,245 - mmdeploy - INFO - **********Codebase information**********
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmdet:      2.23.0
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmseg:      None
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmcls:      0.22.0
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmocr:      None
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmedit:     None
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmdet3d:    None
2022-05-06 02:47:17,247 - mmdeploy - INFO - mmpose:     None

Log of converting to ONNX:

(llens) ➜  experiments sh execute.sh
[2022-05-05 21:18:00.119] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
/workspace/model/mmdetection/mmdet/datasets/utils.py:70: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  'data pipeline in your config file.', UserWarning)
2022-05-05 21:18:01,365 - mmdeploy - INFO - torch2onnx start.
[2022-05-05 21:18:03.182] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
load checkpoint from local path: /workspace/experiments/deploy_test_0505/best_instance_mAP_epoch_18.pth
/workspace/model/mmdetection/mmdet/datasets/utils.py:70: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  'data pipeline in your config file.', UserWarning)
2022-05-05 21:18:05,391 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
2022-05-05:21:18:05,mmdeploy WARNING  [utils.py:92] DeprecationWarning: get_onnx_config will be deprecated in the future. 
/workspace/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/opt/conda/envs/llens/lib/python3.7/site-packages/torch/nn/functional.py:2359: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:299: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_pad = (x.size(2) < self.kernel_size[0]) or (x.size(3) <
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:301: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_pad:
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:181: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  output_size += ((in_size + (2 * pad) - kernel) // stride_ + 1, )
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:182: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not all(map(lambda s: s > 0, output_size)):
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  cur_im2col_step = min(ctx.im2col_step, input.size(0))
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:91: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ) == 0, 'batch size must be divisible by im2col_step'
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/deform_conv.py:310: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_pad:
/opt/conda/envs/llens/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/dense_heads/base_dense_head.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
/workspace/mmdeploy/mmdeploy/pytorch/functions/topk.py:28: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  k = torch.tensor(k, device=input.device, dtype=torch.long)
/workspace/mmdeploy/mmdeploy/pytorch/functions/topk.py:33: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  return ctx.origin_func(input, k, dim=dim, largest=largest, sorted=sorted)
/workspace/model/mmdetection/mmdet/core/bbox/coder/distance_point_bbox_coder.py:58: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert points.size(0) == pred_bboxes.size(0)
/workspace/model/mmdetection/mmdet/core/bbox/coder/distance_point_bbox_coder.py:59: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert points.size(-1) == 2
/workspace/model/mmdetection/mmdet/core/bbox/coder/distance_point_bbox_coder.py:60: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(-1) == 4
/workspace/mmdeploy/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:92: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  iou_threshold = torch.tensor([iou_threshold], dtype=torch.float32)
/workspace/mmdeploy/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:93: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  score_threshold = torch.tensor([score_threshold], dtype=torch.float32)
/workspace/mmdeploy/mmdeploy/mmcv/ops/nms.py:40: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  score_threshold = float(score_threshold)
/workspace/mmdeploy/mmdeploy/mmcv/ops/nms.py:41: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  iou_threshold = float(iou_threshold)
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/nms.py:160: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(1) == 4
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/nms.py:161: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(0) == scores.size(0)
/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/ops/nms.py:29: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if max_num > 0:
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
/opt/conda/envs/llens/lib/python3.7/site-packages/torch/onnx/symbolic_opset9.py:2819: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  "If indices include negative values, the exported graph will produce incorrect results.")
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVDeformConv2d type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
2022-05-05 21:18:29,953 - mmdeploy - INFO - torch2onnx success.
2022-05-05 21:18:29,954 - mmdeploy - WARNING - "visualize_model" has been skipped may be because it's             running on a headless device.
2022-05-05 21:18:29,954 - mmdeploy - INFO - All process success.

Log of evaluating the ONNX model:

[2022-05-05 22:29:19.341] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel'
2022-05-05 22:29:20,600 - mmdeploy - WARNING - The library of onnxruntime custom ops does             not exist: 
Traceback (most recent call last):
  File "/workspace/mmdeploy/tools/test.py", line 137, in <module>
    main()
  File "/workspace/mmdeploy/tools/test.py", line 106, in main
    model = task_processor.init_backend_model(args.model)
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/object_detection.py", line 74, in init_backend_model
    model_files, self.model_cfg, self.deploy_cfg, device=self.device)
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 746, in build_object_detection_model
    **kwargs)
  File "/opt/conda/envs/llens/lib/python3.7/site-packages/mmcv/utils/registry.py", line 215, in build
    return self.build_func(*args, **kwargs, registry=self)
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 34, in __build_backend_model
    **kwargs)
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 65, in __init__
    backend=backend, backend_files=backend_files, device=device)
  File "/workspace/mmdeploy/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 84, in _init_wrapper
    deploy_cfg=self.deploy_cfg)
  File "/workspace/mmdeploy/mmdeploy/codebase/base/backend_model.py", line 62, in _build_wrapper
    output_names=output_names)
  File "/workspace/mmdeploy/mmdeploy/backend/onnxruntime/wrapper.py", line 57, in __init__
    onnx_file, session_options, providers=providers)
  File "/opt/conda/envs/llens/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/opt/conda/envs/llens/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 310, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /workspace/experiments/end2end.onnx failed:Fatal error: MMCVDeformConv2d is not a registered function/op