Unable to get custom op `MMCVModulatedDeformConv2d` to work for ONNX runtime in Nvidia Triton server

Hi, I tried to build MMDeploy the same way as https://github.com/open-mmlab/mmdeploy/issues/465 but for mmocr and targeting Onnx runtime. MMDeploy ran and generated an .onnx file. Similarly to issue #465, i’ve copied the libmmdeploy_onnxruntime_ops.so file and set env LD_PRELOAD= /opt/tritonserver/lib/libmmdeploy_onnxruntime_ops.so before starting tritonserver, yet i’m still running into MMCVModulatedDeformConv2d is not a registered function/op error:

+---------+---------+------------------------------------------------------------------------------------------------------------------+
| Model   | Version | Status                                                                                                           |
+---------+---------+------------------------------------------------------------------------------------------------------------------+
| dbnetpp | 2       | UNAVAILABLE: Internal: onnx runtime error 1: Load model from /opt/app/model-repo/dbnetpp/2/model.onnx failed:Fat |
|         |         | al error: MMCVModulatedDeformConv2d is not a registered function/op                                              |
+---------+---------+------------------------------------------------------------------------------------------------------------------+

I0711 22:06:20.512354 55 tritonserver.cc:1920] 
+----------------------------------+----------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                              |
+----------------------------------+----------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                             |
| server_version                   | 2.15.0                                                                                             |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model |
|                                  | _configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics               |
| model_repository_path[0]         | /opt/app/model-repo                                                                                |
| model_control_mode               | MODE_NONE                                                                                          |
| strict_model_config              | 0                                                                                                  |
| rate_limit                       | OFF                                                                                                |
| pinned_memory_pool_byte_size     | 268435456                                                                                          |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                           |
| response_cache_byte_size         | 0                                                                                                  |
| min_supported_compute_capability | 6.0                                                                                                |
| strict_readiness                 | 1                                                                                                  |
| exit_timeout                     | 30                                                                                                 |
+----------------------------------+----------------------------------------------------------------------------------------------------+

and ldd /opt/tritonserver/bin/tritonserver confirmed that the .so is loaded:

        linux-vdso.so.1 (0x00007fff11ff9000)
        /opt/tritonserver/lib/libmmdeploy_onnxruntime_ops.so (0x00007f92d5c2e000)
        libre2.so.5 => /usr/lib/x86_64-linux-gnu/libre2.so.5 (0x00007f92d5bb3000)
        libb64.so.0d => /usr/lib/x86_64-linux-gnu/libb64.so.0d (0x00007f92d59b0000)
        libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x00007f92d5994000)
        libtritonserver.so => /opt/tritonserver/bin/../lib/libtritonserver.so (0x00007f92d4d00000)
        librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007f92d4cf3000)
        libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d4ced000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d4cca000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f92d4c37000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f92d4961000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f92d477f000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d462e000)
        libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f92d4613000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d4421000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f92d638a000)
        libonnxruntime.so.1.9.0 => /opt/tritonserver/backends/onnxruntime/libonnxruntime.so.1.9.0 (0x00007f92d370b000)
        <shorten for brevity>

Am I missing anything obvious? Thanks for any guidance in advance!

Dockerfile to build MMDeploy is below:

FROM nvcr.io/nvidia/tensorrt:21.10-py3

ARG CUDA=10.2
ARG PYTHON_VERSION=3.8
ARG TORCH_VERSION=1.8.0
ARG TORCHVISION_VERSION=0.9.0
ARG ONNXRUNTIME_VERSION=1.9.0
ARG MMCV_VERSION=1.5.3
ARG PPLCV_VERSION=0.6.2
ENV FORCE_CUDA="1"

#With and without this env defined, I got the same error.
ENV MMCV_WITH_OPS="1"

ENV DEBIAN_FRONTEND=noninteractive

### change the system source for installing libs
ARG USE_SRC_INSIDE=false
RUN if [ ${USE_SRC_INSIDE} == true ] ; \
    then \
        sed -i s/archive.ubuntu.com/mirrors.aliyun.com/g /etc/apt/sources.list ; \
        sed -i s/security.ubuntu.com/mirrors.aliyun.com/g /etc/apt/sources.list ; \
        echo "Use aliyun source for installing libs" ; \
    else \
        echo "Keep the download source unchanged" ; \
    fi

### update apt and install libs
RUN apt-get update &&\
    apt-get install -y vim libsm6 libxext6 libxrender-dev libgl1-mesa-glx git wget libssl-dev libopencv-dev libspdlog-dev --no-install-recommends &&\
    rm -rf /var/lib/apt/lists/*

RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
    chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b -p /opt/conda && \
    rm ~/miniconda.sh && \
    /opt/conda/bin/conda install -y python=${PYTHON_VERSION} conda-build pyyaml numpy ipython cython typing typing_extensions mkl mkl-include ninja && \
    /opt/conda/bin/conda clean -ya

### pytorch
RUN /opt/conda/bin/conda install pytorch==${TORCH_VERSION} torchvision==${TORCHVISION_VERSION} cudatoolkit=${CUDA} -c pytorch
ENV PATH /opt/conda/bin:$PATH

### install mmcv-full
RUN /opt/conda/bin/pip install mmcv-full==${MMCV_VERSION} -f https://download.openmmlab.com/mmcv/dist/cu${CUDA//./}/torch${TORCH_VERSION}/index.html

WORKDIR /root/workspace
### get onnxruntime
RUN wget https://github.com/microsoft/onnxruntime/releases/download/v${ONNXRUNTIME_VERSION}/onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}.tgz \
    && tar -zxvf onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}.tgz &&\
    pip install onnxruntime-gpu==${ONNXRUNTIME_VERSION}
    
## get torchscript
RUN wget https://download.pytorch.org/libtorch/cu111/libtorch-shared-with-deps-1.8.1%2Bcu111.zip \
    && unzip libtorch-shared-with-deps-1.8.1+cu111.zip
    
ENV Torch_DIR=/root/workspace/libtorch
ENV LD_LIBRARY_PATH=$Torch_DIR/lib:$LD_LIBRARY_PATH

### cp trt from pip to conda
RUN cp -r /usr/local/lib/python${PYTHON_VERSION}/dist-packages/tensorrt* /opt/conda/lib/python${PYTHON_VERSION}/site-packages/

### install mmdeploy
ENV ONNXRUNTIME_DIR=/root/workspace/onnxruntime-linux-x64-${ONNXRUNTIME_VERSION}
ENV TENSORRT_DIR=/workspace/tensorrt
ARG VERSION
RUN git clone https://github.com/open-mmlab/mmdeploy &&\
    cd mmdeploy &&\
    if [ -z ${VERSION} ] ; then echo "No MMDeploy version passed in, building on master" ; else git checkout tags/v${VERSION} -b tag_v${VERSION} ; fi &&\
    git submodule update --init --recursive &&\
    mkdir -p build &&\
    cd build &&\
    cmake -DMMDEPLOY_TARGET_BACKENDS="ort;trt;torchscript" .. &&\
    make -j$(nproc) &&\
    cd .. &&\
    pip install -e .
    
# install mmocr && mmdetection
RUN cd / && \
    git clone -b v0.6.0 https://github.com/open-mmlab/mmocr.git && \
    cd mmocr && \
    pip install -r requirements/build.txt && \
    pip install -v -e .
    
RUN cd / && \
    git clone -b v2.25.0 https://github.com/open-mmlab/mmdetection.git && \
    cd mmdetection && \
    pip install -r requirements/build.txt && \
    pip install -v -e .

To build the image mmdeploy:mmocr-0.6.0-onnx_1.9.0_trt-21.10_torch:

docker build --build-arg VERSION=0.6.0 --tag=mmdeploy:mmocr-0.6.0-onnx_1.9.0_trt-21.10_torch .

To export the dbnetpp model:

docker run -it -v /home/jupyter:/volume_share --runtime=nvidia mmdeploy:mmocr-0.6.0-onnx_1.9.0_trt-21.10_torch \
        python /root/workspace/mmdeploy/tools/deploy.py \
        /root/workspace/mmdeploy/configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py \
        /volume_share/mmocr/configs/textdet/dbnetpp/dbnetpp_r50dcnv2_fpnc_1200e_icdar2015.py \
        /volume_share/dbnetpp_r50dcnv2_fpnc_1200e_icdar2015-20220502-d7a76fff.pth \
        /volume_share/test_images/test.png \
        --work-dir /volume_share \
        --show \
        --device cuda:0