❓ [Question] dynamc engines & interpolation align_corners=True

[ArtemisZGL]

❓ Question

What you have already tried

I used the latest docker with tag 23.08-py3. When converting model doing interpolation with align_corners=True and dynamic input, I got error as below.

RuntimeError: [Error thrown at core/conversion/converters/impl/interpolate.cpp:412] Expected !(align_corners && ctx->input_is_dynamic) to be true but got false                                                                                               
Torch-TensorRT currently does not support the compilation of dynamc engines from code using PyTorch [bi/tri]linear interpolation via scale factor and align_corners=True 

And I found this check did exist in code with tag v1.4.0, but not in main branch. Will I need to clone the latest code and recompile torch-tensorrt to escape frome this error and will it work? Or any other simple way ?

Environment

nvcr.io/nvidia/pytorch:23.08-py3

Additional context

[ArtemisZGL]

BTW, there is no setup.py in py folder in main branch.

[gs-olive]

Hello - #2146 removed this warning, so it seems the latest main may resolve this issue. The setup.py file has moved one directory up from py/, so it can now be invoked in the main directory of the repository, either via python setup.py install/develop, or with a pip installation.

[ArtemisZGL]

@gs-olive I uninstall origin torch_tensorrt in docker, and then run python setup.py install in main branch. After that, I got this error when importing torch_tensorrt.

ImportError: /usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib/libtorchtrt.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs

Could you give some help? BTW, I tried python setup.py install/develop but got error.

invalid command name 'install/develop'

[gs-olive]

This is likely a Torch version mismatch error. Could you instead try, from the root of the directory:

pip install . --extra-index-url https://download.pytorch.org/whl/nightly/cu121 --extra-index-url https://pypi.ngc.nvidia.com

Alternatively, you can get a Docker container with the latest main branch here:

docker pull ghcr.io/pytorch/tensorrt/torch_tensorrt:main

[ArtemisZGL]

@gs-olive Thanks for your help, I tried your docker image, but still get error when importing torch_tensorrt 😂.

/opt/python3/site-packages/torch_tensorrt/lib/libtorchtrt.so: undefined symbol: _ZNR5torch7Library4_defEON3c106eitherINS1_12OperatorNameENS1_14FunctionSchemaEEEONS_11CppFunctionE

[gs-olive]

Thanks for letting me know - it seems there is a mismatch in the libtorch libraries being installed through Torch vs Torch-TensorRT on recent nightly versions and on the Docker container. Looking into it.

[ArtemisZGL]

@gs-olive Looking forward to the solution. 🚀

[gs-olive]

I just checked and the latest deployment of ghcr.io/pytorch/tensorrt/torch_tensorrt:main is working on my machine, as the import error has been resolved. Could you retry with the new container?

[ArtemisZGL]

@gs-olive Now getting the cuda error when using gpu.😂

WARNING: [Torch-TensorRT] - Unable to read CUDA capable devices. Return status: 804                                                                                                                                                                           
Traceback (most recent call last):
  File "/code/algorithm/vits-svc-new/convert_torchscript_trt.py", line 1, in <module>                                                                                                                                                                         
    import torch_tensorrt                                                                                                                                                                                                                                     
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/__init__.py", line 84, in <module>                                                                                                                                          
    from torch_tensorrt._compile import *  # noqa: F403                                                                                                                                                                                                       
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/_compile.py", line 9, in <module>                                                                                                                                           
    import torch_tensorrt.ts                                                                                                                                                                                                                                  
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/ts/__init__.py", line 1, in <module>                                                                                                                                        
    from torch_tensorrt.ts._compile_spec import TensorRTCompileSpec  # noqa: F401                                                                                                                                                                             
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/ts/_compile_spec.py", line 348, in <module>                                                                                                                                 
    device: torch.device | Device = Device._current_device(),                                                                                                                                                                                                 
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/_Device.py", line 155, in _current_device                                                                                                                                   
    dev = _C._get_current_device()                                                                                                                                                                                                                            
RuntimeError: [Error thrown at core/runtime/runtime.cpp:97] Expected (cudaGetDevice(reinterpret_cast<int*>(&device)) == cudaSuccess) to be true but got false 

/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)

[gs-olive]

Hi @ArtemisZGL - I have seen this issue before - when running the docker container, do you specify --gpus all? As in:

docker run --rm -it --gpus all ghcr.io/pytorch/tensorrt/torch_tensorrt:main

This might help with exposing the GPUs through the docker container. If you prefer to run the latest versions of Torch-TRT outside of a Docker container, you can install the nightly through pip, like so (switch cu121 for cu118, if on a different CUDA version):

pip install --pre torch torch_tensorrt --extra-index-url https://download.pytorch.org/whl/nightly/cu121

[ArtemisZGL]

@gs-olive Sorry for late reply cause having a long vocation. I used --gpus all to start the container as below.

docker run -it --gpus all --network host ghcr.io/pytorch/tensorrt/torch_tensorrt:main

[ArtemisZGL]

@gs-olive I think that problem is related to driver version, and I reinstall by your command with cu113.

pip install --pre torch torch_tensorrt --extra-index-url https://download.pytorch.org/whl/nightly/cu113

But I met another problem with dynamic shape, here is the code I used to convert the model which should work for dynamic shape accordings to docs.

with torch_tensorrt.logging.debug():
        trt_ts_module = torch_tensorrt.compile(script_module,
            inputs = [ 
                torch_tensorrt.Input(
                min_shape=[1, 192, 1],
                opt_shape=[1, 192, 512],
                max_shape=[1, 192, 1024],
                dtype=torch.float),
                torch_tensorrt.Input(
                min_shape=[1, 1],
                opt_shape=[1, 512],
                max_shape=[1, 1024],
                dtype=torch.float),
                torch_tensorrt.Input(
                shape=[1, 768, 1],
                dtype=torch.float)
            ],
            enabled_precisions = {torch.half},
            # allow_shape_tensors = True,
            truncate_long_and_double = True,) # Run with FP16)

The model have 3 inputs, while the 1st and 2nd are dynamic but the 3rd is static. After conversion, It can infer the shape with 512 properly, but other shape get shape errors. I also tried to set the 3rd input with min(opt max) shape but still got the same error. Is that I missing somethings?
log_convert.txt

ERROR: [Torch-TensorRT] - 3: [executionContext.cpp::setInputShape::2020] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2020, condition: engineDims.d[i] == dims.d[i]. Static dimension mismatch while setting input shape.
)
ERROR: [Torch-TensorRT] - 3: [executionContext.cpp::setInputShape::2020] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2020, condition: engineDims.d[i] == dims.d[i]. Static dimension mismatch while setting input shape.
)