TensorRT 7.1.3 / Cuda11 / CuDNN 8 - supported yet?

[kzuiderveld]

Hi,

I pulled the latest Onnx runtime version and compiled with the latest NVIDIA libraries (see title). The tests passed except for the Python test (which is expected).

Unfortunately, the TensorRT path gives significantly different results compared to the CPU path (I can tell differences by visual inspection of the activation output).

Are there currently known compatibility issues with the latest TensorRT/CuDNN/Cuda libraries? If so, is there an ETA for a fix? If not, is there somebody I can work with to troubleshoot this further?

Karel

[snnn]

Could you please provide more information? We plan to release ORT 1.5 with TensorRT 7.1, it should work.

[kzuiderveld]

Onnx runtime version: a3c953
TensorRT 7.1.3, CuDNN 8.0, Cuda 11.0
Using simple U-net model (can share under NDA if needed, but I imagine other might have similar issues).

Attached one of the output channels. Visual inspection shows they're similar, but quite different in places yielding different results. Only change in the code is switching from default to TensorRT provider, so differences originate inside Onnx Runtime or TensorRT. TensorRT fp16 and fp32 gave virtually identical results (different from CPU version).

Any suggestions on how to troubleshoot this most effectively?

[snnn]

@stevenlix Do you know why?

[jywu-mysoft]

we're unaware of a general systemic issue with ORT + TensorRT EP.
if it's possible, sharing a minimal repro (rather than full model) would be very helpful to debug the issue further from our end.

[kzuiderveld]

Hello,

I re-ported TensorRT 7.1.3.4 - I don't believe it is working correctly. Lots of errors like
F:\onnxruntime\onnxruntime\test\providers\provider_test_utils.cc(155): error: The difference between expected[i] and output[i] is 0.41475188732147217, which exceeds threshold

that suggest that tests are failing. Comments?

I uploaded the output of onnxruntime_test_all.exe here: https://1drv.ms/u/s!AtdjyUUyTjd8_RzBG2s9B4RMYlt9?e=S7EVyA

Here are my installation notes:

1. Downloaded and installed Quadro NVIDIA driver 452.06 (388,202 KB) - my system has a Quadro RTX4000.

2. Downloaded and installed cuda_11.0.3_451.82_win10.
   Installed:

   • Nsight for Visual Studio 2019
   • Nsight for Visual Studio 2017
   • Nsight Monitor
     Not Installed:
   • Nsight for Visual Studio 2015
     Reason: VS2015 was not found
   • Integrated Graphics Frame Debugger and Profiler
     Reason: see https://developer.nvidia.com/nsight-vstools
   • Integrated CUDA Profilers
     Reason: see https://developer.nvidia.com/nsight-vstools

3. Built the CUDA samples. 174 succeeded, 2 failed (MPI and Vulkan not installed on my system)
   Tried various examples, all ran successfully ("Turing" with compute capability 7.5)

4. Downloaded cudnn-11.0-windows-x64-v8.0.2.39. Unzipped into f:\cudnn8. Header file identifies version 8.0.2

5. Downloaded TensorRT-7.1.3.4.Windows10.x86_64.cuda-11.0.cudnn8.0. Unzipped into f:\TenosrRT-7.1.3.4.

6. Performed a "git pull" on OnnxRunTime 10:02 MDT 8/31/2020. Last checkin Optimize MatmulGradient when B is 2D weight #4977, 98f7fdd.

7. Attempted to build OnnxRunTime with TensorRT support:
   .\build.bat --use_tensorrt --tensorrt_home "f:\TensorRT-7.1.3.4" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0" --cudnn_home "f:\cudnn-8.0.2\cuda" --parallel --build_shared_lib --config RelWithDebInfo --cmake_generator "Visual Studio 16 2019"

   Fails: 2020-08-31 10:08:52,352 Build [ERROR] - No version file found in CUDA install directory. Looked for C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\version.txt

   -> That's an Onnx Runtime issue, cmake supports needs to be modified for 11.0

8. Manually created c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\V11.0\version.txt with the string
   CUDA Version 11.0.194

9. Compilation now proceeds. However, the tests give various error messages, including many like
   1: F:\onnxruntime\onnxruntime\test\providers\provider_test_utils.cc(155): error: The difference between expected[i] and output[i] is 0.41475188732147217, which exceeds threshold, where
   1: expected[i] evaluates to 0.63340294361114502,
   1: output[i] evaluates to 0.21865105628967285, and
   1: threshold evaluates to 0.004999999888241291.
   1: i:35730, provider_type: TensorrtExecutionProvider
   This tells me something is broken.

   Last screen after completion of compilation:
   6: [----------] 1 test from TestSessionOptions (0 ms total)
   6:
   6: [----------] Global test environment tear-down
   6: [==========] 1 test from 1 test suite ran. (0 ms total)
   6: [ PASSED ] 1 test.
   6/6 Test Add doxygen generated website for the project #6: onnxruntime_api_tests_without_env ...... Passed 0.11 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 1322.91 sec

The following tests FAILED:
1 - onnxruntime_test_all (Failed)
Errors while running CTest
Traceback (most recent call last):
File "F:\onnxruntime\tools\ci_build\build.py", line 1824, in
sys.exit(main())
File "F:\onnxruntime\tools\ci_build\build.py", line 1766, in main
run_onnxruntime_tests(args, source_dir, ctest_path, build_dir, configs)
File "F:\onnxruntime\tools\ci_build\build.py", line 1221, in run_onnxruntime_tests
run_subprocess(ctest_cmd, cwd=cwd, dll_path=dll_path)
File "F:\onnxruntime\tools\ci_build\build.py", line 428, in run_subprocess
env=my_env, shell=shell)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\Program Files\CMake\bin\ctest.EXE', '--build-config', 'RelWithDebInfo', '--verbose']' returned non-zero exit status 8.

F:\onnxruntime>

[jywu-mysoft]

Thanks for the details.
re: #7, @snnn , do we need to expand on #4706 , seems like the version.txt file may not exist on windows
re: the test failures, I glanced over your log output, and there are a lot of test failures. (203 tests)
Our internal CI test runs shows all the tests passing - currently I don't have a good explanation for why it's failing in your environment/hardware. we haven't tested on a Quadro RTX4000 on Windows but i wouldn't have anticipated issues with it.
if we can procure the hardware we will try to do more testing on different windows hardware to see if we can repro these issues.

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=211587&view=logs&j=72b59a5c-f9fa-5437-adfc-3f855724a45f&t=706f6d43-7b36-5d52-2781-608600615bec

1: [----------] Global test environment tear-down
1: [==========] 2186 tests from 187 test suites ran. (477588 ms total)
1: [ PASSED ] 2168 tests.
1: [ SKIPPED ] 18 tests, listed below:
1: [ SKIPPED ] InferenceSessionTests.TestLenientShapeInferencing
1: [ SKIPPED ] DequantizeLinearOpTest.Per_Channel_Axis_Default
1: [ SKIPPED ] DequantizeLinearOpTest.Per_Channel_Axis_0
1: [ SKIPPED ] DequantizeLinearOpTest.Per_Channel_Axis_1
1: [ SKIPPED ] DequantizeLinearOpTest.Per_Channel_Neg_2
1: [ SKIPPED ] MathOpTest.Sign_BFloat16
1: [ SKIPPED ] LogSoftmaxOperator.InvalidAxis
1: [ SKIPPED ] SoftmaxOperator.InvalidAxis
1: [ SKIPPED ] ConvTest.Conv1D_Invalid_Input_Shape
1: [ SKIPPED ] ConvTest.Conv2D_Invalid_Input_Shape
1: [ SKIPPED ] TfIdfVectorizerTest.Int32_TF_onlyBigrams_Skip0_Empty_Dim1Fail
1: [ SKIPPED ] TfIdfVectorizerTest.Int32_TF_onlyBigrams_Skip0_Empty_Dim2
1: [ SKIPPED ] TfIdfVectorizerTest.Int32_TF_onlyBigrams_Skip01_Empty_Dim2
1: [ SKIPPED ] QuantizeLinearOpTest.Per_Channel_Axis_Default
1: [ SKIPPED ] QuantizeLinearOpTest.Per_Channel_Axis_0
1: [ SKIPPED ] QuantizeLinearOpTest.Per_Channel_Axis_neg
1: [ SKIPPED ] TensorOpTest.Unsqueeze_Duplicate
1: [ SKIPPED ] TensorOpTest.Unsqueeze_OutOfRange

[snnn]

@jywu-msft Yes, I think we should.

[xkszltl]

FYI I also have colleague reported this kind of issue (i.e. mismatch way over fp error) with TensorRT provider.

[jywu-mysoft]

FYI I also have colleague reported this kind of issue (i.e. mismatch way over fp error) with TensorRT provider.

for a specific model? or is it the same case here, where the unit tests fail for certain environments?
we need help getting some repro to investigate both cases.

[jywu-mysoft]

can you re-test with the latest master?

[kzuiderveld]

@jywu-msft
TensorRT 7.1.3, CuDNN 8.0, Cuda 11.0 is still broken on Windows with the latest master.

[jywu-mysoft]

@jywu-msft
TensorRT 7.1.3, CuDNN 8.0, Cuda 11.0 is still broken on Windows with the latest master.

can you please be more specific.
it's running fine in our CI/test environments.
it's difficult to debug without more information/repro.