tensorrt_usage

官方文档

onnx2trt

trtexec

export LD_LIBRARY_PATH=/usr/local/TensorRT-8.6.1.6/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/docker/tmp/TensorRT/out:/usr/local/TensorRT-8.6.1.6/lib:/usr/local/lib:/usr/local/lib:$LD_LIBRARY_PATH
./trtexec --onnx=mygelu.onnx  --saveEngine=mygelu.trt --explicitBatch --buildOnly

运行过程中发现输入的 onnx 模型必须和trtexec 在同一目录下。

如果onnx 中有自定义算子，或者是onnx 中的算子没有和 trt 中的算子建立映射关系，则会提示算子找不到的错误:

[10/24/2024-11:02:35] [I] [TRT] No importer registered for op: MyGelu. Attempting to import as plugin.
[10/24/2024-11:02:35] [I] [TRT] Searching for plugin: MyGelu, plugin_version: 1, plugin_namespace: 
[10/24/2024-11:02:35] [E] [TRT] 3: getPluginCreator could not find plugin: MyGelu version: 1
[10/24/2024-11:02:35] [E] [TRT] ModelImporter.cpp:771: While parsing node number 0 [MyGelu -> "/MyGelu_output_0"]:
[10/24/2024-11:02:35] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[10/24/2024-11:02:35] [E] [TRT] ModelImporter.cpp:773: input: "add_num"
output: "/MyGelu_output_0"
name: "/MyGelu"
op_type: "MyGelu"
attribute {
  name: "add_num"
  f: 0.2
  type: FLOAT
}
domain: "TRT"

[10/24/2024-11:02:35] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[10/24/2024-11:02:35] [E] [TRT] ModelImporter.cpp:777: ERROR: builtin_op_importers.cpp:5404 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[10/24/2024-11:02:35] [E] Failed to parse onnx file

如果是新的算子库 libnvinfer_plugin.so 已经编译出来了，需要把路径添加到LD_LIBRARY_PATH 中；如果还报算子找不到的错误，可以用LD_PRELOAD 加载一下 libnvinfer_plugin.so

trt api 推理带有自定义算子的模型

按照正常的模型的方式进行调用就可以，只是在执行的时候需要加载算子库。

如果没有加载自定义算子库，在模型反序列化的时候会找不到算子 psd-3.1.2.0

[10/24/2024-11:29:21] [E] [TRT] 3: getPluginCreator could not find plugin: CustomGeluPluginDynamic version: 1
[10/24/2024-11:29:21] [E] [TRT] 1: [pluginV2Runner.cpp::load::303] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)

如果没有显式调用算子库，用LD_PRELOAD 加载一下 libnvinfer_plugin.so

tensorrt 自定义算子

模型推理

加载模型文件 -> 创建runtime -> 通过runtime 创建 engine -> 通过engine 反序列化模型 -> 通过engine 创建 context -> 获取输入、输出的 binding 关系