### 部署ONNX-TensorRT模型于NVIDIA Orin平台 #### 准备环境配置 确保开发环境中安装了适用于NVIDIA Orin的软件包版本，包括但不限于JetPack SDK、CUDA Toolkit以及cuDNN库。对于Orin设备而言，推荐使用最新稳定版的JetPack来获取最佳性能和支持[^1]。 #### 安装TensorRT 为了能够在NVIDIA Orin上运行由ONNX转换而来的TensorRT模型，需预先安装好兼容当前硬件架构的TensorRT版本。通常情况下，随着JetPack一起发布的TensorRT已经过优化适配，可以直接利用该预编译二进制文件完成安装过程[^2]。 #### 导出并验证ONNX模型 在准备阶段结束后，应先将训练完毕后的深度神经网络保存成通用中间表示形式即ONNX格式。这一步骤至关重要，因为只有正确无误地完成了模型结构及其参数向ONNX格式迁移之后才能顺利进入后续环节。可以借助PyTorch自带工具链实现这一目标[^3]： ```python import torch.onnx as onnx_utils dummy_input = ... # 构造输入张量 model.eval() # 切换至评估模式 with torch.no_grad(): onnx_path = "model.onnx" onnx_utils.export(model, dummy_input, onnx_path, opset_version=12, do_constant_folding=True, input_names=["input"], output_names=["output"]) print(f"Model has been successfully exported to {onnx_path}") ``` #### 创建TensorRT引擎 一旦拥有了有效的ONNX描述文档，则可通过调用TensorRT API中的`trt.Builder`类实例化对象，并传入必要的构建选项以生成最终用于加速推理计算的任务执行单元——TensorRT Engine。下面给出一段简单的Python代码片段作为示范: ```python import tensorrt as trt TRT_LOGGER = trt.Logger(trt.Logger.WARNING) def build_engine(onnx_file_path): with trt.Builder(TRT_LOGGER) as builder,\ builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)) as network,\ trt.OnnxParser(network, TRT_LOGGER) as parser: config = builder.create_builder_config() config.max_workspace_size = 1 << 30 # 设置最大工作空间大小为1GB with open(onnx_file_path, 'rb') as model: if not parser.parse(model.read()): for error in range(parser.num_errors): print(parser.get_error(error)) return None engine = builder.build_serialized_network(network, config) return engine engine = build_engine("model.onnx") if isinstance(engine, bytes): with open('model.engine', 'wb') as f: f.write(engine) else: raise RuntimeError("Failed to create TensorRT engine.") ``` #### 加载与推断 最后，在实际应用场合下只需加载事先创建好的`.engine`文件即可快速启动高性能预测流程。此处同样附带了一个简易的例子说明如何操作: ```python runtime = trt.Runtime(TRT_LOGGER) with open('model.engine', 'rb') as f: engine_data = f.read() engine = runtime.deserialize_cuda_engine(engine_data) context = engine.create_execution_context() inputs, outputs, bindings, stream = [], [], [], [] for binding in engine: size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size dtype = trt.nptype(engine.get_binding_dtype(binding)) host_mem = cuda.pagelocked_empty(size, dtype) device_mem = cuda.mem_alloc(host_mem.nbytes) bindings.append(int(device_mem)) if engine.binding_is_input(binding): inputs.append({'host': host_mem, 'device': device_mem}) else: outputs.append({'host': host_mem, 'device': device_mem}) stream = cuda.Stream() # 假设已经有了待处理的数据data... np.copyto(inputs[0]['host'], data.ravel()) cuda.memcpy_htod_async(inputs[0]['device'], inputs[0]['host'], stream) context.execute_async_v2(bindings=bindings, stream_handle=stream.handle) cuda.memcpy_dtoh_async(outputs[0]['host'], outputs[0]['device'], stream) stream.synchronize() result = np.array(outputs[0]['host']).reshape(-1, ...) ```