J5 RTX 4090 训练报错：RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

已解决

淼淼Mark2023-07-11

用户您好，请详细描述您所遇到的问题，这会帮助我们快速定位问题~

1.芯片型号：J5

2.天工开物开发包OpenExplorer版本：J5_oe_1.1.52a

3.问题定位：qat模型训练：calibration训练和qat训练

4.问题具体描述：

2023-07-11 11:12:58,659 INFO [logger.py:147] Node[0] ==================================================BEGIN QAT STAGE==================================================

2023-07-11 11:12:58,700 INFO [thread_init.py:38] Node[0] init torch_num_thread is `12`,opencv_num_thread is `12`,openblas_num_thread is `12`,mkl_num_thread is `12`,omp_num_thread is `12`,

2023-07-11 11:12:58,926 INFO [converters.py:56] Node[0] Successfully convert float model to qat model.

2023-07-11 11:12:58,927 WARNING [hash.py:218] Node[0] Don not found hash value in name of /open_explorer/work_dir/114/qat_train_hat_env/calibration_checkpoint/calibration-checkpoint-last.pth.tar, will skip check hash...

2023-07-11 11:12:58,956 WARNING [checkpoint.py:44] Node[0] module. is not at the beginning of state dict

2023-07-11 11:12:59,035 INFO [checkpoint.py:177] Node[0] state_dict in checkpoint num: 1076

2023-07-11 11:12:59,039 INFO [checkpoint.py:178] Node[0] state_dict in model num: 1076

2023-07-11 11:12:59,039 WARNING [checkpoint.py:179] Node[0] miss_key num: 0

2023-07-11 11:12:59,039 WARNING [checkpoint.py:182] Node[0] unexpect_key num: 0

2023-07-11 11:12:59,039 INFO [converters.py:248] Node[0] Load the checkpoint successfully from /open_explorer/work_dir/114/qat_train_hat_env/calibration_checkpoint/calibration-checkpoint-last.pth.tar

2023-07-11 11:13:03,578 INFO [loop_base.py:372] Node[0] Start DistributedDataParallelTrainer loop from epoch 0, num_epochs=10

2023-07-11 11:13:03,579 INFO [grad_scale.py:54] Node[0] [GradScale] []

2023-07-11 11:13:03,581 INFO [monitor.py:107] Node[0] Epoch[0] Begin ==================================================

2023-07-11 11:13:30,340 ERROR [ddp_trainer.py:363] Node[0] Traceback (most recent call last):

File "/usr/local/lib/python3.8/dist-packages/hat/engine/ddp_trainer.py", line 359, in _with_exception

fn(*args)

File "/open_explorer/project/ObjectDetection/tools/train.py", line 187, in train_entrance

trainer.fit()

File "/usr/local/lib/python3.8/dist-packages/hat/engine/loop_base.py", line 433, in fit

self.batch_processor(

File "/usr/local/lib/python3.8/dist-packages/hat/engine/processors/processor.py", line 442, in __call__

model_outs = model(*_as_list(batch_i))

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward

output = self.module(*inputs[0], **kwargs[0])

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/models/structures/detectors/centerpoint.py", line 99, in forward

input_features = self.reader(

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 222, in forward

features = self._extract_feature(features)

File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 237, in _extract_feature

features = pfn(features)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 81, in forward

x = self.linear(inputs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/nn/qat/conv2d.py", line 221, in forward

return self.activation_post_process(out)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/fake_quantize.py", line 207, in forward

self.activation_post_process(x.detach())

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap

return fn(self, *args, **kwargs)

File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl

return forward_call(*input, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/observer.py", line 322, in forward

(self.min_val, self.max_val,) = compute_moving_average(

RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

nvrtc compilation failed:

#define NAN __int_as_float(0x7fffffff)

#define POS_INFINITY __int_as_float(0x7f800000)

#define NEG_INFINITY __int_as_float(0xff800000)

template<typename T>

__device__ T maximum(T a, T b) {

return isnan(a) ? a : (a > b ? a : b);

}

template<typename T>

__device__ T minimum(T a, T b) {

return isnan(a) ? a : (a < b ? a : b);

}

extern "C" __global__

void fused_sub_mul_add_sub_mul_add(float* told_min_1, double vaveraging_constant_1, float* tcurrent_min_1, float* told_max_1, float* tcurrent_max_1, float* aten_add_1, float* aten_add) {

{

if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<64ll ? 1 : 0) {

float told_max_1_1 = __ldg(told_max_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));

float v = __ldg(tcurrent_max_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));

aten_add[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = told_max_1_1 + (v - told_max_1_1) * (float)(vaveraging_constant_1);

float told_min_1_1 = __ldg(told_min_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));

float v_1 = __ldg(tcurrent_min_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));

aten_add_1[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = told_min_1_1 + (v_1 - told_min_1_1) * (float)(vaveraging_constant_1);

}}

}

ERROR:__main__:launch trainer failed! process 0 terminated with exit code 1

Traceback (most recent call last):

File "/open_explorer/project/ObjectDetection/tools/train.py", line 278, in <module>

train(

File "/open_explorer/project/ObjectDetection/tools/train.py", line 273, in train

raise e

File "/open_explorer/project/ObjectDetection/tools/train.py", line 256, in train

launch(

File "/usr/local/lib/python3.8/dist-packages/hat/engine/ddp_trainer.py", line 328, in launch

mp.spawn(

File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn

return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes

while not context.join():

File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 139, in join

raise ProcessExitedException(

torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with exit code 1

算法工具链

0/1000

颜值即正义
Lv.2
您好，根据报错”RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)“来看可能是cuda和显卡的兼容问题，如果您使用OE包提供的docker环境进行开发，可以使用nvidia-smi命令来查看Cuda版本，并使用”pip list |grep torch“来查看是否安装了匹配版本的torch，如不匹配则需要前往官网 PyTorch 安装升级。
2023-07-11
0
12
- 淼淼Mark回复颜值即正义:
  您好：cuda 11.1 仍然报如下错误
  root@sh8:/open_explorer/project/ObjectDetection# pip list | grep torch
  horizon-plugin-pytorch 1.6.3+cu111.torch1102
  horizon-torch-samples 1.2.0+openexplore1.6.29
  torch 1.10.2+cu111
  torchmetrics 0.5.0
  torchvision 0.11.3+cu111
  root@sh8:/open_explorer/project/ObjectDetection# nvcc -V
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2020 NVIDIA Corporation
  Built on Mon_Oct_12_20:09:46_PDT_2020
  Cuda compilation tools, release 11.1, V11.1.105
  Build cuda_11.1.TC455_06.29190527_0
  loop_base.py:372] Node[0] Start DistributedDataParallelTrainer loop from epoch 0, num_epochs=10
  2023-07-11 17:42:26,980 INFO [grad_scale.py:54] Node[0] [GradScale] []
  2023-07-11 17:42:26,983 INFO [monitor.py:107] Node[0] Epoch[0] Begin ==================================================
  2023-07-11 17:42:54,196 ERROR [ddp_trainer.py:363] Node[0] Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/hat/engine/ddp_trainer.py", line 359, in _with_exception
  fn(*args)
  File "/open_explorer/project/ObjectDetection/tools/train.py", line 187, in train_entrance
  trainer.fit()
  File "/usr/local/lib/python3.8/dist-packages/hat/engine/loop_base.py", line 433, in fit
  self.batch_processor(
  File "/usr/local/lib/python3.8/dist-packages/hat/engine/processors/processor.py", line 442, in __call__
  model_outs = model(*_as_list(batch_i))
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
  output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/models/structures/detectors/centerpoint.py", line 99, in forward
  input_features = self.reader(
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 222, in forward
  features = self._extract_feature(features)
  File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 237, in _extract_feature
  features = pfn(features)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/models/task_modules/lidar/pillar_encoder.py", line 81, in forward
  x = self.linear(inputs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/nn/qat/conv2d.py", line 221, in forward
  return self.activation_post_process(out)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/fake_quantize.py", line 207, in forward
  self.activation_post_process(x.detach())
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/hat/utils/module_patch.py", line 46, in _wrap
  return fn(self, *args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
  return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/observer.py", line 322, in forward
  (self.min_val, self.max_val,) = compute_moving_average(
  RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)
  nvrtc compilation failed:
  #define NAN __int_as_float(0x7fffffff)
  #define POS_INFINITY __int_as_float(0x7f800000)
  #define NEG_INFINITY __int_as_float(0xff800000)
  template
  __device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a > b ? a : b);
  }
  template
  __device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a < b ? a : b);
  }
  extern "C" __global__
  void fused_sub_mul_add_sub_mul_add(float* told_min_1, double vaveraging_constant_1, float* tcurrent_min_1, float* told_max_1, float* tcurrent_max_1, float* aten_add_1, float* aten_add) {
  {
  if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)
  float told_max_1_1 = __ldg(told_max_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  float v = __ldg(tcurrent_max_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  aten_add[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = told_max_1_1 + (v - told_max_1_1) * (float)(vaveraging_constant_1);
  float told_min_1_1 = __ldg(told_min_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  float v_1 = __ldg(tcurrent_min_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  aten_add_1[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = told_min_1_1 + (v_1 - told_min_1_1) * (float)(vaveraging_constant_1);
  }}
  }
  ERROR:__main__:launch trainer failed! process 0 terminated with exit code 1
  Traceback (most recent call last):
  File "tools/train.py", line 278, in
  train(
  File "tools/train.py", line 273, in train
  raise e
  File "tools/train.py", line 256, in train
  launch(
  File "/usr/local/lib/python3.8/dist-packages/hat/engine/ddp_trainer.py", line 328, in launch
  mp.spawn(
  File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
  return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
  while not context.join():
  File "/root/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 139, in join
  raise ProcessExitedException(
  torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with exit code 1
  2023-07-11
  0
  回复
- 颜值即正义回复淼淼Mark:
  您好，除了calibration阶段和qat训练阶段，请问您在浮点模型训练阶段有遇到相同的报错吗？
  2023-07-11
  0
  回复
- 颜值即正义回复淼淼Mark:
  您好，您这边可以用单卡跑一下，看是否会报错，排除一下多进程的问题
  2023-07-11
  0
  回复
- 淼淼Mark回复颜值即正义:
  您好：
  1 float训练正常，没有问题。就是calibration和qat训练的问题。
  2 单卡也报同样的错误
  2023-07-12
  0
  回复
- 颜值即正义回复淼淼Mark:
  您好，这边麻烦您使用命令pip list |grep horizon，提供一下输出信息；此外如果方便的话，可以提供一下您使用的GPU驱动版本号和显卡的具体型号吗，我们这边分析一下~~
  2023-07-12
  0
  回复
- 颜值即正义回复颜值即正义:
  您可参考下图：
  2023-07-12
  0
  回复
- 颜值即正义回复淼淼Mark:
  您好，初步分析可能是模型中存在NaN或者inf这样的数值异常层，建议您先参考 4.2.4.5. 分析工具使用指南 — Horizon Open Explorer 进行debug~~
  2023-07-12
  0
  回复
- 淼淼Mark回复颜值即正义:
  您好：
  这是 grep horizon的结果
  root@sh8:/open_explorer/project# pip list |grep horizon
  horizon-nn 0.18.3
  horizon-perception-proto 0.0.1b202305260839+b8f70c0
  horizon-plugin-pytorch 1.6.3+cu111.torch1102
  horizon-tc-ui 1.17.5
  horizon-torch-samples 1.2.0+openexplore1.6.29
  显卡：
  GPU 00000000:B2:00.0
  Product Name : NVIDIA GeForce RTX 4090
  Product Brand : GeForce
  Product Architecture : Ada Lovelace
  Display Mode : Disabled
  Display Active : Disabled
  Persistence Mode : Disabled
  MIG Mode
  Current : N/A
  Pending : N/A
  Accounting Mode : Disabled
  Accounting Mode Buffer Size : 4000
  Driver Model
  Current : N/A
  Pending : N/A
  Serial Number : N/A
  GPU UUID : GPU-e590ba61-f1a1-c060-e4f8-d34974843fb8
  Minor Number : 7
  宿主机：
  驱动：
  NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 12.0 |
  |-------------------------------+----------------------+----------------------+
  | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
  | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
  | | | MIG M. |
  nvcc -V
  gj@sh8:~$ nvcc -V
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2022 NVIDIA Corporation
  Built on Wed_Sep_21_10:33:58_PDT_2022
  Cuda compilation tools, release 11.8, V11.8.89
  Build cuda_11.8.r11.8/compiler.31833905_0
  2023-07-12
  0
  回复
- 淼淼Mark回复颜值即正义:
  还有个奇怪的现象： calibration 每次都是到了80出错：
  2023-07-12 13:39:28,272 INFO [loop_base.py:431] Node[0] 20 / 5231
  2023-07-12 13:39:36,577 INFO [loop_base.py:431] Node[0] 40 / 5231
  2023-07-12 13:39:49,262 INFO [loop_base.py:431] Node[0] 60 / 5231
  2023-07-12 13:40:01,262 INFO [loop_base.py:431] Node[0] 80 / 5231
  /usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/observer.py:821: UserWarning: Compute amax failed
  warnings.warn("Compute amax failed")
  /usr/local/lib/python3.8/dist-packages/horizon_plugin_pytorch/quantization/observer.py:182: UserWarning: must run observer before calling calculate_qparams. Returning default scale and zero point
  warnings.warn(
  2023-07-12 13:40:09,291 ERROR [train.py:270] Node[0] launch trainer failed! The following operation failed in the TorchScript interpreter.
  Traceback of TorchScript (most recent call last):
  RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)
  nvrtc compilation failed:
  2023-07-12
  0
  回复
- 淼淼Mark回复颜值即正义:
  您好：
  这个代码在之前的3070和3090都是能正常训练的，只是换了新的4090才遇到这个问题
  2023-07-12
  0
  回复
- 淼淼Mark回复颜值即正义:
  您好，初步分析可能是模型中存在NaN或者inf这样的数值异常层，建议您先参考 4.2.4.5. 分析工具使用指南 — Horizon Open Explorer 进行debug~~
  这个我验证了：不是的，我只用的一个数据，然后用了您推荐的工具统计了最大最小没有出现异常的NaN或inf
  2023-07-12
  0
  回复
- 颜值即正义回复淼淼Mark:
  您好，4090不在我们明确支持的范围内，在使用上可能会存在cuda版本冲突或调用上的问题，建议您使用最新OE1.1.57版本的docker进行尝试，获取路径为地平线征程®️5 OpenExplorer算法工具链版本发布 (horizon.ai) 。如果使用最新版本的docker后问题仍无法解决，这边建议您更换显卡，使用3090等明确支持的显卡进行开发~~
  2023-07-12
  0
  回复
颜值即正义
Lv.2
您好，地平线工具链在持续迭代优化，为了给您提供更好的服务，希望您能抽出3分钟左右的时间，将您在使用工具链期间的感受和建议告诉我们，您的宝贵意见对我们很重要，非常感谢！
问卷链接：https://wenjuan.feishu.cn/m/?t=st64p6krU3Ji-yvhv
2023-07-11
0
0
张飞飞96
Lv.1
请问这个问题有解决了吗
2024-03-01
0
0

J5 RTX 4090 训练 报错：RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

J5 RTX 4090 训练报错：RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)