[Performance] GPU Fallback to CPU Without Error When CUDA DLLs Are Missing

[ibrahimsoliman97]

Describe the issue

When using ONNX Runtime with GPU and setting CUDA as the provider, if the model fails to load to the GPU due to missing CUDA DLLs or other issues, the execution falls back to the CPU without raising any error. This behavior results in the model running on the CPU while merely logging the error in the logs, but no explicit error is returned to the application.

This can lead to scenarios where the user is unaware that the execution has switched to the CPU, which might significantly impact performance.

Expected Behavior:
The behavior should raise an error and stop the execution if the GPU cannot be initialized correctly, ensuring that the user is immediately aware of the issue.

Current Behavior:

• ONNX Runtime logs the issue (e.g., missing DLLs or GPU initialization errors).
• The model seamlessly falls back to CPU execution.
• No error is returned to the application.

To reproduce

1. Set up ONNX Runtime with CUDA as the provider on a system missing required CUDA DLLs.
2. Attempt to load a model.
3. Observe the logs and the fallback to CPU without any error raised in the application.

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.18.1

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.8

Model File

No response

Is this a quantized model?

No