Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to run Phi3 on NPU with ORT+DML via OnnxRuntimeGenai library #1180

Open
Gusha-nye opened this issue Jan 13, 2025 · 0 comments
Open

How to run Phi3 on NPU with ORT+DML via OnnxRuntimeGenai library #1180

Gusha-nye opened this issue Jan 13, 2025 · 0 comments
Labels

Comments

@Gusha-nye
Copy link

Currently, in the example given in the official microsoft documentation(https://learn.microsoft.com/zh-cn/windows/ai/models/get-started-models-genai), it is possible to run phi3 models on GPUs using DML acceleration under the ORT framework with the OnnxRuntimeAI library. However, I now want to deploy it to reason on NPU, but I don't seem to see any parameter in the GeneratorParams class or Generator class of the OnnxRuntimeGenAI library where I can set the hardware (CPU, GPU, NPU) reasoning platform. Is it currently possible to implement ORT+DML based running Phi3 on NPU? If it is possible, please tell me how to set it up? If there is a demo can you send it for reference? (C#, python, C++ are all acceptable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant