lmdeploy - ERROR - __init__.py:17 - ModuleNotFoundError: No module named 'dlinfer'

[jiabao-wang]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

when i run :
from lmdeploy import pipeline
from lmdeploy import PytorchEngineConfig
if name == "main":
    pipe = pipeline("internlm/internlm2_5-7b-chat",
     backend_config=PytorchEngineConfig(tp=1, device_type="ascend", eager_mode=True))
    question = ["Shanghai is", "Please introduce China", "How are you?"]
    response = pipe(question)
    print(response)
  
at 华为昇腾 Ascend local

output:
Fetching 20 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 71943.46it/s]
2024-11-07 12:55:24,870 - lmdeploy - ERROR - init.py:17 - ModuleNotFoundError: No module named 'dlinfer'
2024-11-07 12:55:24,870 - lmdeploy - ERROR - init.py:18 - test failed!
Please ensure it has been installed correctly.

how to fix it? Thank you very much!

Reproduction

from lmdeploy import pipeline
from lmdeploy import PytorchEngineConfig
if name == "main":
pipe = pipeline("internlm/internlm2_5-7b-chat",
backend_config=PytorchEngineConfig(tp=1, device_type="ascend", eager_mode=True))
question = ["Shanghai is", "Please introduce China", "How are you?"]
response = pipe(question)
print(response)

Environment

Ascend-cann-toolkit_7.0.0.alpha003_linux-x86_64.run

Package                   Version
------------------------- -----------
accelerate                1.1.1
aclruntime                0.0.2
addict                    2.4.0
aiohappyeyeballs          2.4.3
aiohttp                   3.10.10
aiosignal                 1.3.1
ais-bench                 0.0.2
annotated-types           0.7.0
anyio                     4.6.2.post1
async-timeout             4.0.3
attrs                     24.2.0
certifi                   2024.8.30
charset-normalizer        3.4.0
click                     8.1.7
cloudpickle               3.1.0
datasets                  3.1.0
dill                      0.3.8
diskcache                 5.6.3
distro                    1.9.0
einops                    0.8.0
exceptiongroup            1.2.2
fastapi                   0.115.4
filelock                  3.16.1
fire                      0.7.0
frozenlist                1.5.0
fsspec                    2024.9.0
h11                       0.14.0
httpcore                  1.0.6
httpx                     0.27.2
huggingface-hub           0.26.2
idna                      3.10
importlib_metadata        8.5.0
interegular               0.3.3
Jinja2                    3.1.4
jiter                     0.7.0
jsonschema                4.23.0
jsonschema-specifications 2024.10.1
lark                      1.2.2
llvmlite                  0.43.0
lmdeploy                  0.6.2
markdown-it-py            3.0.0
MarkupSafe                3.0.2
mdurl                     0.1.2
mmengine-lite             0.10.5
modelscope                1.19.2
mpmath                    1.3.0
multidict                 6.1.0
multiprocess              0.70.16
nest-asyncio              1.6.0
networkx                  3.2.1
numba                     0.60.0
numpy                     1.26.4
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         9.1.0.70
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu12          2.20.5
nvidia-nvjitlink-cu12     12.6.77
nvidia-nvtx-cu12          12.1.105
openai                    1.54.3
outlines                  0.0.46
packaging                 24.1
pandas                    2.2.3
peft                      0.11.1
pillow                    11.0.0
pip                       24.2
platformdirs              4.3.6
propcache                 0.2.0
protobuf                  5.28.3
psutil                    6.1.0
pyairports                2.1.1
pyarrow                   18.0.0
pycountry                 24.6.1
pydantic                  2.9.2
pydantic_core             2.23.4
Pygments                  2.18.0
pynvml                    11.5.3
python-dateutil           2.9.0.post0
pytz                      2024.2
PyYAML                    6.0.2
referencing               0.35.1
regex                     2024.11.6
requests                  2.32.3
rich                      13.9.4
rpds-py                   0.21.0
safetensors               0.4.5
sentencepiece             0.2.0
setuptools                72.1.0
shortuuid                 1.0.13
six                       1.16.0
sniffio                   1.3.1
starlette                 0.41.2
sympy                     1.13.3
termcolor                 2.5.0
tiktoken                  0.8.0
tokenizers                0.20.3
tomli                     2.0.2
torch                     2.4.0
torchvision               0.19.0
tqdm                      4.66.6
transformers              4.46.2
triton                    3.0.0
typing_extensions         4.12.2
tzdata                    2024.2
urllib3                   2.2.3
uvicorn                   0.32.0
wheel                     0.43.0
xxhash                    3.5.0
yapf                      0.40.2
yarl                      1.17.1
zipp                      3.20.2

Error traceback

Fetching 20 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 71943.46it/s]
2024-11-07 12:55:24,870 - lmdeploy - ERROR - __init__.py:17 - ModuleNotFoundError: No module named 'dlinfer'
2024-11-07 12:55:24,870 - lmdeploy - ERROR - __init__.py:18 - <PyTorch> test failed!
Please ensure it has been installed correctly.

[lvhan028]

https://lmdeploy.readthedocs.io/en/latest/get_started/ascend/get_started.html