An error occurred when starting the API server in Docker and using the API locally

[lililolo0927]

An error occurred when starting the API server in Docker and using the API locally

The cmd I used to create container is
docker run --gpus all -d -v /mnt/data/models:/models -p 10002:10002 --name ktransformers -itd approachingai/ktransformers:0.2.1
and running well with port 10002 opened .

Inside the container, I typed the following command to start server, referring to this link, https://kvcache-ai.github.io/ktransformers/en/install.html#restful-api-and-web-ui

ktransformers --gguf_path /models/gguf/DeepSeek-V3-Q4_K_M --model_path deepseek-ai/DeepSeek-V3 --model_name DeepSeek-V3

Then it shows as the following in the container, (started well)

And if I try the example as https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/api/server/api.md ,
on local server, the followings were returned,

And on the server,

Please share the ideas to fix this problems! Thanks

[ubergarm]

1. Only very recently did the API endpoint start working after PR382 ktransformers@ee24eb8 Before yesterday 2.1 didn't really work with API, local chat only.
2. You're on python 3.10, I've only tested with 3.11 using uv (i don't use conda or poetry)
3. Full guide including prebuilt binary .whl files here if u interested https://github.com/ubergarm/r1-ktransformers-guide

Its working pretty well now with the very tip of main ktransformers@25c5bdd which is what the prebuilt whl i provide above is.

Holler again if you're getting the same errors after updating! Or do git rev-parse --short HEAD inside the container perhaps to see what version you are on.