Some questions about best practice

[wencynyu]

Congratulations for the amazing accelerate project, but I have some questions hoping replies.
In my simply understanding, ktransformers provide an affordable plan to deploy a huge llm model, mainly with a moe struct like mistral7b*22 or deepseek r1 671b, in PC by spliting different model struct to ram and vram.
Q1: Can ktransformers accelerate a dense model like llama3.3, qwen2.5？

With your best practice hardware:

CPU: Intel (R) Xeon (R) Gold 6454S 1T DRAM (2 NUMA nodes)
GPU: 4090D 24G VRAM
Memory: standard DDR5-4800 server DRAM (1 TB)

I think the memory struct maybe 8+8=16 channel, wich can provide 600+GB/s bandwidth. With 4090 1000+GB/s vram bandwidth and 600GB/s ram bandwidth, 330Tops gpu calculate capacity, 3Ghz * 64c * 2 unit = about 382Gops(avx), your doc shows it can provide 13.69 token/s performance in deepseek r1 671b q4.
Q2: What's the main bottleneck in this heterogeneous system?
Q3: Whether epyc 9005 serias cpu * 2 with 24 memory channel can provide a better performance?

[wencynyu]

I find some documents today, according to the official doc and r1 model struct, ktransformers keep some heavily calculate-depend layer in gpu, and storage the experts of moe in ram, which will only activate 8 of 256 in the real progress of inference(about 37b, lightly calculate-depend layer).

• shared expert & gate
  

• normal experts
  

And according to the r1 deployment performance and the hardware analysis in question description, the bottleneck matches the ram bandwidth.

With 4090 1000+GB/s vram bandwidth and 600GB/s ram bandwidth, 330Tops gpu calculate capacity, 3Ghz * 64c * 2 unit = about 382Gops(avx), your doc shows it can provide 13.69 token/s performance in deepseek r1 671b q4.

The official hardware system can provide 600GB/37b2=32token/s performance in theory, actually it performs 14token/s, I think there must exist some utilization reason. And the result fits my PC performance: with 7945hx + ddr5 56002 (120GB/s), when I run a qwen2.5 32b q8 model, it performs 1.83 tokens/s eventually.