Code

[WIP] support paged attention#710

Code

Draft

jslhcl wants to merge 18 commits intomainmicrosoft/onnxruntime-extensions:mainfrom leca/pagedAttentionmicrosoft/onnxruntime-extensions:leca/pagedAttentionCopy head branch name to clipboard

+3,273-1,008

Commits

Commits on May 6, 2024

• support paged attention

  

  Lei Cao

  committed
  0927cca

  Copy full SHA for 0927cca

• Merge branch 'main' of https://github.com/microsoft/onnxruntime-extensions into leca/pagedAttention

  

  Lei Cao

  committed
  cf10813

  Copy full SHA for cf10813

Commits on May 7, 2024

• add kernel functions and build successfully

  

  Lei Cao

  committed
  6931a52

  Copy full SHA for 6931a52

Commits on May 8, 2024

• move checkInput() to paged_attention.h

  

  Lei Cao

  committed
  77ae3b0

  Copy full SHA for 77ae3b0

• call flash attention code for prompt mode

  

  Lei Cao

  committed
  9ac5f18

  Copy full SHA for 9ac5f18

Commits on May 20, 2024

• runtime error in RunMultiHeadAttention

  jslhcl

  committed
  90ee6a6

  Copy full SHA for 90ee6a6

Commits on May 21, 2024

• UT can run now

  jslhcl

  committed
  17c2c41

  Copy full SHA for 17c2c41

Commits on May 29, 2024

• new UT with q,k and v whose shape is 381x512, and not use GetScratchBufferUnderMultiStream

  jslhcl

  committed
  75b2d04

  Copy full SHA for 75b2d04

Commits on Jun 2, 2024

• make context_lens CPU input, change scale in test, change batch_size and seq_len

  jslhcl

  committed
  68dac5d

  Copy full SHA for 68dac5d

Commits on Jun 4, 2024

• change window_size and causal to make ut pass

  jslhcl

  committed
  e8bf9ea

  Copy full SHA for e8bf9ea

Commits on Jun 7, 2024

• enhance flash attention lib to support page attention, and UT

  jslhcl

  committed
  8352a13

  Copy full SHA for 8352a13

Commits on Jun 10, 2024

• runtime error in gpu_data_transfer.cc when copying result back to cpu for test test_cuda_paged_attention_prompt_decoding

  jslhcl

  committed
  4bfcfad

  Copy full SHA for 4bfcfad

Commits on Jun 19, 2024

• sync ort-genai changes

  jslhcl

  committed
  05c0ba2

  Copy full SHA for 05c0ba2

Commits on Jun 22, 2024

• sync flash attn 2.5.9 kernel code. test_cuda_paged_attention3 is good, test_cuda_paged_attention_decoding runtime error

  jslhcl

  committed
  7191ea5

  Copy full SHA for 7191ea5

Commits on Jun 23, 2024

• fix runtime error and add new test case

  jslhcl

  committed
  654cabf

  Copy full SHA for 654cabf

Commits on Jun 25, 2024

• UT for paged attention decoding case works

  jslhcl

  committed
  e374fb4

  Copy full SHA for e374fb4

Commits on Jun 26, 2024

• fix block_tables and context_lens to avoid unknown random error

  jslhcl

  committed
  17cf303

  Copy full SHA for 17cf303

Commits on Jul 1, 2024

• sync with ort-genai and add document

  jslhcl

  committed
  cb65793

  Copy full SHA for cb65793