Code

optimize spm tokenizer for long text#799

Code

Merged

wenbingl merged 3 commits intomainmicrosoft/onnxruntime-extensions:mainfrom spm_optmicrosoft/onnxruntime-extensions:spm_optCopy head branch name to clipboardAug 30, 2024

+53-21

Commits

Commits on Aug 29, 2024

• optimize spm tokenizer for long text

  wenbingl

  committed
  be50c1c

  Copy full SHA for be50c1c

• refine the split logic

  wenbingl

  committed
  0adf533

  Copy full SHA for 0adf533

Commits on Aug 30, 2024

• re-trigger CI pipeline.

  wenbingl

  committed
  6cdf8b7

  Copy full SHA for 6cdf8b7