[wip]chunked prefill

[mailvijayasingh]

Description

Chunked Prefill based on paper - https://arxiv.org/abs/2308.16369

Prototype PR - #1237

The code is not wired and has different components for chunked prefill
mainly 2 parts-
i) Fetching previous cache is there are processed chunks
ii) Generating attention masks for a particular chunk based on article - https://medium.com/byte-sized-ai/llm-inference-optimizations-2-chunked-prefill-764407b3a67a

Need initial feedback on the PR

The rest of the description includes relevant details and context, examples:

• why is this change being made,
• the problem being solved and any relevant context,
• why this is a good solution,
• some information about the specific implementation,
• shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.