Support input embedding size more than 8K token

[rodrigomeireles]

Big files seem to reach the 8k context of text-embedding-3-small using Open AI. There should probably be a chunking/retrieval strategy implementation for this.

[rodrigomeireles]

Example:

read the TODO file
embedding request failed with status code '400' - This model's maximum context length is 8192 tokens, however you requested 11514 tokens (11514 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.

┌────────────────────────────────────────────────────────────────────────────────┐
│ Token Used: 0 - Cost: 0.000000 $ - Chat Model: gpt-4o │
│ Token Used: 55035 - Cost: 0.000106 $ - Embedding Model: text-embedding-3-small │
└────────────────────────────────────────────────────────────────────────────────┘

I think this is just the initial embedding of the whole project because the TODO file itself is very very small.

[meysamhadeli]

Thanks for reporting the issue. each embed for each file send a request asynchronous to AI now. for example here one of your file is need more than 8K token!? base on your input "read the TODO file" RAG try to find similar files and one of these files seems is more that 8k! Do you have any experience about chunk or batch in embedding!?

[rodrigomeireles]

Thanks for reporting the issue. each embed for each file send a request asynchronous to AI now. for example here one of your file is need more than 8K token!? base on your input "read the TODO file" RAG try to find similar files and one of these files seems is more that 8k! Do you have any experience about chunk or batch in embedding!?

I do have experience in chunking and batch embeddings. Why do you ask?
Maybe it's getting context from undesired files? Is it respecting .gitignore? .dockerignore? Is there a command to debug the call to the embeddings API?

[meysamhadeli]

Thanks for reporting the issue. each embed for each file send a request asynchronous to AI now. for example here one of your file is need more than 8K token!? base on your input "read the TODO file" RAG try to find similar files and one of these files seems is more that 8k! Do you have any experience about chunk or batch in embedding!?

I do have experience in chunking and batch embeddings. Why do you ask? Maybe it's getting context from undesired files? Is it respecting .gitignore? .dockerignore? Is there a command to debug the call to the embeddings API?

If you have some experience and have time, feel free to add this enhancement, otherwise I will work on that and fix this issue.
Yes, project handle .gitignore properly but .dockerignore not yet. also I think you get undesired files maybe! there is no separate command for calling embedding api separately. if you wanna check it you can fork the project and debug it.

[meysamhadeli]

Thanks for reporting the issue. each embed for each file send a request asynchronous to AI now. for example here one of your file is need more than 8K token!? base on your input "read the TODO file" RAG try to find similar files and one of these files seems is more that 8k! Do you have any experience about chunk or batch in embedding!?

I do have experience in chunking and batch embeddings. Why do you ask? Maybe it's getting context from undesired files? Is it respecting .gitignore? .dockerignore? Is there a command to debug the call to the embeddings API?

@rodrigomeireles Problem solved in this version v1.7.4.

Also, now you can use .codai-gitignore in root of your directory beside of config for ignore unnecessary files like image.

Please check it out.

[meysamhadeli]

This issue is fixed in PR #88, and I closed this issue. if there is any other issue related to that, create a new issue.