Continuous Data Ingestion without Azure AI Indexers

[svenfeld]

I have a server which serves PDF files and additional metadata. I'm curious about what would be the best way to build a "pipeline" using prepdocs to have continuous data ingestion without relying on something like Azure AI Indexers.

I’d love to hear some ideas on how to approach this.

[poonkuzhali]

I am currently working on a blob triggered Azure Function for a similar situation. The entire prepdocs logic is imported to the Azure Function.

[cforce]

Is there any Code to Share - iac(bicep), Packaging as function and what strategy is implemented in existing codebase to reuse?

[pamelafox]

There's a repo that uses functions for ingestion, with an admin in Streamlit: https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator

For this repo, we've definitely talked about moving prepdocs to an Azure Function. I think we'd still use it with a Blob storage indexer, though, for easier triggering and scheduling. cc @mattgotteiner

@poonkuzhali Are you using an indexer with your setup?

[pamelafox]

Another option is a container apps job, versus Azure Functions, but there's still the question of triggering and scheduling.