Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

demo-python > Data file consolidation and doc updates #304

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

demo-python > Data file consolidation and doc updates #304

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 15 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,50 +4,34 @@ This repository provides Python, C#, REST, and JavaScript code samples for [vect

There are breaking changes from REST API version 2023-07-01-Preview to newer API versions. These breaking changes also apply to the Azure SDK beta packages targeting that REST API version. See [Upgrade REST APIs](https://learn.microsoft.com/azure/search/search-api-migration) for migration guidance.

## Feature status

Vector support consists of generally available features and preview features.

| Feature | Status |
|---------|--------|
| [vector indexing](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index) | generally available (2023-11-01 and stable SDK packages) |
| [vector queries](https://learn.microsoft.com/azure/search/vector-search-how-to-query) | generally available (2023-11-01 and stable SDK packages)|
| [integrated data chunking](https://learn.microsoft.com/azure/search/cognitive-search-skill-textsplit) | public preview (2023-10-01-preview and later, plus beta SDK packages) |
| [integrated embedding](https://learn.microsoft.com/azure/search/cognitive-search-skill-azure-openai-embedding) | public preview (2023-10-01-preview and later, plus beta SDK packages) |
| [index projections](https://learn.microsoft.com/azure/search/index-projections-concept-intro) | public preview (2023-10-01-preview and later, plus beta SDK packages) |
| [vectorizers](https://learn.microsoft.com/azure/search/vector-search-how-to-configure-vectorizer) | public preview (2023-10-01-preview and later, plus beta SDK packages) |
| [scalar quantization](https://learn.microsoft.com/azure/search/vector-search-how-to-configure-compression-storage) | public preview (2024-03-01-preview and later, plus beta SDK packages) |
| [OneLake indexer](https://learn.microsoft.com/azure/search/search-how-to-index-onelake-files) | public preview (2024-05-01-preview, plus beta SDK packages) |
| [Binary vectors support](https://learn.microsoft.com/azure/search/vector-search-how-to-index-binary-data) | public preview (2024-05-01-preview, plus beta SDK packages) |

Preview features are available under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

## demo-python samples

| Sample | Description | Status |
| ------ | ------------|--------|
| [demo-python readme](demo-python/readme.md) | A growing collection of notebooks that demonstrate aspects of vector search support, including data chunking and embedding of both text and image content and queries, using a variety of frameworks and techniques.| GA and preview |
| Sample | Description |
| ------ | ------------|
| [demo-python readme](demo-python/readme.md) | A growing collection of notebooks that demonstrate aspects of vector search support, including data chunking and embedding of both text and image content and queries, using a variety of frameworks and techniques.|

## demo-dotnet samples

| Sample | Description | Status |
| ------ | ------------|--------|
| [DotNetVectorDemo](demo-dotnet/DotNetVectorDemo/readme.md) | A .NET console app that calls Azure OpenAI to vectorize data. It then calls Azure AI Search to create, load, and query vector data.| Generally available (GA) |
| [DotNetIntegratedVectorizationDemo](demo-dotnet/DotNetIntegratedVectorizationDemo/readme.md) | A .NET console app that calls Azure AI Search to create an index, indexer, data source, and skillset. An Azure Storage account provides the data. Azure OpenAI is called by the skillset during indexing, and again during query execution to vectorize text queries. | Public preview |
| [QuantizationAndStorageOptions](demo-dotnet/demo-dotnet/QuantizationAndStorageOptions/readme.md) | A .NET console app that demonstrates [narrow data types](https://learn.microsoft.com/rest/api/searchservice/supported-data-types#edm-data-types-for-vector-fields) and built-in scalar quantization, reducing vector index size in memory and on disk. It also disables storage of vectors returned in query response, which you don't need if you're not returning vectors in a query. | Public preview |
| Sample | Description |
| ------ | ------------|
| [DotNetVectorDemo](demo-dotnet/DotNetVectorDemo/readme.md) | A .NET console app that calls Azure OpenAI to vectorize data. It then calls Azure AI Search to create, load, and query vector data.|
| [DotNetIntegratedVectorizationDemo](demo-dotnet/DotNetIntegratedVectorizationDemo/readme.md) | A .NET console app that calls Azure AI Search to create an index, indexer, data source, and skillset. An Azure Storage account provides the data. Azure OpenAI is called by the skillset during indexing, and again during query execution to vectorize text queries. |
| [QuantizationAndStorageOptions](demo-dotnet/demo-dotnet/QuantizationAndStorageOptions/readme.md) | A .NET console app that demonstrates [narrow data types](https://learn.microsoft.com/rest/api/searchservice/supported-data-types#edm-data-types-for-vector-fields) and built-in scalar quantization, reducing vector index size in memory and on disk. It also disables storage of vectors returned in query response, which you don't need if you're not returning vectors in a query. |

## demo-java samples

| Sample | Description | Status |
| ------ | ------------|--------|
| [demo-vectors](demo-java/demo-vectors/readme.md) | A Java console app that calls Azure OpenAI to vectorize data. It then calls Azure AI Search to create, load, and query vector data.| GA |
| [demo-integrated-vectorization](demo-java/demo-integrated-vectorization/readme.md) | A Java console app that calls Azure AI Search to create an index, indexer, data source, and skillset. An Azure Storage account provides the data. Azure OpenAI is called by the skillset during indexing, and again during query execution to vectorize text queries. | GA and preview |
| Sample | Description |
| ------ | ------------|
| [demo-vectors](demo-java/demo-vectors/readme.md) | A Java console app that calls Azure OpenAI to vectorize data. It then calls Azure AI Search to create, load, and query vector data.|
| [demo-integrated-vectorization](demo-java/demo-integrated-vectorization/readme.md) | A Java console app that calls Azure AI Search to create an index, indexer, data source, and skillset. An Azure Storage account provides the data. Azure OpenAI is called by the skillset during indexing, and again during query execution to vectorize text queries. |

## demo-javascript samples

| Sample | Description | Status |
| ------ | ------------|--------|
| [JavaScriptVectorDemo](demo-javascript/readme.md) | A single folder contains three code samples. The `azure-search-vector-sample.js` script calls just Azure OpenAI and is used to generate embeddings for fields in an index. The `docs-text-openai-embeddings.js` program is an end-to-end code sample that calls Azure OpenAI for embeddings and Azure AI Seach to create, load, and query an index that contains vectors. The `query-text-openai-embeddings.js` script generates an embedding for a vector query. | GA and preview |
| Sample | Description |
| ------ | ------------|
| [JavaScriptVectorDemo](demo-javascript/readme.md) | A single folder contains three code samples. The `azure-search-vector-sample.js` script calls just Azure OpenAI and is used to generate embeddings for fields in an index. The `docs-text-openai-embeddings.js` program is an end-to-end code sample that calls Azure OpenAI for embeddings and Azure AI Seach to create, load, and query an index that contains vectors. The `query-text-openai-embeddings.js` script generates an embedding for a vector query. |

## Other vector samples and tools

Expand Down
0 ...python/data/documents/Benefit_Options.pdf → data/benefitdocs/Benefit_Options.pdf
View file
Open in desktop
File renamed without changes.
0 ...orthwind_Health_Plus_Benefits_Details.pdf → ...orthwind_Health_Plus_Benefits_Details.pdf
View file
Open in desktop
File renamed without changes.
0 ...s/Northwind_Standard_Benefits_Details.pdf → ...s/Northwind_Standard_Benefits_Details.pdf
View file
Open in desktop
File renamed without changes.
0 demo-python/data/documents/PerksPlus.pdf → data/benefitdocs/PerksPlus.pdf
View file
Open in desktop
File renamed without changes.
0 ...thon/data/documents/employee_handbook.pdf → data/benefitdocs/employee_handbook.pdf
View file
Open in desktop
File renamed without changes.
0 demo-python/data/documents/role_library.pdf → data/benefitdocs/role_library.pdf
View file
Open in desktop
File renamed without changes.
0 ...hon/data/evalquestions/ragas/testset.json → data/evalquestions/ragas/testset.json
View file
Open in desktop
File renamed without changes.
0 demo-python/data/images/apples/Braeburn.jpeg → data/images/apples/Braeburn.jpeg
View file
Open in desktop
File renamed without changes
0 demo-python/data/images/apples/Fiji.jpeg → data/images/apples/Fiji.jpeg
View file
Open in desktop
File renamed without changes
0 demo-python/data/images/apples/Gala.jpeg → data/images/apples/Gala.jpeg
View file
Open in desktop
File renamed without changes
0 ...n/data/images/apples/GoldenDelicious.jpeg → data/images/apples/GoldenDelicious.jpeg
View file
Open in desktop
File renamed without changes
0 ...ython/data/images/apples/GrannySmith.jpeg → data/images/apples/GrannySmith.jpeg
View file
Open in desktop
File renamed without changes
0 ...python/data/images/apples/Honeycrisp.jpeg → data/images/apples/Honeycrisp.jpeg
View file
Open in desktop
File renamed without changes
0 demo-python/data/images/apples/Jonagold.jpeg → data/images/apples/Jonagold.jpeg
View file
Open in desktop
File renamed without changes
0 demo-python/data/images/apples/PinkLady.jpeg → data/images/apples/PinkLady.jpeg
View file
Open in desktop
File renamed without changes
0 ...thon/data/images/apples/RedDelicious.jpeg → data/images/apples/RedDelicious.jpeg
View file
Open in desktop
File renamed without changes
0 demo-python/data/images/apples/Rome.jpeg → data/images/apples/Rome.jpeg
View file
Open in desktop
File renamed without changes
0 ...python/data/images/apples/two_apples.jpeg → data/images/apples/two_apples.jpeg
View file
Open in desktop
File renamed without changes
0 ...ython/data/images/indexer-composition.png → data/images/indexer-composition.png
View file
Open in desktop
File renamed without changes
0 ...ython/data/images/python-vector-video.gif → data/images/python-vector-video.gif
View file
Open in desktop
File renamed without changes
0 ...ments/MSFT_cloud_architecture_contoso.pdf → ...ments/MSFT_cloud_architecture_contoso.pdf
View file
Open in desktop
File renamed without changes.
0 ...data-chunking/data/earth_at_night_508.pdf → data/nasa-ebooks/earth_at_night_508.pdf
View file
Open in desktop
File renamed without changes.
0 ...-chunking/data/earth_book_2019_tagged.pdf → data/nasa-ebooks/earth_book_2019_tagged.pdf
View file
Open in desktop
File renamed without changes.
0 ...hon/data/ocrdocuments/Benefit_Options.pdf → data/ocrdocuments/Benefit_Options.pdf
View file
Open in desktop
File renamed without changes.
0 demo-python/data/text-sample.json → data/text-sample.json
View file
Open in desktop
File renamed without changes.
7 changes: 6 additions & 1 deletion demo-javascript/code/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion demo-javascript/code/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"@azure/openai": "^1.0.0-beta.12",
"@azure/search-documents": "^12.1.0-beta.1",
"commander": "^12.0.0",
"dotenv": "^16.4.5"
"dotenv": "^16.4.5",
"text-openai-embedding-js": "file:"
}
}
4 changes: 2 additions & 2 deletions demo-python/code/.env-sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ AZURE_AI_SERVICES_KEY=
AZURE_OPENAI_ENDPOINT=your-openai-endpoint
# Optional, only required if not using RBAC authentication
AZURE_OPENAI_KEY=
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-ada-002
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-large

# Used for ragas
AZURE_OPENAI_ADA002_EMBEDDING_DEPLOYMENT=text-embedding-ada-002
Expand All @@ -39,7 +39,7 @@ AZURE_OPENAI_3_LARGE_EMBEDDING_DEPLOYMENT=text-embedding-3-large
AZURE_OPENAI_API_VERSION=2024-05-01-preview

# Used for azure-search-vector-python-sample, azure-search-integrated-vectorization-sample, azure-search-vector-python-llamaindex-sample
AZURE_OPENAI_EMBEDDING_MODEL_NAME=text-embedding-ada-002
AZURE_OPENAI_EMBEDDING_MODEL_NAME=text-embedding-3-large
AZURE_OPENAI_EMBEDDING_DIMENSIONS=1536

# Used for azure-search-vector-image-index-creation-python-sample
Expand Down
Loading