6. Azure Cognitive Search Index Generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Visual Search - Azure Cognitive Search Index Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/retkowsky/images/blob/master/visualsearchlogo.jpg?raw=true\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Search with Azure Cognitive Search, Sentence Transformers, Azure Computer Vision and bar code/QR code detection\n",
    "\n",
    "## Description\n",
    "The goal of this is **Azure AI asset is to enable search over Text and Images using Azure Cognitive Search**. The technique was inspired by a research article which show how to **convert vectors (embeddings) to text which allows the Cognitive Search service to leverage the inverted index to quickly find the most relevant items**. For this reason, any model that will convert an object to a vector can be leveraged if the number of dimensions in the resulting vector is less than 3,000. It also allows users to leverage existing pretrained or fine-tuned models.<br><br>\n",
    "\n",
    "This technique has shown to be incredibly effective and easy to implement. We are using **Sentence Transformers, which is an OpenAI clip model wrapper**. We need to embed all our existing catalog of images. Then the objects embedding are converted into a set of fake terms and all the results are stored into an Azure Cognitive Search index for handling all the search requests.\n",
    "For example, if an embedding looked like [-0,21, .123, ..., .876], this might be converted to a set of fake terms such as: “A1 B3 … FED0”. This is what is sent as the search query to Azure Cognitive Search.<br><br>\n",
    "\n",
    "We can **enrich the Azure Cognitive Search index by using extracted text from the images using Azure Read API**. We can also detect and extract any information from **bar code and/or QR code** that might be available in the products catalog images. And we can use also **Azure Computer Vision as well to detect the dominant colors of the image, the tags that can describe the image and the caption of each image**. All these information will be ingested into the Azure Cognitive Search index.<br><br>\n",
    "\n",
    "The goal of this asset is to be able to use the inverted index within Azure Cognitive Search to be able to quickly find vectors stored in the search index that are like a vector provided as part of a search query and/or using any AI extracted information (text, dominant colors, …). Unlike techniques like cosine similarity which are slow to process large numbers of items, this leverages an inverted index which enables much more data to be indexed and searched.<br>\n",
    "\n",
    "## Process\n",
    "\n",
    "- We have here a collection of catalog images (466 images).\n",
    "- For each of these images, we will embed them using Sentence Transformers.  Sentence Transformer can be used to map images and texts to the same vector space. As model, we use the OpenAI CLIP Model which was trained on a large set of images and image alt texts.\n",
    "- We can retrieve any text from these images using Azure Read API (if any text is available)\n",
    "- We can retrieve any text information from any bar code or QR code (if any)\n",
    "- All these information will be ingested into an Azure Cognitive Search index\n",
    "- Then if you have a field image, you can embed it and extract any text/barcode information and call the Azure Cognitive Search index to retrieve any similar images using vecText similarity and/or using any query text from the extracted text\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/retkowsky/images/blob/master/process.png?raw=true\">\n",
    "\n",
    "Field images are available in the field images directory (number of images=53)\n",
    "\n",
    "\n",
    "## Azure products documentation\n",
    "- https://azure.microsoft.com/en-us/products/search/ \n",
    "- https://azure.microsoft.com/en-us/products/cognitive-services/computer-vision/#overview \n",
    "- https://learn.microsoft.com/en-us/azure/cognitive-services/Computer-vision/how-to/call-read-api \n",
    "- https://zbar.sourceforge.net/ \n",
    "- https://github.com/liamca/vector-search\n",
    "\n",
    "## Research article\n",
    "https://www.researchgate.net/publication/305910626_Large_Scale_Indexing_and_Searching_Deep_Convolutional_Neural_Network_Features\n",
    "    \n",
    "## Directories\n",
    "- **images**: We have two directories (catalog images, field images)\n",
    "- **model**: Directory to save the clusters of the model\n",
    "- **results**: Directory to save some results\n",
    "- **test**: Directory that contains some testing images\n",
    "\n",
    "## Python notebooks\n",
    "\n",
    "### 0. Settings.ipynb\n",
    "Notebook that contains the link to the images and the importation process of the python required libraries\n",
    "\n",
    "### 1. Catalog images exploration.ipynb\n",
    "This notebook will display some catalog and field images\n",
    "\n",
    "### 2. OpenAI Clip and VecText Clusters.ipynb\n",
    "This notebook will explain what sentence transformers is and will generate the clusters\n",
    "This notebook analyzes a set of existing images to determine a set of \"cluster centers\" that will be used to determine which \"fake words\" are generated for a vector\n",
    "This notebook will take a test set of files (testSamplesToTest) and determine the optimal way to cluster vectors into fake words that will be indexed into Azure Cognitive Search\n",
    "\n",
    "### 3. VecText generation.ipynb\n",
    "This notebook will generate the vectext embedding for all the catalog images\n",
    "\n",
    "### 4. BarCode Information extraction.ipynb\n",
    "This notebook will detect any barcode or QR code from the catalog images and will extract the information\n",
    "\n",
    "### 5. Azure CV for OCR, tags, colors and captions.ipynb\n",
    "This notebook will use Azure Computer Vision or OCR, colors, tags and caption extraction for each of the catalog images.\n",
    "\n",
    "### 6. Azure Cognitive Search Index Generation.ipynb\n",
    "This notebook will show how to ingest all the information into an Azure Cognitive Search index.\n",
    "\n",
    "### 7. Calling Azure Cognitive Search.ipynb\n",
    "We can now test the index using some images similarity visual search or free text queries using azure Cognitive Search.\n",
    "\n",
    "## Python files\n",
    "\n",
    "- **azureCognitiveSearch.py**\n",
    "This python file contains many functions to manage and use Azure Cognitive Search\n",
    "\n",
    "- **myfunctions.py**\n",
    "This python file contains many generic functions used in all the notebooks\n",
    "\n",
    "- **vec2Text.py**\n",
    "This python file contains some functions for the sentence transformers model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "24-oct-2022 Serge Retkowsky | serge.retkowsky@microsoft.com | https://www.linkedin.com/in/serger/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import configparser\n",
    "import json\n",
    "import sys\n",
    "import requests\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "import vec2Text\n",
    "import azureCognitiveSearch as acs\n",
    "import myfunctions as my\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Viewing file: azureCognitiveSearch.py \n",
      "\n",
      "# Azure Cognitive Search python functions\n",
      "# 20-Oct-2022\n",
      "\n",
      "\n",
      "# Multiple imports\n",
      "import os\n",
      "import configparser\n",
      "import requests\n",
      "import json\n",
      "import matplotlib.pyplot as plt\n",
      "import matplotlib.image as mpimg\n",
      "import time\n",
      "import vec2Text\n",
      "\n",
      "import myfunctions as my\n",
      "\n",
      "from azure.core.credentials import AzureKeyCredential\n",
      "from azure.search.documents.indexes import SearchIndexClient\n",
      "from azure.search.documents import SearchClient\n",
      "from azure.search.documents.indexes.models import (\n",
      "    ComplexField,\n",
      "    CorsOptions,\n",
      "    SearchIndex,\n",
      "    ScoringProfile,\n",
      "    SearchFieldDataType,\n",
      "    SimpleField,\n",
      "    SearchableField,\n",
      ")\n",
      "from IPython.display import Image\n",
      "\n",
      "\n",
      "# Azure Cognitive Search connection\n",
      "config_file = 'azureservices.py'\n",
      "\n",
      "# Azure Cognitive Search Index name\n",
      "index_name = \"demo-retail\"\n",
      "\n",
      "# Azure Cognitive Search credentials\n",
      "config = configparser.ConfigParser()\n",
      "config.read(config_file)\n",
      "acs_key = config.get('AzureCognitiveSearch', 'key')\n",
      "acs_endpoint = config.get('AzureCognitiveSearch', 'endpoint')\n",
      "servicename = config.get('AzureCognitiveSearch', 'servicename')\n",
      "\n",
      "\n",
      "# Azure Cognitive Search admin and search clients\n",
      "adminClient = SearchIndexClient(endpoint=acs_endpoint,\n",
      "                                index_name=index_name,\n",
      "                                credential=AzureKeyCredential(acs_key))\n",
      "\n",
      "searchClient = SearchClient(endpoint=acs_endpoint,\n",
      "                            index_name=index_name,\n",
      "                            credential=AzureKeyCredential(acs_key))\n",
      "\n",
      "# User functions for Azure Cognitive Search\n",
      "\n",
      "\n",
      "def create_index():\n",
      "    \"\"\"\n",
      "    Creating a new index\n",
      "    \"\"\"\n",
      "    name = index_name\n",
      "    fields = [\n",
      "        SimpleField(name=\"Id\", type=SearchFieldDataType.String, key=True),\n",
      "        SearchableField(name=\"Content\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True,\n",
      "                        analyzer_name=\"en.microsoft\"),\n",
      "        SearchableField(name=\"VecText\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=False,\n",
      "                        sortable=False),\n",
      "        SearchableField(name=\"Ocrtext\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True),\n",
      "        SearchableField(name=\"Barcodetext\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True),\n",
      "        SearchableField(name=\"Colors\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True),\n",
      "        SearchableField(name=\"Tags\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True),\n",
      "        SearchableField(name=\"Caption\",\n",
      "                        type=SearchFieldDataType.String,\n",
      "                        facetable=False,\n",
      "                        filterable=True,\n",
      "                        sortable=True),\n",
      "    ]\n",
      "\n",
      "    cors_options = CorsOptions(allowed_origins=[\"*\"], max_age_in_seconds=60)\n",
      "    index = SearchIndex(name=name, fields=fields, cors_options=cors_options)\n",
      "\n",
      "    try:\n",
      "        print(\"Creating new index\", index_name)\n",
      "        result = adminClient.create_index(index)\n",
      "        print(result.name, \"index has been created\")\n",
      "\n",
      "    except Exception as ex:\n",
      "        print(ex)\n",
      "\n",
      "\n",
      "def delete_index():\n",
      "    \"\"\"\n",
      "    Delete any existing Azure cognitive search index\n",
      "    \"\"\"\n",
      "    try:\n",
      "        print(\"Deleting index\", index_name)\n",
      "        result = adminClient.delete_index(index_name)\n",
      "        print(index_name, 'index has been deleted')\n",
      "\n",
      "    except Exception as ex:\n",
      "        print(ex)\n",
      "\n",
      "\n",
      "def execute_file_search(imagefile, centers, model, topn=10):\n",
      "    \"\"\"\n",
      "    File search\n",
      "    \"\"\"\n",
      "    return searchClient.search(search_text=vec_image_to_text(\n",
      "        imagefile, centers, model),\n",
      "                               include_total_count=True,\n",
      "                               select='Content',\n",
      "                               top=topn)\n",
      "\n",
      "\n",
      "def execute_query_search(textquery, centers, model, topn):\n",
      "    \"\"\"\n",
      "    Query search\n",
      "    \"\"\"\n",
      "    return searchClient.search(search_text=vec_query_to_text(\n",
      "        textquery, centers, model),\n",
      "                               include_total_count=True,\n",
      "                               select='Content',\n",
      "                               top=topn)\n",
      "\n",
      "\n",
      "def index_status(index_name):\n",
      "    \"\"\"\n",
      "    Azure cognitive search Index status\n",
      "    \"\"\"\n",
      "    print(\"Azure Cognitive Search Index:\", index_name, \"\\n\")\n",
      "    headers = {'Content-Type': 'application/json', 'api-key': acs_key}\n",
      "    params = {'api-version': '2020-06-30'}\n",
      "    indexstatus = requests.get(acs_endpoint + \"/indexes/\" + index_name,\n",
      "                               headers=headers,\n",
      "                               params=params)\n",
      "    print(json.dumps((indexstatus.json()), indent=4))\n",
      "\n",
      "\n",
      "def global_search(imagefile, centers, model, topn=5):\n",
      "    \"\"\"\n",
      "    Global Search using vecText, OCR and BarCode with Azure Cognitive Search\n",
      "    \"\"\"\n",
      "    start = time.time()\n",
      "\n",
      "    if not os.path.exists(imagefile):\n",
      "        print(\"[Error] File does not exist:\", imagefile)\n",
      "        return\n",
      "\n",
      "    global results_list\n",
      "\n",
      "    # 1 Images search using sentence transformers\n",
      "    print(\n",
      "        \"\\033[1;31;32m[Step 1. Running images similarity visual search...]\\033[0m\\n\"\n",
      "    )\n",
      "    vectext_list, scores_list = similar_images(\n",
      "        imagefile, centers, model, False,\n",
      "        topn=topn)  # Sentence Transformers only\n",
      "    final_list = vectext_list\n",
      "\n",
      "    # 2 Get OCR text\n",
      "    print(\n",
      "        \"\\n\\033[1;31;32m[Step 2. Extracting any text from the image...]\\033[0m\\n\"\n",
      "    )\n",
      "    # Getting results from Azure Read API\n",
      "    ocr_text = my.get_ocr_text(imagefile)\n",
      "\n",
      "    if ocr_text != '':\n",
      "        print(\"\\033[1;31;34m[Result] Text has been found:\")\n",
      "        print(\"OCR results from the image:\", ocr_text)\n",
      "        print(\"\\033[0m\")\n",
      "        ocr_list = open_text_query(\n",
      "            ocr_text, False,\n",
      "            topn)  # Searching using the Azure Read API results\n",
      "\n",
      "    if ocr_text == '':\n",
      "        print(\"\\033[1;31;91m[Result] No text has been found\\033[0m\")\n",
      "        ocr_list = []\n",
      "\n",
      "    # 3. Get Barcode OCR\n",
      "    print(\n",
      "        \"\\n\\033[1;31;32m[Step 3. Extracting any text any barcode/QRCode of the image...]\\033[0m\\n\"\n",
      "    )\n",
      "    barcode_txt = my.get_barcode_text(\n",
      "        imagefile)  # Get Barcode/QRCode text informations\n",
      "\n",
      "    if barcode_txt != '':\n",
      "        print(\"\\033[1;31;34m[Result] Barcode/QRCode text has been found:\")\n",
      "        print(\"Barcode results from the image:\", barcode_txt)\n",
      "        print(\"\\033[0m\")\n",
      "        barcode_list = open_text_query(\n",
      "            barcode_txt, False,\n",
      "            topn)  # Searching using the Barcode/QRCode results\n",
      "\n",
      "    if barcode_txt == '':\n",
      "        print(\"\\033[1;31;91m[Result] No barcode/QRCode has been found\\033[0m\")\n",
      "        barcode_list = []\n",
      "\n",
      "    results_list = vectext_list + ocr_list + barcode_list\n",
      "\n",
      "    idx = 0\n",
      "    txt_to_remove = ['[', ']']\n",
      "    clean_list = list()\n",
      "\n",
      "    # Need to remove some extra characters\n",
      "    while idx < len(results_list):\n",
      "        results_list[idx] = \"\".join(i for i in results_list[idx]\n",
      "                                    if i not in txt_to_remove)\n",
      "        clean_list.append(results_list[idx])\n",
      "        idx += 1\n",
      "\n",
      "    results_list = list()\n",
      "\n",
      "    for item in clean_list:\n",
      "        if item not in results_list:\n",
      "            results_list.append(item)\n",
      "\n",
      "    del clean_list\n",
      "\n",
      "    print(\"\\nDone in\", round((time.time() - start), 5), \"sec\")\n",
      "\n",
      "    return results_list, vectext_list, ocr_list, barcode_list\n",
      "\n",
      "\n",
      "def global_search_img_list(filelist):\n",
      "    \"\"\"\n",
      "    View all the images from a list of images files\n",
      "    \"\"\"\n",
      "    try:\n",
      "        idx = 0\n",
      "        while idx < len(filelist):\n",
      "            img = mpimg.imread(filelist[idx])\n",
      "            plt.figure()\n",
      "            plt.title(\"Image \" + str(idx + 1) + \" : \" + filelist[idx])\n",
      "            plt.imshow(img)\n",
      "            idx += 1\n",
      "\n",
      "    except:\n",
      "        print(\"[Error] Cannot do a global search\")\n",
      "\n",
      "\n",
      "def open_text_query(mytext, view=False, maxlimit=10):\n",
      "    \"\"\"\n",
      "    Using text query on the Azure Cognitive Search index\n",
      "    \"\"\"\n",
      "    results = searchClient.search(search_text=mytext)\n",
      "\n",
      "    idx = 1\n",
      "    imageslist = []\n",
      "\n",
      "    print('\\033[1;31;34m')\n",
      "    print(\"Search using query =\", mytext, \"- note: displaying only the first\",\n",
      "          maxlimit, \"results\", \"\\n\")\n",
      "\n",
      "    for result in results:\n",
      "        print('\\033[1;31;34m', idx, \"Image file:\", result['Content'],\n",
      "              '\\033[0m')\n",
      "\n",
      "        if view:\n",
      "            display(Image(filename=result[\"Content\"], height=256, width=256))\n",
      "\n",
      "        print(\"\\033[1;31;34m\")\n",
      "        print(\"OCR:\", \"\\033[1;31;32m\", result['Ocrtext'], \"\\033[1;31;34m\")\n",
      "        print(\"BarCode:\", \"\\033[1;31;32m\", result['Barcodetext'],\n",
      "              \"\\033[1;31;34m\")\n",
      "        print(\"Colors:\", \"\\033[1;31;32m\", result['Colors'], \"\\033[1;31;34m\")\n",
      "        print(\"Tags:\", \"\\033[1;31;32m\", result['Tags'], \"\\033[1;31;34m\")\n",
      "        print(\"Caption:\", \"\\033[1;31;32m\", result['Caption'], \"\\033[0m\", \"\\n\")\n",
      "\n",
      "        if idx <= maxlimit:\n",
      "            imageslist.append(result[\"Content\"])\n",
      "\n",
      "        idx += 1\n",
      "\n",
      "    return imageslist\n",
      "\n",
      "\n",
      "def sentence_transformers_query_search(textquery, centers, model, topn=5):\n",
      "    \"\"\"\n",
      "    Sentence transformers text query using Open AI clip model\n",
      "    \"\"\"\n",
      "    idx = 1\n",
      "    print(\"Finding\", topn, \"images with Open AI text query:\",\n",
      "          str.upper(textquery), \"\\n\")\n",
      "\n",
      "    results = execute_query_search(textquery, centers, model, topn)\n",
      "\n",
      "    for result in results:\n",
      "        print(result)\n",
      "        print(\"\\033[1;31;34m\")\n",
      "        print(idx, \"Image file:\", result[\"Content\"])\n",
      "        print(\"Similarity score =\", result['@search.score'])\n",
      "        display(Image(filename=result[\"Content\"], height=256, width=256))\n",
      "        print()\n",
      "\n",
      "        idx += 1\n",
      "\n",
      "\n",
      "def similar_images(imagefile, centers, model, view, topn):\n",
      "    \"\"\"\n",
      "    Finding similar images using Azure Cognitive Search index\n",
      "    \"\"\"\n",
      "    if not os.path.exists(imagefile):\n",
      "        print(\"Error. File not exist:\", imagefile)\n",
      "        return\n",
      "\n",
      "    images_list = list()\n",
      "    scores_list = list()\n",
      "\n",
      "    results = execute_file_search(imagefile, centers, model, topn)\n",
      "\n",
      "    print(\"Finding\", topn, \"similar images of reference image:\", imagefile,\n",
      "          \"\\n\")\n",
      "    idx = 1\n",
      "\n",
      "    for result in results:\n",
      "        imgfile = result[\"Content\"]\n",
      "        score = result[\"@search.score\"]\n",
      "        print(\"\\033[1;31;34m\", idx, \"Similar image:\", imgfile,\n",
      "              \"Similarity score =\", score, \"\\033[0m\")\n",
      "\n",
      "        if view:\n",
      "            display(Image(filename=imgfile, width=256, height=256))\n",
      "            print(\"\\n\")\n",
      "\n",
      "        images_list.append(imgfile)\n",
      "        scores_list.append(score)\n",
      "\n",
      "        idx += 1\n",
      "\n",
      "    return images_list, scores_list\n",
      "\n",
      "\n",
      "def upload_documents(documents):\n",
      "    \"\"\"\n",
      "    Uploading documents into an Azure cognitive search index\n",
      "    \"\"\"\n",
      "    try:\n",
      "        result = searchClient.upload_documents(documents=documents)\n",
      "        print(\"Uploading new document...\")\n",
      "        print(\"Upload of new document succeeded: {}\".format(\n",
      "            result[0].succeeded))\n",
      "        print(\"Done\\n\")\n",
      "\n",
      "    except Exception as ex:\n",
      "        print(\"[Error] Cannot load the documents into the index\")\n",
      "        print(ex.message)\n",
      "\n",
      "\n",
      "def vec_image_to_text(imagefile, centers, model):\n",
      "    \"\"\"\n",
      "    Doing vec2Text\n",
      "    \"\"\"\n",
      "    curVec = vec2Text.image_embedding(imagefile, model)\n",
      "    vecText = ''\n",
      "\n",
      "    for d in range(len(curVec)):\n",
      "        vecText += vec2Text.convert_field_num_to_string(d) + str(\n",
      "            vec2Text.closest(centers[d], curVec[d])) + ' '\n",
      "\n",
      "    return vecText\n",
      "\n",
      "\n",
      "def vec_query_to_text(query, centers, model):\n",
      "    \"\"\"\n",
      "    Vec query to text\n",
      "    \"\"\"\n",
      "    curVec = vec2Text.text_embedding(query, model)\n",
      "    vecText = ''\n",
      "\n",
      "    for d in range(len(curVec)):\n",
      "        vecText += vec2Text.convert_field_num_to_string(d) + str(\n",
      "            vec2Text.closest(centers[d], curVec[d])) + ' '\n",
      "    return vecText\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "my.view_file('azureCognitiveSearch.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Viewing file: vec2Text.py \n",
      "\n",
      "# Vec2Text functions\n",
      "# 20-Oct-2022\n",
      "\n",
      "\n",
      "# Multiple imports\n",
      "import base64\n",
      "import collections\n",
      "import gensim\n",
      "import glob\n",
      "import math\n",
      "import numpy as np\n",
      "import os\n",
      "import random\n",
      "import shutil\n",
      "\n",
      "from os import listdir\n",
      "from os.path import isfile, join\n",
      "from pathlib import Path\n",
      "from PIL import Image\n",
      "from sentence_transformers import SentenceTransformer, util\n",
      "from shutil import rmtree\n",
      "from sklearn.cluster import KMeans\n",
      "\n",
      "\n",
      "# Functions definitions\n",
      "\n",
      "\n",
      "def calculate_dimensions(file, model):\n",
      "    \"\"\"\n",
      "    Calculate dimensions\n",
      "    \"\"\"\n",
      "    return image_embedding(file, model).shape[0]\n",
      "\n",
      "\n",
      "def calculate_wcss(data):\n",
      "    \"\"\"\n",
      "    Clustering K Means algorithm from sci-kit learn\n",
      "    \"\"\"\n",
      "    data = np.array(data).reshape(-1, 1)\n",
      "    wcss = []\n",
      "\n",
      "    for n in range(2, 21):\n",
      "        kmeans = KMeans(n_clusters=n)\n",
      "        kmeans.fit(X=data)\n",
      "        wcss.append(kmeans.inertia_)\n",
      "\n",
      "    return wcss\n",
      "\n",
      "\n",
      "def convert_vec_to_bucket_str(curVec, bucketRanges):\n",
      "    \"\"\"\n",
      "    Convert Vect Text To Bucket string\n",
      "    \"\"\"\n",
      "    vecStr = ''\n",
      "\n",
      "    for d in range(len(curVec)):\n",
      "        bucketVal = -1\n",
      "        idx = 0\n",
      "\n",
      "        for i in bucketRanges[d]:\n",
      "            if curVec[d] < i:\n",
      "                bucketVal = idx\n",
      "                break\n",
      "\n",
      "            idx += 1\n",
      "        vecStr += convert_field_num_to_string(d) + '_' + str(\n",
      "            bucketVal).replace('-', '~') + ' '\n",
      "\n",
      "    return vecStr.strip()\n",
      "\n",
      "\n",
      "def convert_field_num_to_string(dim):\n",
      "    \"\"\"\n",
      "    Generate fake terms from\n",
      "    \"\"\"\n",
      "    dimStr = str(dim)\n",
      "    curStr = ''\n",
      "\n",
      "    for i in range(len(dimStr)):\n",
      "        curChar = dimStr[i]\n",
      "        if curChar == '0':\n",
      "            curStr += 'A'\n",
      "        elif curChar == '1':\n",
      "            curStr += 'B'\n",
      "        elif curChar == '2':\n",
      "            curStr += 'C'\n",
      "        elif curChar == '3':\n",
      "            curStr += 'D'\n",
      "        elif curChar == '4':\n",
      "            curStr += 'E'\n",
      "        elif curChar == '5':\n",
      "            curStr += 'F'\n",
      "        elif curChar == '6':\n",
      "            curStr += 'G'\n",
      "        elif curChar == '7':\n",
      "            curStr += 'H'\n",
      "        elif curChar == '8':\n",
      "            curStr += 'I'\n",
      "        elif curChar == '9':\n",
      "            curStr += 'J'\n",
      "\n",
      "    return curStr\n",
      "\n",
      "\n",
      "def closest(lst, K):\n",
      "    \"\"\"\n",
      "    Find the closest cluster center index to a specified number (K)\n",
      "    \"\"\"\n",
      "    b = lst[min(range(len(lst)), key=lambda i: abs(lst[i] - K))]\n",
      "\n",
      "    return lst.index(b)\n",
      "\n",
      "\n",
      "def find_cluster_centers(dimensions, vecDict):\n",
      "    \"\"\"\n",
      "    Get Cluster Centers\n",
      "    \"\"\"\n",
      "    clusterCenters = {}\n",
      "    idx = 0\n",
      "\n",
      "    for d in range(dimensions):\n",
      "        idx += 1\n",
      "\n",
      "        if idx % 10 == 0:\n",
      "            print('Processed', idx, 'of', dimensions)\n",
      "\n",
      "        numberOfClusters = optimal_number_of_clusters(\n",
      "            calculate_wcss(vecDict[str(d)]))\n",
      "        x = np.array(vecDict[str(d)])\n",
      "        km = KMeans(n_clusters=numberOfClusters)\n",
      "        km.fit(x.reshape(-1, 1))\n",
      "        cluster_centers = km.cluster_centers_\n",
      "        cluster_centers = sorted(cluster_centers.tolist())\n",
      "\n",
      "        clusterList = []\n",
      "\n",
      "        for cc in cluster_centers:\n",
      "            clusterList.append(cc[0])\n",
      "        clusterCenters[d] = clusterList\n",
      "\n",
      "    return clusterCenters\n",
      "\n",
      "\n",
      "def get_files_in_dir(in_dir):\n",
      "    \"\"\"\n",
      "    Get files from a directory\n",
      "    \"\"\"\n",
      "    return [\n",
      "        os.path.join(dp, f) for dp, dn, filenames in os.walk(in_dir)\n",
      "        for f in filenames\n",
      "    ]\n",
      "\n",
      "\n",
      "def image_embedding(file, model):\n",
      "    \"\"\"\n",
      "    Encoding image file using sentence transformers OpenAI CLIP model\n",
      "    \"\"\"\n",
      "    img_emb = model.encode(Image.open(file))\n",
      "\n",
      "    return np.array(img_emb)\n",
      "\n",
      "\n",
      "def initialize_vector_dictionary(dimensions):\n",
      "    \"\"\"\n",
      "    Initialize vector dictionary\n",
      "    \"\"\"\n",
      "    vecDict = {}\n",
      "\n",
      "    for d in range(dimensions):\n",
      "        vecDict[str(d)] = []\n",
      "\n",
      "    return vecDict\n",
      "\n",
      "\n",
      "def openai_clip_model(clipmodel='clip-ViT-B-32'):\n",
      "    \"\"\"\n",
      "    Definition of the Open AI Clip model to use\n",
      "    \"\"\"\n",
      "    try:\n",
      "        print(\"Loading OpenAI Clip model:\", clipmodel)\n",
      "        model = SentenceTransformer(clipmodel)\n",
      "        print(\"Done\")\n",
      "\n",
      "    except:\n",
      "        print(\"[Error] Cannot load Open AI CLip model\", clipmodel)\n",
      "\n",
      "    return model\n",
      "\n",
      "\n",
      "def optimal_number_of_clusters(wcss):\n",
      "    \"\"\"\n",
      "    Get optimal Number of Clusters\n",
      "    \"\"\"\n",
      "    x1, y1 = 2, wcss[0]\n",
      "    x2, y2 = 20, wcss[len(wcss) - 1]\n",
      "    distances = []\n",
      "\n",
      "    for i in range(len(wcss)):\n",
      "        x0 = i + 2\n",
      "        y0 = wcss[i]\n",
      "        numerator = abs((y2 - y1) * x0 - (x2 - x1) * y0 + x2 * y1 - y2 * x1)\n",
      "        denominator = math.sqrt((y2 - y1)**2 + (x2 - x1)**2)\n",
      "        distances.append(numerator / denominator)\n",
      "\n",
      "    return distances.index(max(distances)) + 2\n",
      "\n",
      "\n",
      "def string_to_base64(content):\n",
      "    \"\"\"\n",
      "    String to Base 64\n",
      "    \"\"\"\n",
      "    return str(base64.b64encode(content.encode('ascii'))).replace(\"b'\",\n",
      "                                                                  \"\").replace(\n",
      "                                                                      \"'\", \"\")\n",
      "\n",
      "\n",
      "def text_embedding(query, model):\n",
      "    \"\"\"\n",
      "    Encoding text using sentence transformers OpenAI Clip model\n",
      "    \"\"\"\n",
      "    img_emb = model.encode(query)\n",
      "\n",
      "    return np.array(img_emb)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "my.view_file('vec2Text.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today is: 24-10-2022 15:04:49\n"
     ]
    }
   ],
   "source": [
    "print(\"Today is:\", my.get_today())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storage:\n",
      "\n",
      "Total: 126.8 GB\n",
      "- Used: 66.4 GB | 52.37 %\n",
      "- Free: 60.4 GB | 47.61 %\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALEAAADBCAYAAACAC1EEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAUbklEQVR4nO2de5QcVZ3HP9/OmyRMEkNCSEJaMIFAEjK8BOURUFzWcQUR0V1wYRGOIAZdEWkfZ08hchjFIxxZFDWisPLytRDTKoHFLAQIBCUEArp5TV4kBELSJhCTkPz2j1sjzTDPzFTdqp77OafOdN976/5+Nf3tX9+qur9bMjMCgTxT8O1AINBdgogDuSeIOJB7gogDuSeIOJB7gogDuSeIOJB7goirkHSCpMckVSS9KulRScdIukDSfN/+BVqnr28HsoKkfYE5wKXAz4H+wInAjh7ou6+ZvdHdfgKtEyLxm0wCMLO7zGy3mW03s7nALuAW4HhJ2yRtAZBUJ+l2SS9LWiXpa5IKcd0FcRS/QdKrQCTpYEkPSdok6RVJd0ga1mxc0pGSnpa0VdIvJN0j6RtV9R+StEjSlvjXYlp6/5qMY2Zhc7fe9wU2AbcB/wgMr6q7AJjfov3twH3AUKAI/B/wqar2bwAzcb92g4B3AacBA4D9gIeBG+P2/YFVwOeAfsBZwE7gG3H9kcBG4N1AH+B8oAkY4Pv/loXNuwNZ2oDJwE+BtbEIZwOjW4o4FtIO4LCqsk8D8+LXFwCrO7B1JvB0/PokYB2gqvr5VSL+PnBNi/3/Apzs+3+WhS0MJ6owsxfM7AIzGwdMAQ4Abmyl6UjejJ7NrALGVr1fU72DpFGS7pa0TtJfgZ/F/RDbWWexOlvZfwJwRTyU2BIPacbH+/V6gojbwMz+jIvKU4CWU/1ewY2VJ1SVHYiLpn/vosU+18Vl08xsX+A8QHHdemCsJFW1H1/1eg1wrZkNq9r2MbO7un5ktUcQcYykQyVdIWlc/H488M/AAuAlYJyk/gBmtht3BeNaSUMlTQC+gIuubTEU2AZskTQWuLKq7nFgN/BZSX0lnQEcW1X/I+ASSe+WY7CkBklDe+LY804Q8ZtsxZ04PSHpNZx4nwOuAB4ClgAbJL0St58JvAaswI1f7wRubaf/q3EnaBWgDPy6ucLMduJO5j4FbMFF6TnEl/fM7CngYuA/gc3AMty4O0B8IhHIHpKeAG4xs5/49iXrhEicESSdLGn/eDhxPjAN+L1vv/JAuGOXHQ7BjbOHAMuBs81svV+X8kEYTgRyTxhOBHJPEHEg9wQRB3JPEHEg9wQRB3JPEHEg9wQRB3JPl0UsqSjpuRZlkaQvdtcZSfMkHd2JdudIel7SEkl3VpUfKGmupBfi+mI7fZwtydqyJ2m0pDslrZD0R0mPS/pIXDcjzsNbJGmxpAcljdqLQw70ALmLxJImAl8G3mtmhwOfr6q+HbjezCbjZoFtbKOPocDlwBNt1Au4F3jYzA4ys6OATwDjqpo9YmbTzWwasBC4rDvHFdh7elzEki6Po+BiSXfHZYMl3SppYZxHdkZcPiieKL5Y0j24NJ6OuBi42cw2A5jZxrivw4C+ZvZAXL7NzF5vo49rgG8Bf2uj/lRgp5nd0lxgZqvM7KZWjle4aZabO+F7IAGSiMQloD6OUJfEZV8FHjKzY4BTgOslDcZlFr8et70WOKq5E0mz2vipnwRMihMxF0g6vap8i6Rfx1+U6yX1abmzpHpgvJnNaecYDgf+1MFxnihpEbAaeD/tT8MMJMjeiLityRbN5YuBOySdh8tTA/gAUIo/9HnAQFwmxEnEE8nNbHG8L/H7i+J5tC3pC0wEZuAmrc+Ks4b74lLsvwgcAxxEizm3cTbyDbg5wp1G0s2SnpG0sKq4eTgxHvgJLrIHPLA3It4EDG9RNgKXsgPQANyMi6p/lNQXl4bz0fhDn25mB5rZC3H7rs5AWgvcZ2a7zGwlLmFyYlz+tJmtMLfGw724SejVDMWlG82T1AQcB8xuJeIvqd7XzC4D3ofLUm6N2bgvZMADXRaxmW0D1kt6H4CkEcDpwPw40o03sz8AXwKG4aYW3g/MbM4hi3/SwaWtnxuXTcHNoe2Ie3FDEiSNxA0jVuBOroZLahbaqcDzLXyvmNlIMyuaWRGXvfHhViL+Q8BASZdWle3Tjk8n4KZPBnywNynSwGHAH4BF8XZuXN4Pl6rzLC61pxSXDwJ+UFU+p6r8btww4nbgMeDouG5W8+sWtgV8ByfQZ4FPVNWdFvf1LC7Js39c/nWcWFv2Na81G3HdmNi3lcCT8fF+PK6bgUszWgQ8g/syTvKdut5btzCfOJB7cnedOBBoSRBxIPeEHLu9pFgqT8CdiB6GW/lnf9ySVyOAungbiFtTbQfuxsqOqtcbcSekK+O/K4AVTY0NlVQPpAYIY+JOUCyVpwLvwYl2GjAVJ9IkeAW3mMp84BHgqabGhl0J2aoJgohboVgqD8HdhfsgboXMce3vkSjbcXM8HsFdqnysqbEhfGhVBBHHFEvld+BW3vkw7rpvf78etcmLuNWD7m5qbHjUtzNZoNeLuFgqn4RblvWjuLWD88Ry3PX125oaG1Z11LhW6ZUiLpbKw4F/xYl3smd3eoLmBQ4bmxobFnfUuNboVSIulsqjgKtws+vau42cZ34LXNfU2NBrHpTTK0QcR96rgM8Cgz27kxaPAtc0NTbc79uRpKlpERdL5f444X4Vd/22N/Ib4PKmxoYm344kRc2KuFgqz8BNIjrYsytZYDvQCHyzqbGh2480yxo1J+JiqTwY+CbwGd58nEDAsQyY2dTYUFNLxtaUiOPo+2NcVkegbe4ALmlqbNjm25GeoCZEHKLvXvEX4GNNjQ3P+naku+RexMVSuYhLD5rq2ZU8sh130jfLtyPdIdciLpbKJwK/ou3ct0Dn+C/g0qbGhtd8O7I35FbExVL5QtyTNrM6xyFvPAd8sKmxYU2HLTNG7kRcLJX7AN/mrSv/BHqGtcDpTY0NS3w70hVyJeJiqdwPuAf4iG9fapgtQENTY8Njvh3pLLlJTyqWyn2BuwgCTpphwNxiqXyKb0c6Sy5EHA8hfoabLhlInsHAb4ul8ukdtswAmR9OFEvlAm7O7Lm+femFvA6c0tTY8KRvR9oj0yKOBXwrcL5vX3oxG4HjmxobVvh2pC2yPpy4jiBg34wCfhenb2WSzEbiYqn8SdwwIpANHgXe39TY0Naazt7IpIiLpfKxuPXN8pbzVuv8Ejgna9nWmRtOFEvlkbh/VhBw9jibDN5kylQkjk/kfodblDuQTXYAx2YpITVrkfhKgoCzzgDgjmKpPNC3I81kRsTFUnkiEPn2I9AppuDmb2eCTAwniqWycKuzz/DsSqDzGG7Wm/dUp6xE4osIAs4bAm6N163zincRF0vlMYQnD+WVMbgHY3rFu4iBm3AzpwL55AvxWs3e8CriYql8AmFmWt4ZiFvTwhu+I/HVnu0HeoZPFEvl430Z9ybiYql8Mu5Zc4Ha4DvxVabU8RmJQxSuLY4D/smHYS8ijlNfTvZhO5AoV/ow6isShyhcm5xQLJWPS9to6iIulsr1wIlp2w2kxhfSNugjEv+bB5uB9DgzvoGVGqmKuFgqDyAkfNY6/YCL0zSYdiQ+g967Yntv4lNpGktbxBembC/ghwOLpfIxaRlLTcTFUnkccFpa9gLeSW2lpjQj8Tkp2wv45ay0DKUpqg+maCvgn0OKpXIqD7pMRcTxxOlwbbj3kUo0TisSzyAsht0bOTMNI2mJODfLhAZ6lPpiqZz444fTjMSB3kcf4MikjSQu4mKpPBSYnrSdQGY5NmkDaUTiKSnZCWSTxG96pCGuVC6zBDJLTUTiIOLezUFJr20cRBxIg0Sf9hpEHEiDsUl2nqiI45UTi0naCOSCA5LsPOlIPCEFG4Hsk99ITFieKuDIdSSuS7j/QD7ItYj3Tbj/QD7I9XAiiDgAMDTJzsNwIpAGfZPsPGkRJ/oNDOSGfkl2nug3BLckfq9iDJtemlJYub6+sGzrNK3Y887C+kED2JXoh5h1DO2E1Yn1n7SItyXcvxcGs33bZK1eU19Ytnl6YdmuQ7Sm3xhtGrEPO8ZKjAZG+/YxYySqgyDiNiiwZ/c7tX7tdC3fWF9Y+vrhhSZN0MZ969i2fx/Z/oTb6V3hjSQ77/UiHsmWl6cWVr5YX1i69QiteOPgwov77MeWkf15Y7zEBNxdx0D32J5k571CxAPZsf1QrVlzRGHZpiMLy3YcqtX9DtCmYUPYPk5iP2A/3z7WOC8l2XnNiFjs2TNBL704VSs3HFlY+trUwkoVtWHIMLbt34c9YyQmpeVL4G1sSLLzpEX8157ucBhbN08pNK09UksrRxSW756odQNGafPIAewaJzEOGNfTNgPdJtciXrM3O/Vn145JWrNmemH5K/WFZTsma1WfcXpl2BC2H1CQjQCG97CfgWTJr4ibGhs2FEvl7cCgt9eajdfL66dq5Yb6wtJtUwsr7Z1aP2QEW0f1ZfdYiXcB70rSv0Bq5FfEAMPYuvjQwuoh9Vq2ZXph+a5JWjtgtF4dMYid4yUOIOEZToFMsDbJzhMX8aKBn36RFJf5DGSSZ5LsPI2siyUp2Ahkl63A8iQNBBEHkuYZooolaSANES9MwUYguzydtIHkRRxVlpPwwD6QaRYlbSCtTOT/TclOIHsk/kuclojnpWQnkC3WEVWeTdpIEHEgSX6fhpF0RBxVlgHrUrEVyBK/TcNImqvz/CZFWwH/7AIeSMNQmiK+M0VbAf/MJ6psTcNQmiKeT5LZgoGs8eu0DKUnYnfX5u7U7AV88jfgjrSMpb1i5V0p2wv44VdElc1pGUtXxFFlEWEuRW/gR2ka87F28Hc92Aykx1KiSqp3aH2I+DYSzn4NeGVW2gbTF3FU2QHclLrdQBpUgB+mbdTXowi+R0bWpAj0KDcSVbakbdSPiN2Za+o/O4FE2QLc4MOwz4fCfBt43aP9QM9yA1Gl4sOwPxFHlXXAN73ZD/Qkm4EbfRn3/Xiu64FVnn0IdJ/riCo9vtpTZ/Er4qiyHfiSVx8C3WUxnsbCzcgs0UTUzhHVPQyc6NuNQJfZA7yHqPKETyd8DyeamYmbfxrIF9/3LWDIioijyjPAf/h2I9AlXgS+4tsJyIqIHd8iZEXnic/4PJmrJjsijip7gE/iLpoHss0NRJX7fDvRTHZEDBBV1gCX+HYj0C6PkrErStkSMUBUuQf4sW83Aq3yMvBxokqiT0PqKtkTseNS4A++nQi8hT3Av8R3WjNFNkUcVXYBHwX+7NuVwN/5ClHlQd9OtEY2bna0RVR3ELCA8Igu39xEVLnctxNtkc1I3ExUWQGcgcueDfjh58DnfTvRHtkWMUBUeRw4iyBkH8wBzosvf2aWbA8nqonqTgVmA4N9u9JLeBD4UJxOlmmyH4mbiSoPAaeTwAMeA2/jl+REwJAnEQNElfnAabhJ2IFkuAl3LTgXAoY8DSeqieqm4VbZPNC3KzWEASWiyrd8O9JV8iligKhuFG7Ruvf6dqUG2AVcSFT5mW9H9oZ8DSeqiSobgVOBW3y7knNWAifkVcCQ50hcTVT3SZyY9/HtSs74BXCxryzlnqI2RAwQ1U0GfgK827crOWA78HmiSuqr9SRBfocTLYkqLwDvAa7AfUiB1nkaOLZWBAy1FImrieoOxq0wNMOzJ1liM/A14Jas34HrKrUpYoCoTsBFwDXAaM/e+MSAnwJXEVVe9uxLItSuiJuJ6gYDnwOuBIb5dSZ1ngT+najymG9HkqT2RdxMVDccuAq4HBjk2ZukeRi4lqgy17cjadB7RNxMVDcGF5kvpPbmKf8eJ975vh1Jk94n4maiugHAx4DPAMd79qY7bAHuAX5IVPmTZ1+80HtFXE1UdwQuy/osYJRnbzrDbmAu7tER9xFVevVc6yDiaqK6AnAcLpvkDOAQvw69hddxY937gXuIKus9+5MZgojbI6o7BGjACfsYoJii9T3AU7jJ6Q8AjxFVdqZoPzcEEXeFqG4/nJiPBepxU0HHA+/oZs9rcc/3e65qe4Go8lo3++0VBBH3BFHdPsA4nKDHAAOBfi22PsBW3BOGKsBGYAPwUhBr9wgiDuSe2pkAlEMk7Za0qGorJmzvHEnPS1oi6c6q8vMlLY238zvo42xJJunoNupHS7pT0gpJf5T0uKSPxHUzJFXiY10s6UFJ3b8aZGZh87QB29qpE1DoQVsTcTPYhsfvR8V/RwAr4r/D49fD2+hjKO4KyQLg6DZ8fhy4pKpsAjAzfj0DmFNVdx1wdXePLUTiDCGpKOkFSd8D/gSMl3SlpIVx5Lq6qu15kp6Mo9oPJPXpoPuLgZvNbDOAmW2My/8BeMDMXo3rHsBllbfGNbh1pNu6Ln0qsNPM/p5tY2arzOxtT5CVJNyXottJv0HEfhlUNZT477jsEOB2M6uPX0/EXQ2ZDhwl6SRJk4GPA+81s+m4mx/nAkia1cZP/SRgkqRHJS2Q1CzUscCaqnZr47K3IKkeGG9mc9o5nsNxX772OFHSImA18H7g1g7ad0jf7nYQ6BbbYxECLhIDq8xsQVz0gXh7On4/BCfqacBRwEIX0BiEu9qBmV3Uhq2+8b4zcFdSHpE0BTcEaMlbzvYlFXBPSLqgC8eGpJuBE3DR+Zi4+BEz+1BcfxUusndrTeog4uxRfblNwHVm9oPqBpJmAreZ2Ze70O9aYIGZ7QJWSvoLTtRreWvywDhgXot9hwJTgHnxl2Z/YLakD5vZU1XtluBWMwXAzC6TNBJ306Y1ZgO/6sIxtEoYTmSb+4ELJQ0BkDQ2Ppv/H+Ds5jN7SSMkTeigr3uBU+L2I3HDixWxjQ9IGi5pOC7y31+9o5lVzGykmRXNrIg7sWspYICHgIGSLq0qay959wRgeQd+d0iIxBnGzObG49/H4wi4DTjPzJ6X9DVgbvxTvwu4DFglaRZwSysCaxbr87gx9JVmtglA0jXAwrjd183s1bj868BTZja7k/6apDOBGyR9Cbey/Gu4edzNNI+Jhbvp09bwp9OEmx2B3BOGE4HcE0QcyD1BxIHcE0QcyD1BxIHcE0QcyD1BxIHcE0QcyD1BxIHcE0QcyD1BxIHcE0QcyD1BxIHc8/8D/WaxR9U0yAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(126774263808, 66395750400, 60361736192)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my.get_storage_infos()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGES_DIR = \"./images/catalog_images\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of images Catalog files: 466\n"
     ]
    }
   ],
   "source": [
    "files = vec2Text.get_files_in_dir(IMAGES_DIR)\n",
    "print('Total number of images Catalog files:', len(files))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Processing results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "RESULTS_DIR = \"results\"\n",
    "pd.reset_option('display.max_rows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files in directory: results \n",
      "\n",
      "1 \t 2022-10-24 14:48:44.006870 27.1 kB \t barcode_catalog_results.pkl\n",
      "2 \t 2022-10-24 14:48:43.851959 23.8 kB \t barcode_catalog_results.png\n",
      "3 \t 2022-10-24 14:48:34.347439 1.1 MB \t barcode_result1.jpg\n",
      "4 \t 2022-10-24 15:03:32.351564 33.5 kB \t caption_catalog_results.pkl\n",
      "5 \t 2022-10-24 15:03:32.129692 30.1 kB \t colors_catalog_results.pkl\n",
      "6 \t 2022-10-24 15:03:31.974781 198.3 kB \t ocr_catalog_results.pkl\n",
      "7 \t 2022-10-24 15:03:31.695942 25.6 kB \t ocr_catalog_results.png\n",
      "8 \t 2022-10-24 15:03:32.220640 44.0 kB \t tags_catalog_results.pkl\n",
      "9 \t 2022-10-24 14:41:06.619912 1.2 MB \t vectext_catalog_results.pkl\n",
      "10 \t 2022-10-24 14:41:06.400039 23.4 kB \t vectext_catalog_results.png\n"
     ]
    }
   ],
   "source": [
    "my.list_dir(RESULTS_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_vectext = pd.read_pickle(RESULTS_DIR + \"/vectext_catalog_results.pkl\")\n",
    "df_barcode = pd.read_pickle(RESULTS_DIR + \"/barcode_catalog_results.pkl\")\n",
    "df_ocr = pd.read_pickle(RESULTS_DIR + \"/ocr_catalog_results.pkl\")\n",
    "df_colors = pd.read_pickle(RESULTS_DIR + \"/colors_catalog_results.pkl\")\n",
    "df_tags = pd.read_pickle(RESULTS_DIR + \"/tags_catalog_results.pkl\")\n",
    "df_captions = pd.read_pickle(RESULTS_DIR + \"/caption_catalog_results.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import reduce\n",
    "\n",
    "dfs = [df_vectext, df_barcode, df_ocr, df_colors, df_tags, df_captions]\n",
    "\n",
    "df = reduce(lambda left,\n",
    "            right: pd.merge(left,\n",
    "                            right,\n",
    "                            on=['imagefile'],\n",
    "                            how='outer'), \n",
    "            dfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>imagefile</th>\n",
       "      <th>vectext</th>\n",
       "      <th>barcodetext</th>\n",
       "      <th>ocrtext</th>\n",
       "      <th>img_colors</th>\n",
       "      <th>img_tags</th>\n",
       "      <th>img_caption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>./images/catalog_images/catalog_image_00001.jpg</td>\n",
       "      <td>A2 B0 C1 D5 E2 F3 G0 H2 I1 J1 BA1 BB3 BC3 BD4 ...</td>\n",
       "      <td></td>\n",
       "      <td>FABRIQUÉ EN FRANCE La Mère Poulard La Grande G...</td>\n",
       "      <td>Grey White</td>\n",
       "      <td>text fast food snack food</td>\n",
       "      <td>a box of cookies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>./images/catalog_images/catalog_image_00002.jpg</td>\n",
       "      <td>A2 B3 C1 D4 E2 F3 G3 H3 I3 J2 BA1 BB0 BC2 BD0 ...</td>\n",
       "      <td></td>\n",
       "      <td>yaourt Nature MALO</td>\n",
       "      <td>Grey White</td>\n",
       "      <td>text tin</td>\n",
       "      <td>a can of food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>./images/catalog_images/catalog_image_00003.jpg</td>\n",
       "      <td>A1 B3 C3 D3 E3 F0 G1 H3 I4 J2 BA4 BB1 BC1 BD3 ...</td>\n",
       "      <td></td>\n",
       "      <td>EST. 1907 SAXA SEA SALT NO NEED TO GRIND FINE</td>\n",
       "      <td>Grey</td>\n",
       "      <td>tin can soft drink plastic drink lid indoor ti...</td>\n",
       "      <td>a blue and white container</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>./images/catalog_images/catalog_image_00004.jpg</td>\n",
       "      <td>A2 B3 C1 D3 E3 F2 G2 H3 I4 J1 BA3 BB1 BC2 BD2 ...</td>\n",
       "      <td></td>\n",
       "      <td>SAXA SEA SALT FINE 960EQL-6866105</td>\n",
       "      <td>Grey</td>\n",
       "      <td>indoor text tin can tin cup</td>\n",
       "      <td>a blue and white container</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>./images/catalog_images/catalog_image_00005.jpg</td>\n",
       "      <td>A2 B3 C0 D2 E2 F0 G2 H3 I3 J1 BA2 BB0 BC2 BD2 ...</td>\n",
       "      <td></td>\n",
       "      <td>SAXA SEA SALT FINE 5 019989 103096</td>\n",
       "      <td>Grey White</td>\n",
       "      <td>text indoor tin can household supply tin wall ...</td>\n",
       "      <td>a blue can of energy drink</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         imagefile  \\\n",
       "0  ./images/catalog_images/catalog_image_00001.jpg   \n",
       "1  ./images/catalog_images/catalog_image_00002.jpg   \n",
       "2  ./images/catalog_images/catalog_image_00003.jpg   \n",
       "3  ./images/catalog_images/catalog_image_00004.jpg   \n",
       "4  ./images/catalog_images/catalog_image_00005.jpg   \n",
       "\n",
       "                                             vectext barcodetext  \\\n",
       "0  A2 B0 C1 D5 E2 F3 G0 H2 I1 J1 BA1 BB3 BC3 BD4 ...               \n",
       "1  A2 B3 C1 D4 E2 F3 G3 H3 I3 J2 BA1 BB0 BC2 BD0 ...               \n",
       "2  A1 B3 C3 D3 E3 F0 G1 H3 I4 J2 BA4 BB1 BC1 BD3 ...               \n",
       "3  A2 B3 C1 D3 E3 F2 G2 H3 I4 J1 BA3 BB1 BC2 BD2 ...               \n",
       "4  A2 B3 C0 D2 E2 F0 G2 H3 I3 J1 BA2 BB0 BC2 BD2 ...               \n",
       "\n",
       "                                             ocrtext  img_colors  \\\n",
       "0  FABRIQUÉ EN FRANCE La Mère Poulard La Grande G...  Grey White   \n",
       "1                                 yaourt Nature MALO  Grey White   \n",
       "2      EST. 1907 SAXA SEA SALT NO NEED TO GRIND FINE        Grey   \n",
       "3                  SAXA SEA SALT FINE 960EQL-6866105        Grey   \n",
       "4                 SAXA SEA SALT FINE 5 019989 103096  Grey White   \n",
       "\n",
       "                                            img_tags  \\\n",
       "0                          text fast food snack food   \n",
       "1                                           text tin   \n",
       "2  tin can soft drink plastic drink lid indoor ti...   \n",
       "3                        indoor text tin can tin cup   \n",
       "4  text indoor tin can household supply tin wall ...   \n",
       "\n",
       "                  img_caption  \n",
       "0            a box of cookies  \n",
       "1               a can of food  \n",
       "2  a blue and white container  \n",
       "3  a blue and white container  \n",
       "4  a blue can of energy drink  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of empty vectext text = 0\n",
      "Number of empty vectext text in % = 0.0\n",
      "Number of detected vectext text = 466\n",
      "Number of detected vectext text in % = 100.0\n"
     ]
    }
   ],
   "source": [
    "noval = (df['vectext'] == '').sum()\n",
    "nobs = df_ocr.shape[0]\n",
    "withtext = nobs - noval\n",
    "\n",
    "print(\"Number of empty vectext text =\", noval)\n",
    "print(\"Number of empty vectext text in % =\", round(noval / nobs * 100, 2))\n",
    "print(\"Number of detected vectext text =\", withtext)\n",
    "print(\"Number of detected vectext text in % =\", round(withtext / nobs * 100,\n",
    "                                                      2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of empty barcodetext text = 389\n",
      "Number of empty barcodetext text in % = 83.48\n",
      "Number of detected barcodetext text = 77\n",
      "Number of detected barcodetext text in % = 16.52\n"
     ]
    }
   ],
   "source": [
    "noval = (df['barcodetext'] == '').sum()\n",
    "nobs = df_ocr.shape[0]\n",
    "withtext = nobs - noval\n",
    "\n",
    "print(\"Number of empty barcodetext text =\", noval)\n",
    "print(\"Number of empty barcodetext text in % =\", round(noval / nobs * 100, 2))\n",
    "print(\"Number of detected barcodetext text =\", withtext)\n",
    "print(\"Number of detected barcodetext text in % =\",\n",
    "      round(withtext / nobs * 100, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of empty ocrtext text = 31\n",
      "Number of empty ocrtext text in % = 6.65\n",
      "Number of detected ocrtext text = 435\n",
      "Number of detected ocrtext text in % = 93.35\n"
     ]
    }
   ],
   "source": [
    "noval = (df['ocrtext'] == '').sum()\n",
    "nobs = df_ocr.shape[0]\n",
    "withtext = nobs - noval\n",
    "\n",
    "print(\"Number of empty ocrtext text =\", noval)\n",
    "print(\"Number of empty ocrtext text in % =\", round(noval / nobs * 100, 2))\n",
    "print(\"Number of detected ocrtext text =\", withtext)\n",
    "print(\"Number of detected ocrtext text in % =\", round(withtext / nobs * 100,\n",
    "                                                      2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Azure Cognitive Search Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_file = 'azureservices.py'\n",
    "config = configparser.ConfigParser()\n",
    "config.read(config_file)\n",
    "search_key = config.get('AzureCognitiveSearch', 'key')\n",
    "search_endpoint = config.get('AzureCognitiveSearch', 'endpoint')\n",
    "search_servicename = config.get('AzureCognitiveSearch', 'servicename')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your credentials (values have been masked for privacy)\n",
      "Azure Cognitive Search endpoint: https://az************indows.net\n",
      "Azure Cognitive Search key: 0C************C6\n"
     ]
    }
   ],
   "source": [
    "print(\"Your credentials (values have been masked for privacy)\")\n",
    "print(\"Azure Cognitive Search endpoint:\",\n",
    "      search_endpoint[:10] + \"************\" + search_endpoint[-10:])\n",
    "print(\"Azure Cognitive Search key:\",\n",
    "      search_key[:2] + \"************\" + search_key[-2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure Cognitive Search service: azu****\n",
      "Azure Cognitive Search index is: demo-retail\n"
     ]
    }
   ],
   "source": [
    "print(\"Azure Cognitive Search service:\", search_servicename[:3] + '****')\n",
    "print(\"Azure Cognitive Search index is:\", acs.index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-10-24T17:09:40.946269+02:00\n",
      "Deleting index demo-retail\n",
      "demo-retail index has been deleted\n"
     ]
    }
   ],
   "source": [
    "print(my.now())\n",
    "acs.delete_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating new empty index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-10-24T17:09:43.541433+02:00\n",
      "Creating new index demo-retail\n",
      "demo-retail index has been created\n"
     ]
    }
   ],
   "source": [
    "print(my.now())\n",
    "acs.create_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Loading data into the Azure Cognitive Search index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading images to Azure Cognitive Search...\n",
      "\n",
      "Using max_batch_size = 100\n",
      "Number of images files to load: 466 \n",
      "\n",
      "Uploading new document...\n",
      "Upload of new document succeeded: True\n",
      "Done\n",
      "\n",
      "24-10-2022 15:09:52 Elapsed time in min: 0.02 second  : Number of processed images: 100 / 466 | Done: 21 % | To do: 78.5 % \n",
      "\n",
      "Uploading new document...\n",
      "Upload of new document succeeded: True\n",
      "Done\n",
      "\n",
      "24-10-2022 15:09:52 Elapsed time in min: 0.02 second  : Number of processed images: 200 / 466 | Done: 43 % | To do: 57.1 % \n",
      "\n",
      "Uploading new document...\n",
      "Upload of new document succeeded: True\n",
      "Done\n",
      "\n",
      "24-10-2022 15:09:53 Elapsed time in min: 0.03 second  : Number of processed images: 300 / 466 | Done: 64 % | To do: 35.6 % \n",
      "\n",
      "Uploading new document...\n",
      "Upload of new document succeeded: True\n",
      "Done\n",
      "\n",
      "24-10-2022 15:09:53 Elapsed time in min: 0.04 second  : Number of processed images: 400 / 466 | Done: 86 % | To do: 14.2 % \n",
      "\n",
      "Uploading new document...\n",
      "Upload of new document succeeded: True\n",
      "Done\n",
      "\n",
      "\n",
      "Number of processed files = 466\n",
      "Done in 2 seconds\n"
     ]
    }
   ],
   "source": [
    "max_batch_size = 100\n",
    "documents = []\n",
    "idx = 0\n",
    "debugmode = False\n",
    "ocrtext = ''\n",
    "start = my.now()\n",
    "\n",
    "print(\"Uploading images to Azure Cognitive Search...\")\n",
    "print(\"\\nUsing max_batch_size =\", max_batch_size)\n",
    "nbimagesfiles = len(files)\n",
    "print('Number of images files to load:', nbimagesfiles, \"\\n\")\n",
    "\n",
    "for file in files:  # Reading image by image (fullpath image)\n",
    "    file = df['imagefile'].iloc[idx]\n",
    "    vecText = df['vectext'].iloc[idx]\n",
    "    ocrtext = df['ocrtext'].iloc[idx]\n",
    "    barcodetext = df['barcodetext'].iloc[idx]\n",
    "    colors = df['img_colors'].iloc[idx]\n",
    "    tags = df['img_tags'].iloc[idx]\n",
    "    caption = df['img_caption'].iloc[idx]\n",
    "\n",
    "    if debugmode:\n",
    "        print(file)\n",
    "        print(vecText)\n",
    "        print(ocrtext)\n",
    "        print(barcodetext)\n",
    "        print(colors)\n",
    "        print(tags)\n",
    "        print(caption)\n",
    "    \n",
    "    # Adding all fields to the documents\n",
    "    documents.append({\n",
    "        \"@search.action\": \"upload\",\n",
    "        \"Id\": vec2Text.string_to_base64(file),  # Image file id\n",
    "        \"Content\": file,  # filename\n",
    "        \"VecText\": vecText,  # vec2text results\n",
    "        \"Ocrtext\":\n",
    "        ocrtext,  # get text from image using Azure OCR (if there are some text, otherwise blank)\n",
    "        \"Barcodetext\":\n",
    "        barcodetext,  # Get text from barcode (if there is one, otherwise blank)\n",
    "        \"Colors\": colors,  # Get Azure CV colors detection\n",
    "        \"Tags\": tags,  # Get Azure CV tags\n",
    "        \"Caption\": caption,  # Get Azure CV caption\n",
    "    })\n",
    "\n",
    "    idx += 1\n",
    "\n",
    "    if len(documents) == max_batch_size:\n",
    "        if debugmode:\n",
    "            print(documents)\n",
    "\n",
    "        acs.upload_documents(documents)\n",
    "        documents = []\n",
    "        pct_done = (idx / nbimagesfiles) * 100\n",
    "        pct_todo = 100 - pct_done\n",
    "        print(my.get_today(), 'Elapsed time in min:', (my.now() - start) / 60,\n",
    "              ' : Number of processed images:', idx, \"/\",\n",
    "              nbimagesfiles, '| Done:', round(pct_done), \"%\", '| To do:',\n",
    "              round(pct_todo, 1), '%', \"\\n\")\n",
    "\n",
    "if len(documents) > 0:\n",
    "    acs.upload_documents(documents)\n",
    "    print('\\nNumber of processed files =', idx)\n",
    "\n",
    "print(\"Done in\", (my.now() - start).in_words(locale='en'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Azure Cognitive Search Index status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure Cognitive Search index name: demo-retail\n"
     ]
    }
   ],
   "source": [
    "print(\"Azure Cognitive Search index name:\", acs.index_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure Cognitive Search Index: demo-retail \n",
      "\n",
      "{\n",
      "    \"@odata.context\": \"https://azurecognitivesearchsr.search.windows.net/$metadata#indexes/$entity\",\n",
      "    \"@odata.etag\": \"\\\"0x8DAB5D1C886D578\\\"\",\n",
      "    \"name\": \"demo-retail\",\n",
      "    \"defaultScoringProfile\": null,\n",
      "    \"fields\": [\n",
      "        {\n",
      "            \"name\": \"Id\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": false,\n",
      "            \"filterable\": false,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": false,\n",
      "            \"facetable\": false,\n",
      "            \"key\": true,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Content\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": \"en.microsoft\",\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"VecText\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": false,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": false,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Ocrtext\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Barcodetext\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Colors\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Tags\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        },\n",
      "        {\n",
      "            \"name\": \"Caption\",\n",
      "            \"type\": \"Edm.String\",\n",
      "            \"searchable\": true,\n",
      "            \"filterable\": true,\n",
      "            \"retrievable\": true,\n",
      "            \"sortable\": true,\n",
      "            \"facetable\": false,\n",
      "            \"key\": false,\n",
      "            \"indexAnalyzer\": null,\n",
      "            \"searchAnalyzer\": null,\n",
      "            \"analyzer\": null,\n",
      "            \"synonymMaps\": []\n",
      "        }\n",
      "    ],\n",
      "    \"scoringProfiles\": [],\n",
      "    \"corsOptions\": {\n",
      "        \"allowedOrigins\": [\n",
      "            \"*\"\n",
      "        ],\n",
      "        \"maxAgeInSeconds\": 60\n",
      "    },\n",
      "    \"suggesters\": [],\n",
      "    \"analyzers\": [],\n",
      "    \"tokenizers\": [],\n",
      "    \"tokenFilters\": [],\n",
      "    \"charFilters\": [],\n",
      "    \"encryptionKey\": null,\n",
      "    \"similarity\": {\n",
      "        \"@odata.type\": \"#Microsoft.Azure.Search.BM25Similarity\",\n",
      "        \"k1\": null,\n",
      "        \"b\": null\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "acs.index_status(acs.index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> End of notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}