ai-gateway/how_to_setup_fallback_from_openai_to_azure_openai.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/how_to_setup_fallback_from_openai_to_azure_openai.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pln7ewgvA3Gs"
      },
      "source": [
        "## How to Setup Fallback from OpenAI to Azure OpenAI\n",
        "\n",
        "Let’s say you’ve built an LLM-based app and deployed it to production. It relies on OpenAI’s gpt-4 model. It’s [Mar 12, 2023](https://status.portkey.ai/incident/339664), and suddenly your users find errors with the functionality of the app — “It doesn’t work!”\n",
        "\n",
        "It turns out that in the logs, the app has encountered [503 errors](https://platform.openai.com/docs/guides/error-codes) due to overloaded requests on the server-side. What could you do? If you are in such a situation, we have an answer for you: Portkey Fallbacks.\n",
        "\n",
        "Portkey Fallbacks can automatically switch your app's requests from one LLM provider to another, ensuring reliability by allowing you to fallback among multiple LLMs. This is especially useful given the unpredictable nature of LLM APIs. With Portkey, you can switch to a different LLM provider, such as Azure, when needed, making your app Production-Ready.\n",
        "\n",
        "In this cookbook, we will learn how to implement a fallback mechanism in our apps that allows us to automatically switch the LLM provider from OpenAI to Azure OpenAI with just a few lines of code. Both providers have the exact same set of models, but they are deployed differently. Azure OpenAI comes with its own deployment mechanisms, which are generally considered to be more reliable.\n",
        "\n",
        "<span style=\"text-decoration:underline;\">Prerequisites:</span>\n",
        "\n",
        "1. You have the [Portkey API Key](https://portkey.ai/docs/api-reference/authentication#obtaining-your-api-key). [ [Sign Up](https://portkey.ai) ]\n",
        "2. You stored OpenAI and Azure OpenAI API keys as [virtual keys](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "leMVtjKrBce2"
      },
      "source": [
        "## 1. Import the SDK and authenticate with Portkey\n",
        "\n",
        "We start by importing Portkey SDK into our NodeJS project using npm and authenticate by passing the Portkey API Key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vOostjRiBfq_"
      },
      "outputs": [],
      "source": [
        "!pip install portkey-ai openai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "b_985bXCCGGZ"
      },
      "outputs": [],
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cKbbE0t9DKV4"
      },
      "source": [
        "## 2. Create Fallback Configs\n",
        "\n",
        "Next, we will create a configs object that influences the behavior of the request sent using Portkey.\n",
        "\n",
        "```json\n",
        "{\n",
        " strategy: {\n",
        "    mode: \"fallback\",\n",
        "  },\n",
        "  targets: [\n",
        "    {\n",
        "      virtual_key: OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "    {\n",
        "      virtual_key: AZURE_OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "  ],\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFjHxpzUCbUJ"
      },
      "source": [
        "This configuration instructs Portkey to use a \\_fallback \\_strategy with the requests. The \\_targets_array lists the virtual keys of LLMs in the order Portkey should fallback to an alternative.\n",
        "\n",
        "Most users find it way more cleaner to define the configs in the Portkey UI and reference the config ID in the code. [Try it out](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs#creating-configs).\n",
        "\n",
        "Add this configuration to the _portkey_ instance to apply the fallback behavior to all the requests."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "t_-VBo6ZDX2v"
      },
      "outputs": [],
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "import json\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "AZURE_OPENAI_VIRTUAL_KEY=userdata.get('AZURE_OPENAI_VIRTUAL_KEY')\n",
        "\n",
        "config_data = {\n",
        " 'strategy': {\n",
        "    'mode': \"fallback\",\n",
        "  },\n",
        "  'targets': [\n",
        "    {\n",
        "      'virtual_key': OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "    {\n",
        "      'virtual_key': AZURE_OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "  ],\n",
        "}\n",
        "\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        "    virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "    config=json.dumps(config_data)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lBvXjINhEbxg"
      },
      "source": [
        "Always reference the credentials from the environment variables to prevent exposure of any sensitive data. Portkey will automatically infer the LLM providers based on the passed virtual keys.\n",
        "\n",
        "> The Azure OpenAI virtual key only needs to be set up once, and it will then be accessible through Portkey in all subsequent API calls."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<details>\n",
        "\n",
        "<summary>Fallback Configs without virtual keys</summary>\n",
        "\n",
        "```json\n",
        "{\n",
        "  \"strategy\": {\n",
        "    \"mode\": \"fallback\"\n",
        "  },\n",
        "  \"targets\": [\n",
        "    {\n",
        "      \"provider\": \"openai\",\n",
        "      \"api_key\": \"sk-xxxxxxxxpRT4xxxx5\"\n",
        "    },\n",
        "    {\n",
        "      \"provider\": \"azure-openai\",\n",
        "      \"api_key\": \"*******\"\n",
        "    }\n",
        "  ]\n",
        "}\n",
        "```\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQ5JMtO3EeFu"
      },
      "source": [
        "## 3. Make a request\n",
        "\n",
        "All the requests will hit OpenAI since Portkey proxies all those requests to the target(s) we already specified. Notice that the changes to the requests do not demand any code changes in the business logic implementation. Smooth!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c8gYcOg3MOKq"
      },
      "outputs": [],
      "source": [
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "response = portkey.chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-3.5-turbo'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content) # Here is the plan"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gehAN1uiMd2e"
      },
      "source": [
        "When OpenAI returns any 4xx or 5xx errors, Portkey will automatically switch to Azure OpenAI to ensure the same specified model is used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjDa1SXDMiGH"
      },
      "source": [
        "## 4. View the Fallback Status in Logs\n",
        "\n",
        "Since all the requests go through Portkey, Portkey can log them for better observability of your app. You can find the specific requests by passing an _trace ID_. It can be any desired string name. In this case, `my-trace-id`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Lgg6X2VDMlZF"
      },
      "outputs": [],
      "source": [
        "response = portkey.with_options(trace_id=\"<my-trace-id>\").chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-4'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_sY3uQbNH4W"
      },
      "source": [
        "You can apply filter with Trace ID to list requests. Instances when the fallbacks are activated will highlight the fallback icon. The logs can be filtered by cost, tokens, status, config, trace id and so on.\n",
        "\n",
        "![](https://raw.githubusercontent.com/Saif-Shines/pk-cookbook/next/ai-gateway/images/1.png?raw=true)\n",
        "\n",
        "Learn more about [Logs](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/logs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R2KpXJfcO80T"
      },
      "source": [
        "## 5. Advanced: Fallback on Specific Status Codes\n",
        "\n",
        "Portkey provides finer control over the when it should apply fallback strategy to your requests to LLMs. You can define the configuration to condition based on specific status codes returned by the LLM provider."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xD1AYPf1PDGQ"
      },
      "outputs": [],
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "import json\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "AZURE_OPENAI_VIRTUAL_KEY=userdata.get('AZURE_OPENAI_VIRTUAL_KEY')\n",
        "\n",
        "config_data = {\n",
        " 'strategy': {\n",
        "    'mode': \"fallback\",\n",
        "    'on_status_codes': [429]\n",
        "  },\n",
        "  'targets': [\n",
        "    {\n",
        "      'virtual_key': OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "    {\n",
        "      'virtual_key': AZURE_OPENAI_VIRTUAL_KEY,\n",
        "    },\n",
        "  ],\n",
        "}\n",
        "\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        "    virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "    config=json.dumps(config_data)\n",
        ")\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "response = portkey.chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-3.5-turbo'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content) # Here is the plan"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znyPLa3uPT8L"
      },
      "source": [
        "In the above case for all the request that are acknowledged with the status code of 429 will fallback from OpenAI to Azure OpenAI."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "itZYQAdwPU0b"
      },
      "source": [
        "## 6. Considerations\n",
        "\n",
        "That’s it; you can implement production-grade fallback mechanisms with just a few lines of code. While you are equipped with all the tools to implement fallbacks to your next GenAI app, here are few considerations:\n",
        "\n",
        "- The implementation of Fallback does not alter the quality of LLM outputs received by your app.\n",
        "- Azure requires you to deploy specific models. Portkey will automatically trigger the chat completions endpoint using GPT4 if it is available instead of GPT3.5."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyNTD3h7w6NbWJdNj5scDoZ7",
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}