python-recipes/RAG/04_advanced_redisvl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R2-i8jBl9GRH"
   },
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "\n",
    "# Advanced RAG example\n",
    "\n",
    "Now that you have a good foundation in Redis data structures, search capabilities, and basic RAG with the redisvl client from [/getting_started/02_redisvl](../getting_started/02_redisvl.ipynb).\n",
    "\n",
    "We will extend the basic RAG example with a few special topics/techniques:\n",
    "- Dense content representation\n",
    "- Query rewriting / expansion\n",
    "- Semantic caching\n",
    "- Conversational memory persistence\n",
    "\n",
    "## Let's Begin!\n",
    "<a href=\"https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/04_advanced_redisvl.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improve accuracy with dense content representations\n",
    "In the basic example, we took raw chunks of text from our pdf documents and generated embeddings for them to be stored in the vector database. This is okay but one technique we can use to improve the quality of retrieval is to leverage an LLM from OpenAI during ETL. We will prompt the LLM to summarize and decompose the raw pdf text into more discrete propositional phrases. This will enhance the clarity of the text and improve semantic retrieval for RAG.\n",
    "\n",
    "The goal is to utilize a preprocessing technique similar to what's outlined here:\n",
    "https://github.com/langchain-ai/langchain/blob/master/templates/propositional-retrieval/propositional_retrieval/proposal_chain.py\n",
    "\n",
    "If you already have a redis-stack instance running locally from before feel free to jump ahead but if not execute the following commands to get the environment properly setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rT9HzsnQ1uiz"
   },
   "source": [
    "## Environment Setup\n",
    "\n",
    "### Pull Github Materials\n",
    "Because you are likely running this notebook in **Google Colab**, we need to first\n",
    "pull the necessary dataset and materials directly from GitHub.\n",
    "\n",
    "**If you are running this notebook locally**, FYI you may not need to perform this\n",
    "step at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AJJ2UW6M1ui0"
   },
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n",
    "!mv temp_repo/python-recipes/RAG/resources .\n",
    "!rm -rf temp_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z67mf6T91ui2"
   },
   "source": [
    "### Install Python Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "DgxBQFXQ1ui2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "!pip install -q redis redisvl pandas \"unstructured[pdf]\" sentence-transformers langchain langchain-community \"openai>=1.57.0\" tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Redis Stack\n",
    "\n",
    "Later in this tutorial, Redis will be used to store, index, and query vector\n",
    "embeddings created from PDF document chunks. **We need to make sure we have a Redis\n",
    "instance available.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Colab\n",
    "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly\n",
    "from the Redis package archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "%%sh\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update  > /dev/null 2>&1\n",
    "sudo apt-get install redis-stack-server  > /dev/null 2>&1\n",
    "redis-stack-server --daemonize yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Alternative Environments\n",
    "There are many ways to get the necessary redis-stack instance running\n",
    "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
    "own version of Redis Enterprise running, that works too!\n",
    "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
    "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Redis Connection URL\n",
    "\n",
    "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "import nest_asyncio\n",
    "# Apply the nest_asyncio patch: let's us run async code in Jupyter\n",
    "nest_asyncio.apply()\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Replace values below with your own if using Redis Cloud instance\n",
    "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
    "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\")      # ex: 18374\n",
    "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")  # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
    "\n",
    "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
    "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now that our environment is setup we can again load our financial documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KrtWWU4I1ui3"
   },
   "source": [
    "### Dataset Preparation (PDF Documents)\n",
    "\n",
    "To best demonstrate Redis as a vector database layer, we will load a single\n",
    "financial (10k filings) doc and preprocess it using some helpers from LangChain:\n",
    "\n",
    "- `PyPDFLoader` is not the only document loader type that LangChain provides. Docs: https://python.langchain.com/docs/integrations/document_loaders/unstructured_file\n",
    "- `RecursiveCharacterTextSplitter` is what we use to create smaller chunks of text from the doc. Docs: https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "uijl2qFH1ui3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done preprocessing. Created 210 chunks of the original pdf resources/nke-10k-2023.pdf\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "# pdf to load\n",
    "path = 'resources/nke-10k-2023.pdf'\n",
    "assert os.path.exists(path), f\"File not found: {path}\"\n",
    "\n",
    "# load and split\n",
    "loader = PyPDFLoader(path)\n",
    "pages = loader.load()\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=2500, chunk_overlap=0)\n",
    "chunks = text_splitter.split_documents(pages)\n",
    "\n",
    "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': 'resources/nke-10k-2023.pdf', 'page': 0, 'page_label': '1'}, page_content=\"Table of Contents\\nUNITED STATES\\nSECURITIES AND EXCHANGE COMMISSION\\nWashington, D.C. 20549\\nFORM 10-K\\n(Mark One)\\n☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE FISCAL YEAR ENDED MAY 31, 2023\\nOR\\n☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE TRANSITION PERIOD FROM                         TO                         .\\nCommission File No. 1-10635\\nNIKE, Inc.\\n(Exact name of Registrant as specified in its charter)\\nOregon 93-0584541\\n(State or other jurisdiction of incorporation) (IRS Employer Identification No.)\\nOne Bowerman Drive, Beaverton, Oregon 97005-6453\\n(Address of principal executive offices and zip code)\\n(503) 671-6453\\n(Registrant's telephone number, including area code)\\nSECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:\\nClass B Common Stock NKE New York Stock Exchange\\n(Title of each class) (Trading symbol) (Name of each exchange on which registered)\\nSECURITIES REGISTERED PURSUANT TO SECTION 12(G) OF THE ACT:\\nNONE\\nIndicate by check mark: YES NO\\n•if the registrant is a well-known seasoned issuer, as defined in Rule 405 of the Securities Act. þ ¨\\n•if the registrant is not required to file reports pursuant to Section 13 or Section 15(d) of the Act. ¨ þ\\n•whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding\\n12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the\\npast 90 days.þ ¨\\n•whether the registrant has submitted electronically every Interactive Data File required to be submitted pursuant to Rule 405 of Regulation S-T\\n(§232.405 of this chapter) during the preceding 12 months (or for such shorter period that the registrant was required to submit such files).þ ¨\\n•whether the registrant is a large accelerated filer, an accelerated filer, a non-accelerated filer, a smaller reporting company or an emerging growth company. See the definitions of “large accelerated filer,”\\n“accelerated filer,” “smaller reporting company,” and “emerging growth company” in Rule 12b-2 of the Exchange Act.\\nLarge accelerated filer þ Accelerated filer ☐ Non-accelerated filer ☐ Smaller reporting company ☐ Emerging growth company ☐\\n•if an emerging growth company, if the registrant has elected not to use the extended transition period for complying with any new or revised financial\")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the previous example, we would have gone ahead and embed the chunks as extracted here.\n",
    "\n",
    "Now we will instead leverage an LLM to create dense content representations to improve our retrieval accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup OpenAI as LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "import openai\n",
    "\n",
    "CHAT_MODEL = \"gpt-3.5-turbo-0125\"\n",
    "\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tqdm\n",
    "import json\n",
    "\n",
    "\n",
    "def create_dense_props(chunk):\n",
    "    \"\"\"Create dense representation of raw text content.\"\"\"\n",
    "\n",
    "    # The system message here should be HEAVILY customized for your specific use case\n",
    "    SYSTEM_PROMPT = \"\"\"\n",
    "    You are a helpful PDF extractor tool. You will be presented with segments from\n",
    "    raw PDF documents composed of 10k SEC filings information about public companies.\n",
    "\n",
    "    Decompose and summarize the raw content into clear and simple propositions,\n",
    "    ensuring they are interpretable out of context. Consider the following rules:\n",
    "    1. Split compound sentences into simpler dense phrases that retain existing\n",
    "    meaning.\n",
    "    2. Simplify technical jargon or wording if possible while retaining existing\n",
    "    meaning.\n",
    "    2. For any named entity that is accompanied by additional descriptive information,\n",
    "    separate this information into its own distinct proposition.\n",
    "    3. Decontextualize the proposition by adding necessary modifier to nouns or\n",
    "    entire sentences and replacing pronouns (e.g., \"it\", \"he\", \"she\", \"they\", \"this\", \"that\")\n",
    "    with the full name of the entities they refer to.\n",
    "    4. Present the results as a list of strings, formatted in JSON, under the key \"propositions\".\n",
    "    \"\"\"\n",
    "\n",
    "    response = openai.OpenAI().chat.completions.create(\n",
    "        model=CHAT_MODEL,\n",
    "        response_format={ \"type\": \"json_object\" },\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
    "            {\"role\": \"user\", \"content\": f\"Decompose this raw content using the rules above:\\n{chunk.page_content} \"}\n",
    "        ]\n",
    "    )\n",
    "    res = response.choices[0].message.content\n",
    "\n",
    "    try:\n",
    "        return json.loads(res)[\"propositions\"]\n",
    "    except Exception as e:\n",
    "        print(f\"Failed to parse propositions\", str(e), flush=True)\n",
    "        # Retry\n",
    "        return create_dense_props(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create text propositions using OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/210 [00:00<?, ?it/s]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  0%|          | 1/210 [00:04<15:22,  4.42s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  1%|          | 2/210 [00:07<12:16,  3.54s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  1%|▏         | 3/210 [00:09<09:45,  2.83s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  2%|▏         | 4/210 [00:12<10:35,  3.08s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  2%|▏         | 5/210 [00:16<11:05,  3.25s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  3%|▎         | 6/210 [00:19<11:08,  3.28s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  3%|▎         | 7/210 [00:28<16:58,  5.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  4%|▍         | 8/210 [00:30<13:44,  4.08s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  4%|▍         | 9/210 [00:33<12:31,  3.74s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  5%|▍         | 10/210 [00:36<11:57,  3.59s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  5%|▌         | 11/210 [00:40<12:19,  3.72s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  6%|▌         | 12/210 [00:42<10:06,  3.06s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  6%|▌         | 13/210 [00:43<08:13,  2.51s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  7%|▋         | 14/210 [00:45<08:02,  2.46s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  7%|▋         | 15/210 [00:48<08:21,  2.57s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  8%|▊         | 16/210 [00:50<07:43,  2.39s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  8%|▊         | 17/210 [00:56<11:12,  3.49s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  9%|▊         | 18/210 [00:57<09:03,  2.83s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "  9%|▉         | 19/210 [00:59<08:09,  2.56s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 10%|▉         | 20/210 [01:07<12:48,  4.05s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 10%|█         | 21/210 [01:10<12:04,  3.84s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 10%|█         | 22/210 [01:14<11:35,  3.70s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 11%|█         | 23/210 [01:15<09:37,  3.09s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 11%|█▏        | 24/210 [01:18<09:01,  2.91s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 12%|█▏        | 25/210 [01:19<07:51,  2.55s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 12%|█▏        | 26/210 [01:22<07:35,  2.47s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 13%|█▎        | 27/210 [01:24<07:14,  2.37s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 13%|█▎        | 28/210 [01:26<07:15,  2.39s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 14%|█▍        | 29/210 [01:30<08:51,  2.94s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 14%|█▍        | 30/210 [01:32<07:51,  2.62s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 15%|█▍        | 31/210 [01:35<07:24,  2.48s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 15%|█▌        | 32/210 [01:37<07:00,  2.36s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 16%|█▌        | 33/210 [01:40<07:38,  2.59s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 16%|█▌        | 34/210 [01:45<10:10,  3.47s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 17%|█▋        | 35/210 [01:48<09:43,  3.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 17%|█▋        | 36/210 [01:53<10:53,  3.76s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 18%|█▊        | 37/210 [01:56<10:29,  3.64s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 18%|█▊        | 38/210 [01:58<08:57,  3.12s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 19%|█▊        | 39/210 [02:00<07:40,  2.70s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 19%|█▉        | 40/210 [02:03<08:04,  2.85s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 20%|█▉        | 41/210 [02:09<10:28,  3.72s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 20%|██        | 42/210 [02:11<09:09,  3.27s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 20%|██        | 43/210 [02:13<07:34,  2.72s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 21%|██        | 44/210 [02:15<07:19,  2.65s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 21%|██▏       | 45/210 [02:21<09:48,  3.57s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 22%|██▏       | 46/210 [02:23<08:43,  3.19s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 22%|██▏       | 47/210 [02:25<07:37,  2.80s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 23%|██▎       | 48/210 [02:28<07:57,  2.95s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 23%|██▎       | 49/210 [02:30<07:00,  2.61s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 24%|██▍       | 50/210 [02:34<08:09,  3.06s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 24%|██▍       | 51/210 [02:37<07:51,  2.97s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 25%|██▍       | 52/210 [02:40<07:40,  2.92s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 25%|██▌       | 53/210 [02:41<06:32,  2.50s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 26%|██▌       | 54/210 [02:44<06:19,  2.43s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 26%|██▌       | 55/210 [02:46<06:28,  2.51s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 27%|██▋       | 56/210 [02:49<06:45,  2.63s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 27%|██▋       | 57/210 [02:51<05:56,  2.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 28%|██▊       | 58/210 [02:53<05:26,  2.15s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 28%|██▊       | 59/210 [02:54<05:07,  2.03s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 29%|██▊       | 60/210 [02:57<05:30,  2.20s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 29%|██▉       | 61/210 [02:59<05:26,  2.19s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 30%|██▉       | 62/210 [03:01<05:09,  2.09s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 30%|███       | 63/210 [03:05<06:30,  2.65s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 30%|███       | 64/210 [03:06<05:39,  2.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 31%|███       | 65/210 [03:09<05:25,  2.24s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 31%|███▏      | 66/210 [03:11<05:33,  2.32s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 32%|███▏      | 67/210 [03:13<05:33,  2.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 32%|███▏      | 68/210 [03:14<04:37,  1.96s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 33%|███▎      | 69/210 [03:19<06:23,  2.72s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 33%|███▎      | 70/210 [03:21<05:54,  2.53s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 34%|███▍      | 71/210 [03:24<06:07,  2.65s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 34%|███▍      | 72/210 [03:25<05:00,  2.18s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 35%|███▍      | 73/210 [03:28<05:49,  2.55s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 35%|███▌      | 74/210 [03:31<05:47,  2.56s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 36%|███▌      | 75/210 [03:34<05:49,  2.59s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 36%|███▌      | 76/210 [03:37<06:06,  2.74s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 37%|███▋      | 77/210 [03:39<05:26,  2.45s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 37%|███▋      | 78/210 [03:40<04:40,  2.12s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 38%|███▊      | 79/210 [03:45<06:25,  2.94s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 38%|███▊      | 80/210 [03:48<06:45,  3.12s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 39%|███▊      | 81/210 [03:50<05:49,  2.71s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 39%|███▉      | 82/210 [03:53<06:07,  2.87s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 40%|███▉      | 83/210 [03:56<05:59,  2.83s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 40%|████      | 84/210 [03:57<04:49,  2.29s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 40%|████      | 85/210 [04:00<04:56,  2.38s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 41%|████      | 86/210 [04:01<04:19,  2.09s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 41%|████▏     | 87/210 [04:03<04:25,  2.15s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 42%|████▏     | 88/210 [04:12<08:35,  4.22s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 42%|████▏     | 89/210 [04:16<08:05,  4.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 43%|████▎     | 90/210 [04:17<06:27,  3.23s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 43%|████▎     | 91/210 [04:22<07:25,  3.75s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 44%|████▍     | 92/210 [04:24<05:52,  2.98s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 44%|████▍     | 93/210 [04:26<05:41,  2.92s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 45%|████▍     | 94/210 [04:28<04:54,  2.54s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 45%|████▌     | 95/210 [04:33<06:20,  3.31s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 46%|████▌     | 96/210 [04:36<06:10,  3.25s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 46%|████▌     | 97/210 [04:42<07:19,  3.89s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 47%|████▋     | 98/210 [04:44<06:38,  3.56s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 47%|████▋     | 99/210 [04:48<06:25,  3.47s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 48%|████▊     | 100/210 [04:50<05:46,  3.15s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 48%|████▊     | 101/210 [04:53<05:29,  3.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 49%|████▊     | 102/210 [04:55<05:07,  2.85s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 49%|████▉     | 103/210 [04:58<04:48,  2.70s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 50%|████▉     | 104/210 [05:02<05:37,  3.18s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 50%|█████     | 105/210 [05:03<04:43,  2.70s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 50%|█████     | 106/210 [05:07<05:15,  3.03s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 51%|█████     | 107/210 [05:11<05:42,  3.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 51%|█████▏    | 108/210 [05:13<05:01,  2.95s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 52%|█████▏    | 109/210 [05:16<04:42,  2.80s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 52%|█████▏    | 110/210 [05:19<04:51,  2.92s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 53%|█████▎    | 111/210 [05:20<03:58,  2.41s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 53%|█████▎    | 112/210 [05:22<03:40,  2.25s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 54%|█████▍    | 113/210 [05:24<03:43,  2.30s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 54%|█████▍    | 114/210 [05:26<03:14,  2.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 55%|█████▍    | 115/210 [05:29<03:44,  2.36s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 55%|█████▌    | 116/210 [05:31<03:34,  2.28s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 56%|█████▌    | 117/210 [05:34<03:49,  2.47s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 56%|█████▌    | 118/210 [05:40<05:21,  3.49s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 57%|█████▋    | 119/210 [05:42<04:39,  3.07s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 57%|█████▋    | 120/210 [05:43<03:55,  2.62s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 58%|█████▊    | 121/210 [05:47<04:15,  2.87s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 58%|█████▊    | 122/210 [05:49<03:58,  2.71s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 59%|█████▊    | 123/210 [05:51<03:21,  2.32s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 59%|█████▉    | 124/210 [05:53<03:31,  2.46s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 60%|█████▉    | 125/210 [05:56<03:31,  2.49s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 60%|██████    | 126/210 [05:58<03:23,  2.43s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 60%|██████    | 127/210 [06:01<03:23,  2.45s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 61%|██████    | 128/210 [06:03<03:16,  2.39s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 61%|██████▏   | 129/210 [06:08<04:09,  3.08s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 62%|██████▏   | 130/210 [06:11<04:07,  3.10s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 62%|██████▏   | 131/210 [06:15<04:30,  3.43s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 63%|██████▎   | 132/210 [06:19<04:29,  3.46s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 63%|██████▎   | 133/210 [06:22<04:27,  3.48s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 64%|██████▍   | 134/210 [06:26<04:25,  3.50s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 64%|██████▍   | 135/210 [06:28<04:05,  3.28s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 65%|██████▍   | 136/210 [06:32<04:10,  3.39s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 65%|██████▌   | 137/210 [06:33<03:16,  2.70s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 66%|██████▌   | 138/210 [06:36<03:21,  2.80s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 66%|██████▌   | 139/210 [06:42<04:15,  3.60s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 67%|██████▋   | 140/210 [06:45<04:02,  3.46s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 67%|██████▋   | 141/210 [06:49<04:20,  3.77s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 68%|██████▊   | 142/210 [06:53<04:16,  3.77s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 68%|██████▊   | 143/210 [06:57<04:11,  3.75s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 69%|██████▊   | 144/210 [06:58<03:22,  3.07s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 69%|██████▉   | 145/210 [07:04<04:08,  3.82s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 70%|██████▉   | 146/210 [07:08<04:01,  3.77s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 70%|███████   | 147/210 [07:12<04:10,  3.97s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 70%|███████   | 148/210 [07:19<05:11,  5.03s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 71%|███████   | 149/210 [07:22<04:26,  4.38s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 71%|███████▏  | 150/210 [07:26<04:01,  4.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 72%|███████▏  | 151/210 [07:29<03:46,  3.83s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 72%|███████▏  | 152/210 [07:31<03:05,  3.20s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 73%|███████▎  | 153/210 [07:35<03:29,  3.68s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 73%|███████▎  | 154/210 [07:37<02:49,  3.02s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 74%|███████▍  | 155/210 [07:43<03:29,  3.80s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 74%|███████▍  | 156/210 [07:45<03:03,  3.40s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 75%|███████▍  | 157/210 [07:48<02:56,  3.33s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 75%|███████▌  | 158/210 [07:49<02:15,  2.61s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 76%|███████▌  | 159/210 [07:53<02:31,  2.96s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 76%|███████▌  | 160/210 [07:54<02:06,  2.52s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 77%|███████▋  | 161/210 [07:58<02:24,  2.94s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 77%|███████▋  | 162/210 [08:03<02:50,  3.55s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 78%|███████▊  | 163/210 [08:05<02:24,  3.07s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 78%|███████▊  | 164/210 [08:08<02:11,  2.86s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 79%|███████▊  | 165/210 [08:11<02:16,  3.03s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 79%|███████▉  | 166/210 [08:17<02:51,  3.91s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 80%|███████▉  | 167/210 [08:20<02:43,  3.80s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 80%|████████  | 168/210 [08:30<03:50,  5.48s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 80%|████████  | 169/210 [08:34<03:21,  4.92s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 81%|████████  | 170/210 [08:36<02:45,  4.14s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 81%|████████▏ | 171/210 [08:37<02:07,  3.26s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 82%|████████▏ | 172/210 [08:41<02:15,  3.57s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 82%|████████▏ | 173/210 [08:44<01:58,  3.19s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 83%|████████▎ | 174/210 [08:48<02:02,  3.40s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 83%|████████▎ | 175/210 [08:51<01:58,  3.40s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 84%|████████▍ | 176/210 [08:52<01:35,  2.82s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 84%|████████▍ | 177/210 [08:57<01:50,  3.35s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 85%|████████▍ | 178/210 [09:00<01:45,  3.29s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 85%|████████▌ | 179/210 [09:03<01:34,  3.03s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 86%|████████▌ | 180/210 [09:06<01:32,  3.08s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 86%|████████▌ | 181/210 [09:09<01:32,  3.21s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 87%|████████▋ | 182/210 [09:14<01:46,  3.81s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 87%|████████▋ | 183/210 [09:17<01:28,  3.28s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 88%|████████▊ | 184/210 [09:23<01:51,  4.30s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 88%|████████▊ | 185/210 [09:27<01:43,  4.15s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 89%|████████▊ | 186/210 [09:32<01:49,  4.55s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 89%|████████▉ | 187/210 [09:36<01:36,  4.21s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 90%|████████▉ | 188/210 [09:40<01:29,  4.06s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 90%|█████████ | 189/210 [09:41<01:08,  3.28s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 90%|█████████ | 190/210 [09:45<01:08,  3.44s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 91%|█████████ | 191/210 [09:46<00:52,  2.77s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 91%|█████████▏| 192/210 [09:49<00:47,  2.66s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 92%|█████████▏| 193/210 [09:50<00:40,  2.38s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 92%|█████████▏| 194/210 [09:54<00:44,  2.77s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 93%|█████████▎| 195/210 [10:01<01:02,  4.15s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 93%|█████████▎| 196/210 [10:06<00:58,  4.20s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 94%|█████████▍| 197/210 [10:12<01:01,  4.73s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 94%|█████████▍| 198/210 [10:16<00:55,  4.62s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 95%|█████████▍| 199/210 [10:18<00:42,  3.90s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 95%|█████████▌| 200/210 [10:24<00:45,  4.50s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 96%|█████████▌| 201/210 [10:25<00:30,  3.43s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 96%|█████████▌| 202/210 [10:27<00:23,  2.88s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 97%|█████████▋| 203/210 [10:33<00:26,  3.81s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 97%|█████████▋| 204/210 [10:36<00:21,  3.66s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 98%|█████████▊| 205/210 [10:38<00:15,  3.18s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 98%|█████████▊| 206/210 [10:43<00:15,  3.76s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 99%|█████████▊| 207/210 [10:45<00:09,  3.26s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      " 99%|█████████▉| 208/210 [10:46<00:05,  2.58s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|█████████▉| 209/210 [10:47<00:02,  2.12s/it]INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "100%|██████████| 210/210 [10:49<00:00,  3.09s/it]\n"
     ]
    }
   ],
   "source": [
    "# Load from disk to save time or regenerate as needed.\n",
    "try:\n",
    "    with open(\"resources/propositions.json\", \"r\") as f:\n",
    "        propositions = json.load(f)\n",
    "except:\n",
    "    # create props\n",
    "    propositions = [create_dense_props(chunk) for chunk in tqdm.tqdm(chunks)]\n",
    "    propositions = [\" \".join(prop) for prop in propositions]\n",
    "\n",
    "    # Save to disk for faster reload..\n",
    "    with open(\"resources/propositions.json\", \"w\") as f:\n",
    "        json.dump(propositions, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's evaluate the proposition vs the raw chunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Document: FORM 10-K Registrant: NIKE, Inc. State of Incorporation: Oregon IRS Employer Identification No.: 93-0584541 Address of Principal Executive Offices: One Bowerman Drive, Beaverton, Oregon 97005-6453 Telephone Number: (503) 671-6453 Securities Registered: Class B Common Stock NKE on New York Stock Exchange Well-Known Seasoned Issuer: Not applicable Filing of Reports: Registrant has filed all required reports for the past 12 months Submission of Interactive Data File: Registrant has submitted all required files for the past 12 months Filing Status: Large accelerated filer'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propositions[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': 'resources/nke-10k-2023.pdf', 'page': 0, 'page_label': '1'}, page_content=\"Table of Contents\\nUNITED STATES\\nSECURITIES AND EXCHANGE COMMISSION\\nWashington, D.C. 20549\\nFORM 10-K\\n(Mark One)\\n☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE FISCAL YEAR ENDED MAY 31, 2023\\nOR\\n☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE TRANSITION PERIOD FROM                         TO                         .\\nCommission File No. 1-10635\\nNIKE, Inc.\\n(Exact name of Registrant as specified in its charter)\\nOregon 93-0584541\\n(State or other jurisdiction of incorporation) (IRS Employer Identification No.)\\nOne Bowerman Drive, Beaverton, Oregon 97005-6453\\n(Address of principal executive offices and zip code)\\n(503) 671-6453\\n(Registrant's telephone number, including area code)\\nSECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:\\nClass B Common Stock NKE New York Stock Exchange\\n(Title of each class) (Trading symbol) (Name of each exchange on which registered)\\nSECURITIES REGISTERED PURSUANT TO SECTION 12(G) OF THE ACT:\\nNONE\\nIndicate by check mark: YES NO\\n•if the registrant is a well-known seasoned issuer, as defined in Rule 405 of the Securities Act. þ ¨\\n•if the registrant is not required to file reports pursuant to Section 13 or Section 15(d) of the Act. ¨ þ\\n•whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding\\n12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the\\npast 90 days.þ ¨\\n•whether the registrant has submitted electronically every Interactive Data File required to be submitted pursuant to Rule 405 of Regulation S-T\\n(§232.405 of this chapter) during the preceding 12 months (or for such shorter period that the registrant was required to submit such files).þ ¨\\n•whether the registrant is a large accelerated filer, an accelerated filer, a non-accelerated filer, a smaller reporting company or an emerging growth company. See the definitions of “large accelerated filer,”\\n“accelerated filer,” “smaller reporting company,” and “emerging growth company” in Rule 12b-2 of the Exchange Act.\\nLarge accelerated filer þ Accelerated filer ☐ Non-accelerated filer ☐ Smaller reporting company ☐ Emerging growth company ☐\\n•if an emerging growth company, if the registrant has elected not to use the extended transition period for complying with any new or revised financial\")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create embeddings from propositions data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "09:41:44 sentence_transformers.SentenceTransformer INFO   Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/robert.shelton/.pyenv/versions/3.11.9/lib/python3.11/site-packages/huggingface_hub/file_download.py:1142: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "09:41:45 sentence_transformers.SentenceTransformer INFO   Use pytorch device_name: mps\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05f458d0c8ed43be8cd08905e96fdfda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc95bb77c46e4d63986caa4e29ca223c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from redisvl.utils.vectorize import HFTextVectorizer\n",
    "\n",
    "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
    "\n",
    "prop_embeddings = hf.embed_many([\n",
    "    proposition for proposition in propositions\n",
    "])\n",
    "\n",
    "# Check to make sure we've created enough embeddings, 1 per document chunk\n",
    "len(prop_embeddings) == len(propositions) == len(chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5baI0xDQ1ui-"
   },
   "source": [
    "### Define a schema and create an index\n",
    "\n",
    "Below we connect to Redis and create an index that contains a text field, tag field, and vector field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "zB1EW_9n1ui-"
   },
   "outputs": [],
   "source": [
    "from redis import Redis\n",
    "from redisvl.index import SearchIndex\n",
    "\n",
    "\n",
    "index_name = \"redisvl\"\n",
    "\n",
    "\n",
    "schema = {\n",
    "  \"index\": {\n",
    "    \"name\": index_name,\n",
    "    \"prefix\": \"chunk\"\n",
    "  },\n",
    "  \"fields\": [\n",
    "    {\n",
    "        \"name\": \"chunk_id\",\n",
    "        \"type\": \"tag\",\n",
    "        \"attrs\": {\n",
    "            \"sortable\": True\n",
    "        }\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"proposition\",\n",
    "        \"type\": \"text\"\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"text_embedding\",\n",
    "        \"type\": \"vector\",\n",
    "        \"attrs\": {\n",
    "            \"dims\": hf.dims,\n",
    "            \"distance_metric\": \"cosine\",\n",
    "            \"algorithm\": \"hnsw\",\n",
    "            \"datatype\": \"float32\"\n",
    "        }\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to redis\n",
    "client = Redis.from_url(REDIS_URL)\n",
    "\n",
    "# create an index from schema and the client\n",
    "index = SearchIndex.from_dict(schema)\n",
    "index.set_client(client)\n",
    "index.create(overwrite=True, drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "C70C-UWj1ujA"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Index Information:\n",
      "╭──────────────┬────────────────┬────────────┬─────────────────┬────────────╮\n",
      "│ Index Name   │ Storage Type   │ Prefixes   │ Index Options   │   Indexing │\n",
      "├──────────────┼────────────────┼────────────┼─────────────────┼────────────┤\n",
      "│ redisvl      │ HASH           │ ['chunk']  │ []              │          0 │\n",
      "╰──────────────┴────────────────┴────────────┴─────────────────┴────────────╯\n",
      "Index Fields:\n",
      "╭────────────────┬────────────────┬────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮\n",
      "│ Name           │ Attribute      │ Type   │ Field Option   │ Option Value   │ Field Option   │ Option Value   │ Field Option   │   Option Value │ Field Option    │ Option Value   │ Field Option   │   Option Value │ Field Option    │   Option Value │\n",
      "├────────────────┼────────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤\n",
      "│ chunk_id       │ chunk_id       │ TAG    │ SEPARATOR      │ ,              │                │                │                │                │                 │                │                │                │                 │                │\n",
      "│ proposition    │ proposition    │ TEXT   │ WEIGHT         │ 1              │                │                │                │                │                 │                │                │                │                 │                │\n",
      "│ text_embedding │ text_embedding │ VECTOR │ algorithm      │ HNSW           │ data_type      │ FLOAT32        │ dim            │            384 │ distance_metric │ COSINE         │ M              │             16 │ ef_construction │            200 │\n",
      "╰────────────────┴────────────────┴────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯\n"
     ]
    }
   ],
   "source": [
    "# get info about the index\n",
    "# NBVAL_SKIP\n",
    "!rvl index info -i redisvl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qrj-jeGmBRTL"
   },
   "source": [
    "### Process and load dataset\n",
    "Below we use the RedisVL index to simply load the list of document chunks to Redis db."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "Zsg09Keg1ujA"
   },
   "outputs": [],
   "source": [
    "# load expects an iterable of dictionaries\n",
    "from redisvl.redis.utils import array_to_buffer\n",
    "\n",
    "data = [\n",
    "    {\n",
    "        'chunk_id': f'{i}',\n",
    "        'proposition': proposition,\n",
    "        # For HASH -- must convert embeddings to bytes\n",
    "        'text_embedding': array_to_buffer(prop_embeddings[i], dtype=\"float32\")\n",
    "    } for i, proposition in enumerate(propositions)\n",
    "]\n",
    "\n",
    "# RedisVL handles batching automatically\n",
    "keys = index.load(data, id_field=\"chunk_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup RedisVL AsyncSearchIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redis.asyncio import Redis as AsyncRedis\n",
    "from redisvl.index import AsyncSearchIndex\n",
    "\n",
    "client = AsyncRedis.from_url(REDIS_URL)\n",
    "index = AsyncSearchIndex.from_dict(schema)\n",
    "_ = await index.set_client(client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test the updated RAG workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redisvl.query import VectorQuery\n",
    "from redisvl.index import AsyncSearchIndex\n",
    "\n",
    "\n",
    "def promptify(query: str, context: str) -> str:\n",
    "    return f'''Use the provided context below derived from public financial\n",
    "    documents to answer the user's question. If you can't answer the user's\n",
    "    question, based on the context; do not guess. If there is no context at all,\n",
    "    respond with \"I don't know\".\n",
    "\n",
    "    User question:\n",
    "\n",
    "    {query}\n",
    "\n",
    "    Helpful context:\n",
    "\n",
    "    {context}\n",
    "\n",
    "    Answer:\n",
    "    '''\n",
    "\n",
    "# Update the retrieval helper to use propositions\n",
    "async def retrieve_context(index: AsyncSearchIndex, query_vector) -> str:\n",
    "    \"\"\"Fetch the relevant context from Redis using vector search\"\"\"\n",
    "    print(\"Using dense content representation\", flush=True)\n",
    "    results = await index.query(\n",
    "        VectorQuery(\n",
    "            vector=query_vector,\n",
    "            vector_field_name=\"text_embedding\",\n",
    "            return_fields=[\"proposition\"],\n",
    "            num_results=3\n",
    "        )\n",
    "    )\n",
    "    content = \"\\n\".join([result[\"proposition\"] for result in results])\n",
    "    return content\n",
    "\n",
    "# Update the answer_question method\n",
    "async def answer_question(index: AsyncSearchIndex, query: str):\n",
    "    \"\"\"Answer the user's question\"\"\"\n",
    "\n",
    "    SYSTEM_PROMPT = \"\"\"You are a helpful financial analyst assistant that has access\n",
    "    to public financial 10k documents in order to answer users questions about company\n",
    "    performance, ethics, characteristics, and core information.\n",
    "    \"\"\"\n",
    "\n",
    "    query_vector = hf.embed(query)\n",
    "    # Fetch context from Redis using vector search\n",
    "    context = await retrieve_context(index, query_vector)\n",
    "    # Generate contextualized prompt and feed to OpenAI\n",
    "    response = await openai.AsyncClient().chat.completions.create(\n",
    "        model=CHAT_MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
    "            {\"role\": \"user\", \"content\": promptify(query, context)}\n",
    "        ],\n",
    "        temperature=0.1,\n",
    "        seed=42\n",
    "    )\n",
    "    # Response provided by LLM\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a list of questions\n",
    "questions = [\n",
    "    \"What is the trend in the company's revenue and profit over the past few years?\",\n",
    "    \"What are the company's primary revenue sources?\",\n",
    "    \"How much debt does the company have, and what are its capital expenditure plans?\",\n",
    "    \"What does the company say about its environmental, social, and governance (ESG) practices?\",\n",
    "    \"What is the company's strategy for growth?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10acf9f3e1c44fa98f78e904cdc5fa11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f96d478f3b014fe9bd3dc9548110c677",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "647771c27bf9484ab71f6aef3f98a054",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20ba4e4353b246f9bd0398dbe4758d47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dde7aec0e15c49b5b3fa166e0d79774b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n",
      "09:42:46 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "09:42:46 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "09:42:46 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "09:42:47 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "09:42:47 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the trend in the company's revenue and...</td>\n",
       "      <td>Based on the provided financial information fr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What are the company's primary revenue sources?</td>\n",
       "      <td>The company's primary revenue sources are from...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>How much debt does the company have, and what ...</td>\n",
       "      <td>The company has long-term debt of $8,927 milli...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What does the company say about its environmen...</td>\n",
       "      <td>The company acknowledges the importance of env...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is the company's strategy for growth?</td>\n",
       "      <td>NIKE's strategy for growth focuses on long-ter...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What is the trend in the company's revenue and...   \n",
       "1    What are the company's primary revenue sources?   \n",
       "2  How much debt does the company have, and what ...   \n",
       "3  What does the company say about its environmen...   \n",
       "4         What is the company's strategy for growth?   \n",
       "\n",
       "                                              answer  \n",
       "0  Based on the provided financial information fr...  \n",
       "1  The company's primary revenue sources are from...  \n",
       "2  The company has long-term debt of $8,927 milli...  \n",
       "3  The company acknowledges the importance of env...  \n",
       "4  NIKE's strategy for growth focuses on long-ter...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import asyncio\n",
    "import pandas as pd\n",
    "\n",
    "results = await asyncio.gather(*[\n",
    "    answer_question(index, question) for question in questions\n",
    "])\n",
    "\n",
    "pd.DataFrame(columns=[\"question\", \"answer\"], data=list(zip(questions, results)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TnkK0NwIIM9q"
   },
   "source": [
    "### Improve accuracy with query rewriting / expansion\n",
    "\n",
    "We can also use the power on an LLM to rewrite or expand an input question.\n",
    "\n",
    "Example: https://github.com/langchain-ai/langchain/blob/master/templates/rewrite-retrieve-read/rewrite_retrieve_read/chain.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f92142ba61a4b5492677b34265e5487",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n",
      "09:42:49 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Based on the provided context, NIKE, INC. has total assets of $37,531 million in 2023. This indicates that NIKE, INC. is a large company with significant assets.'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# An example question that is a bit simplistic...\n",
    "await answer_question(index, \"How big is the company?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "Tg55HqLFIRXJ"
   },
   "outputs": [],
   "source": [
    "async def rewrite_query(query: str, prompt: str = None):\n",
    "    \"\"\"Rewrite the user's original query\"\"\"\n",
    "\n",
    "    SYSTEM_PROMPT = prompt if prompt else \"\"\"Given the user's input question below, find a better or\n",
    "    more complete way to phrase this question in order to improve semantic search\n",
    "    engine retrieval quality over a set of SEC 10K PDF docs. Return the rephrased\n",
    "    question as a string in a JSON response under the key \"query\".\"\"\"\n",
    "\n",
    "    response = await openai.AsyncClient().chat.completions.create(\n",
    "        model=CHAT_MODEL,\n",
    "        response_format={ \"type\": \"json_object\" },\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
    "            {\"role\": \"user\", \"content\": f\"Original input question from user: {query}\"}\n",
    "        ],\n",
    "        temperature=0.1,\n",
    "        seed=42\n",
    "    )\n",
    "    # Response provided by LLM\n",
    "    rewritten_query = json.loads(response.choices[0].message.content)[\"query\"]\n",
    "    return rewritten_query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "09:42:50 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'What is the size of the company in terms of revenue, assets, and market capitalization?'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example Sinple Query Rewritten\n",
    "await rewrite_query(\"How big is the company?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "9ubNQrJOYL42"
   },
   "outputs": [],
   "source": [
    "async def answer_question(index: AsyncSearchIndex, query: str, **kwargs):\n",
    "    \"\"\"Answer the user's question\"\"\"\n",
    "\n",
    "    SYSTEM_PROMPT = \"\"\"You are a helpful financial analyst assistant that has access\n",
    "    to public financial 10k documents in order to answer users questions about company\n",
    "    performance, ethics, characteristics, and core information.\n",
    "    \"\"\"\n",
    "\n",
    "    # Rewrite the query using an LLM\n",
    "    rewritten_query = await rewrite_query(query, **kwargs)\n",
    "    print(\"User query updated to:\\n\", rewritten_query, flush=True)\n",
    "\n",
    "    query_vector = hf.embed(rewritten_query)\n",
    "    # Fetch context from Redis using vector search\n",
    "    context = await retrieve_context(index, query_vector)\n",
    "    print(\"Context retrieved\", flush=True)\n",
    "\n",
    "    # Generate contextualized prompt and feed to OpenAI\n",
    "    response = await openai.AsyncClient().chat.completions.create(\n",
    "        model=CHAT_MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
    "            {\"role\": \"user\", \"content\": promptify(rewritten_query, context)}\n",
    "        ],\n",
    "        temperature=0.1,\n",
    "        seed=42\n",
    "    )\n",
    "    # Response provided by LLM\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "BIO_jW6KYsMU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "09:42:52 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "User query updated to:\n",
      " What is the size of the company in terms of revenue, assets, and market capitalization?\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99da3fa22c7f4b0aa66b57f802d9b1a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n",
      "Context retrieved\n",
      "09:42:54 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Based on the provided context from the financial documents:\\n\\n- Revenue information is not explicitly provided in the context.\\n- Total assets for NIKE, INC. in 2023 amount to $37,531 million.\\n- Market capitalization is not directly provided in the context.\\n\\nTherefore, based on the information provided, the size of the company in terms of assets is $37,531 million. Unfortunately, without specific revenue figures or market capitalization data, we cannot provide those metrics at this time.'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "# Now try again with query re-writing enabled\n",
    "await answer_question(index, \"How big is the company?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p97uL4g9T6LQ"
   },
   "source": [
    "### Improve performance and cut costs with LLM caching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "7geEAsYST6LQ"
   },
   "outputs": [],
   "source": [
    "from redisvl.extensions.llmcache import SemanticCache\n",
    "\n",
    "llmcache = SemanticCache(\n",
    "    name=\"llmcache\",\n",
    "    vectorizer=hf,\n",
    "    redis_url=REDIS_URL,\n",
    "    ttl=120,\n",
    "    distance_threshold=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "1ALcQXAqT6LQ"
   },
   "outputs": [],
   "source": [
    "from functools import wraps\n",
    "\n",
    "# Create an LLM caching decorator\n",
    "def cache(func):\n",
    "    @wraps(func)\n",
    "    async def wrapper(index, query_text, *args, **kwargs):\n",
    "        query_vector = llmcache._vectorizer.embed(query_text)\n",
    "\n",
    "        # Check the cache with the vector\n",
    "        if result := llmcache.check(vector=query_vector):\n",
    "            return result[0]['response']\n",
    "\n",
    "        response = await func(index, query_text, query_vector=query_vector)\n",
    "        llmcache.store(query_text, response, query_vector)\n",
    "        return response\n",
    "    return wrapper\n",
    "\n",
    "\n",
    "@cache\n",
    "async def answer_question(index: AsyncSearchIndex, query: str, **kwargs):\n",
    "    \"\"\"Answer the user's question\"\"\"\n",
    "\n",
    "    SYSTEM_PROMPT = \"\"\"You are a helpful financial analyst assistant that has access\n",
    "    to public financial 10k documents in order to answer users questions about company\n",
    "    performance, ethics, characteristics, and core information.\n",
    "    \"\"\"\n",
    "\n",
    "    context = await retrieve_context(index, kwargs[\"query_vector\"])\n",
    "    response = await openai.AsyncClient().chat.completions.create(\n",
    "        model=CHAT_MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
    "            {\"role\": \"user\", \"content\": promptify(query, context)}\n",
    "        ],\n",
    "        temperature=0.1,\n",
    "        seed=42\n",
    "    )\n",
    "    # Response provided by GPT-3.5\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "BXK_BXuhT6LQ"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "abc6b2b43fd84288be593a26bfbd754e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using dense content representation\n",
      "09:42:55 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Nike's total revenues increased from $46,710 million in 2022 to $51,217 million in 2023.\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "query = \"What was Nike's revenue last year compared to this year??\"\n",
    "\n",
    "await answer_question(index, query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "7mZpSpf9T6LQ"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea9587cb3ba546ccb4b92c9e4031483e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Nike's total revenues increased from $46,710 million in 2022 to $51,217 million in 2023.\""
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "query = \"What was Nike's total revenue in the last year compared to now??\"\n",
    "\n",
    "await answer_question(index, query)\n",
    "\n",
    "# notice no HTTP request to OpenAI since this question is \"close enough\" to the last one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UaiF_ws7itsi"
   },
   "source": [
    "### Improve personalization with including chat session history\n",
    "\n",
    "In order to preserve state in the conversation, it's imperitive to offload conversation history to a database that can handle high transaction throughput for writes/reads to limit system latency.\n",
    "\n",
    "We can store message history for a particular user session in a Redis List data type.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "WMOF7fJQdhgN"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "class ChatBot:\n",
    "    def __init__(self, index: AsyncSearchIndex, user: str):\n",
    "        self.index = index\n",
    "        self.user = user\n",
    "\n",
    "    async def get_messages(self) -> list:\n",
    "        \"\"\"Get all messages associated with a session\"\"\"\n",
    "        return [\n",
    "            json.loads(msg) for msg in await self.index.client.lrange(f\"messages:{self.user}\", 0, -1)\n",
    "        ]\n",
    "\n",
    "    async def add_messages(self, messages: list):\n",
    "        \"\"\"Add chat messages to a Redis List\"\"\"\n",
    "        return await self.index.client.rpush(\n",
    "            f\"messages:{self.user}\", *[json.dumps(msg) for msg in messages]\n",
    "        )\n",
    "\n",
    "    async def clear_history(self):\n",
    "        \"\"\"Clear session chat\"\"\"\n",
    "        await index.client.delete(f\"messages:{self.user}\")\n",
    "\n",
    "    @staticmethod\n",
    "    def promptify(query: str, context: str) -> str:\n",
    "        return f'''Use the provided context below derived from public financial\n",
    "        documents to answer the user's question. If you can't answer the user's\n",
    "        question, based on the context; do not guess. If there is no context at all,\n",
    "        respond with \"I don't know\".\n",
    "\n",
    "        User question:\n",
    "\n",
    "        {query}\n",
    "\n",
    "        Helpful context:\n",
    "\n",
    "        {context}\n",
    "\n",
    "        Answer:\n",
    "        '''\n",
    "\n",
    "    async def retrieve_context(self, query_vector) -> str:\n",
    "        \"\"\"Fetch the relevant context from Redis using vector search\"\"\"\n",
    "        results = await self.index.query(\n",
    "            VectorQuery(\n",
    "                vector=query_vector,\n",
    "                vector_field_name=\"text_embedding\",\n",
    "                return_fields=[\"proposition\"],\n",
    "                num_results=3\n",
    "            )\n",
    "        )\n",
    "        content = \"\\n\".join([result[\"proposition\"] for result in results])\n",
    "        return content\n",
    "\n",
    "    async def answer_question(self, query: str):\n",
    "        \"\"\"Answer the user's question with historical context and caching baked-in\"\"\"\n",
    "\n",
    "        SYSTEM_PROMPT = \"\"\"You are a helpful financial analyst assistant that has access\n",
    "        to public financial 10k documents in order to answer users questions about company\n",
    "        performance, ethics, characteristics, and core information.\n",
    "        \"\"\"\n",
    "\n",
    "        # Create query vector\n",
    "        query_vector = llmcache._vectorizer.embed(query)\n",
    "\n",
    "        # TODO - implement semantic gaurdrails?\n",
    "\n",
    "        # Check the cache with the vector\n",
    "        if result := llmcache.check(vector=query_vector):\n",
    "            answer = result[0]['response']\n",
    "        else:\n",
    "            # TODO - implement query rewriting?\n",
    "            context = await self.retrieve_context(query_vector)\n",
    "            session = await self.get_messages()\n",
    "            # TODO - implement session summarization?\n",
    "            messages = (\n",
    "                [{\"role\": \"system\", \"content\": SYSTEM_PROMPT}] +\n",
    "                session +\n",
    "                [{\"role\": \"user\", \"content\": self.promptify(query, context)}]\n",
    "            )\n",
    "            # Response provided by GPT-3.5\n",
    "            response = await openai.AsyncClient().chat.completions.create(\n",
    "                model=CHAT_MODEL,\n",
    "                messages=messages,\n",
    "                temperature=0.1,\n",
    "                seed=42\n",
    "            )\n",
    "            answer = response.choices[0].message.content\n",
    "            llmcache.store(query, answer, query_vector)\n",
    "\n",
    "        # Add message history\n",
    "        await self.add_messages([\n",
    "            {\"role\": \"user\", \"content\": query},\n",
    "            {\"role\": \"assistant\", \"content\": answer}\n",
    "        ])\n",
    "\n",
    "        return answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the entire RAG workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "_Z3RUvyxdhiz"
   },
   "outputs": [],
   "source": [
    "# Setup Session\n",
    "chat = ChatBot(index, \"tyler\")\n",
    "await chat.clear_history()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run a simple chat\n",
    "stopterms = [\"exit\", \"quit\", \"end\", \"cancel\"]\n",
    "\n",
    "# Simple Chat\n",
    "# NBVAL_SKIP\n",
    "while True:\n",
    "    user_query = input()\n",
    "    if user_query.lower() in stopterms:\n",
    "        break\n",
    "    answer = await chat.answer_question(user_query)\n",
    "    print(answer, flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZoPQMAShZ5Uy"
   },
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "await chat.get_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5l4uEgKzljes"
   },
   "source": [
    "## Your Next Steps\n",
    "\n",
    "While a good start, there is still more to do. **For example**:\n",
    "- we could utilize message history to generate an updated and contextualized query to use for retrieval and answer generation (with an LLM). Otherwise, there can be a disconnect between what a user is asking (in context) and what they are asking in isolation.\n",
    "- we could utilize an LLM to summarize conversation history to use as context instead of passing the whole slew of messages to the Chat endpoint.\n",
    "- we could utilize semantic properties of the message history (or summaries) in order to fetch only relevant conversation bits (vector search).\n",
    "- we could utilize a technique like HyDE ( a form of query rewriting ) to improve the retrieval quality from raw user input to source documents OR try to break down user questions into sub questions and fetch / join context based on the different searces.\n",
    "- we could incorporate semantic routing to take a broken down question and route to different data sources, indices, or query types (etc).\n",
    "- we could add semantic guardrails on the front end or back end of the conversation I/O to ensure we are within bounds of approved topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wscs4Mvo1ujD"
   },
   "source": [
    "## Cleanup\n",
    "\n",
    "Clean up the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "On6yNuQn1ujD"
   },
   "outputs": [],
   "source": [
    "await index.client.flushall()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}