python-recipes/RAG/06_ragas_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "# Evaluating RAG\n",
    "\n",
    "This notebook uses the [ragas library](https://docs.ragas.io/en/stable/getstarted/index.html) and [Redis](https://redis.com) to evaluate the performance of sample RAG application. Also see the original [source paper](https://arxiv.org/pdf/2309.15217) to build a more detailed understanding.\n",
    "\n",
    "## Let's Begin!\n",
    "<a href=\"https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/RAG/06_ragas_evaluation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we need a RAG app to evaluate. Let's create one using LangChain and connect it with Redis as the vector DB.\n",
    "\n",
    "## Init redis, data prep, and populating the vector DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -q redis \"unstructured[pdf]\" sentence-transformers langchain langchain-redis langchain-huggingface langchain-openai ragas datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running Redis in Colab\n",
    "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "%%sh\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update  > /dev/null 2>&1\n",
    "sudo apt-get install redis-stack-server  > /dev/null 2>&1\n",
    "redis-stack-server --daemonize yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Alternative Environments\n",
    "There are many ways to get the necessary redis-stack instance running\n",
    "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
    "own version of Redis Enterprise running, that works too!\n",
    "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
    "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Replace values below with your own if using Redis Cloud instance\n",
    "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
    "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\")      # ex: 18374\n",
    "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")  # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
    "\n",
    "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
    "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "CHUNK_SIZE = 2500\n",
    "CHUNK_OVERLAP = 0\n",
    "\n",
    "# pdf to load\n",
    "path = 'resources/nke-10k-2023.pdf'\n",
    "assert os.path.exists(path), f\"File not found: {path}\"\n",
    "\n",
    "# load and split\n",
    "loader = PyPDFLoader(path)\n",
    "pages = loader.load()\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "chunks = text_splitter.split_documents(pages)\n",
    "\n",
    "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Table of ContentsUNITED STATESSECURITIES AND EXCHANGE COMMISSIONWashington, D.C. 20549FORM 10-K(Mark One)☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE FISCAL YEAR ENDED MAY 31, 2023OR☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE TRANSITION PERIOD FROM TO .Commission File No. 1-10635\\n\\nAs of November 30, 2022, the aggregate market values of the Registrant's Common Stock held by non-affiliates were:Class A$7,831,564,572 Class B136,467,702,472 $144,299,267,044\\n\\nNIKE, Inc.(Exact name of Registrant as specified in its charter)Oregon93-0584541(State or other jurisdiction of incorporation)(IRS Employer Identification No.)One Bowerman Drive, Beaverton, Oregon 97005-6453(Address of principal executive offices and zip code)(503) 671-6453(Registrant's telephone number, including area code)SECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:Class B Common StockNKENew York Stock Exchange(Title of each class)(Trading symbol)(Name of each exchange on which registered)SECURITIES REGISTERED PURSUANT TO SECTION 12(G) OF THE ACT:NONE\")"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "\n",
    "embeddings = HuggingFaceEmbeddings(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_redis import RedisVectorStore\n",
    "\n",
    "# set the index name for this example\n",
    "index_name = \"ragas_ex\"\n",
    "\n",
    "# construct the vector store class from texts and metadata\n",
    "rds = RedisVectorStore.from_documents(\n",
    "    chunks,\n",
    "    embeddings,\n",
    "    index_name=index_name,\n",
    "    redis_url=REDIS_URL,\n",
    "    metadata_schema=[\n",
    "        {\n",
    "            \"name\": \"source\",\n",
    "            \"type\": \"text\"\n",
    "        },\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the vector store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rds.similarity_search(\"What was nike's revenue last year?\")[0].page_content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup RAG\n",
    "\n",
    "Now that the vector db is populated let's initialize our RAG app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")\n",
    "\n",
    "llm = ChatOpenAI(\n",
    "    openai_api_key=os.environ[\"OPENAI_API_KEY\"],\n",
    "    model=\"gpt-3.5-turbo-16k\",\n",
    "    max_tokens=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "system_prompt = \"\"\"\n",
    "    Use the following pieces of context from financial 10k filings data to answer the user question at the end. \n",
    "    If you don't know the answer, say that you don't know, don't try to make up an answer.\n",
    "\n",
    "    Context:\n",
    "    ---------\n",
    "    {context}\n",
    "\"\"\"\n",
    "\n",
    "def format_docs(docs):\n",
    "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        (\"human\", \"{input}\")\n",
    "    ]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test it out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': \"What was nike's revenue last year?\",\n",
       " 'context': [Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content='As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'),\n",
       "  Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"NIKE, INC. CONSOLIDATED STATEMENTS OF INCOME\\n\\n(In millions, except per share data)\\n\\nRevenues Cost of sales\\n\\nGross profit\\n\\nDemand creation expense Operating overhead expense\\n\\nTotal selling and administrative expense\\n\\nInterest expense (income), net\\n\\nOther (income) expense, net Income before income taxes\\n\\nIncome tax expense NET INCOME\\n\\nEarnings per common share:\\n\\nBasic Diluted\\n\\nWeighted average common shares outstanding:\\n\\nBasic Diluted\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n$\\n\\n$\\n\\n$ $\\n\\nYEAR ENDED MAY 31,\\n\\n2023\\n\\n2022\\n\\n2021\\n\\n51,217 $ 28,925\\n\\n46,710 $ 25,231\\n\\n44,538 24,576\\n\\n22,292 4,060 12,317\\n\\n21,479 3,850 10,954\\n\\n19,962 3,114 9,911\\n\\n16,377 (6)\\n\\n14,804 205\\n\\n13,025 262\\n\\n(280) 6,201\\n\\n(181) 6,651\\n\\n14 6,661\\n\\n1,131 5,070 $\\n\\n605 6,046 $\\n\\n934 5,727\\n\\n3.27 $ 3.23 $\\n\\n3.83 $ 3.75 $\\n\\n3.64 3.56\\n\\n1,551.6 1,569.8\\n\\n1,578.8 1,610.8\\n\\n1,573.0 1,609.4\\n\\n2023 FORM 10-K 55\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED STATEMENTS OF COMPREHENSIVE INCOME\\n\\nYEAR ENDED MAY 31,\\n\\n(Dollars in millions)\\n\\n2023\\n\\n2022\\n\\nNet income Other comprehensive income (loss), net of tax:\\n\\n$\\n\\n5,070 $\\n\\n6,046 $\\n\\nChange in net foreign currency translation adjustment\\n\\n267\\n\\n(522)\\n\\nChange in net gains (losses) on cash flow hedges Change in net gains (losses) on other\\n\\n(348) (6)\\n\\n1,214 6\\n\\nTotal other comprehensive income (loss), net of tax TOTAL COMPREHENSIVE INCOME\\n\\n$\\n\\n(87) 4,983 $\\n\\n698 6,744 $\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n2023 FORM 10-K 56\\n\\n2021\\n\\n5,727\\n\\n496\\n\\n(825) 5\\n\\n(324) 5,403\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED BALANCE SHEETS\\n\\n(In millions)\\n\\nASSETS\\n\\nCurrent assets:\\n\\nCash and equivalents Short-term investments\\n\\nAccounts receivable, net Inventories Prepaid expenses and other current assets\\n\\nTotal current assets\\n\\nProperty, plant and equipment, net\\n\\nOperating lease right-of-use assets, net Identifiable intangible assets, net Goodwill\\n\\nDeferred income taxes and other assets\\n\\nTOTAL ASSETS\\n\\nLIABILITIES AND SHAREHOLDERS' EQUITY Current liabilities:\\n\\nCurrent portion of long-term debt Notes payable Accounts payable\\n\\nCurrent portion of operating lease liabilities Accrued liabilities Income taxes payable\\n\\nTotal current liabilities\\n\\nLong-term debt\\n\\nOperating lease liabilities Deferred income taxes and other liabilities Commitments and contingencies (Note 16)\\n\\nRedeemable preferred stock Shareholders' equity: Common stock at stated value:\"),\n",
       "  Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Tax (expense) benefit Gain (loss) net of tax\\n\\n5 (14)\\n\\n(9) 22\\n\\nTotal net gain (loss) reclassified for the period\\n\\n$\\n\\n463 $\\n\\n30\\n\\n2023 FORM 10-K 82\\n\\nTable of Contents\\n\\nNOTE 14 — REVENUES\\n\\nDISAGGREGATION OF REVENUES The following tables present the Company's Revenues disaggregated by reportable operating segment, major product line and distribution channel:\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nYEAR ENDED MAY 31, 2023 ASIA PACIFIC & LATIN (1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nAMERICA\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear\\n\\n$\\n\\n14,897 $\\n\\n8,260 $\\n\\n5,435 $\\n\\n4,543 $\\n\\n— $\\n\\n33,135 $\\n\\n2,155 $\\n\\n— $\\n\\n35,290\\n\\nApparel Equipment Other\\n\\n5,947 764 —\\n\\n4,566 592 —\\n\\n1,666 147 —\\n\\n1,664 224 —\\n\\n— — 58\\n\\n13,843 1,727 58\\n\\n90 28 154\\n\\n— — 27\\n\\n13,933 1,755 239\\n\\nTOTAL REVENUES\\n\\n$\\n\\n21,608 $\\n\\n13,418 $\\n\\n7,248 $\\n\\n6,431 $\\n\\n58 $\\n\\n48,763 $\\n\\n2,427 $\\n\\n27 $\\n\\n51,217\\n\\nRevenues by:\\n\\nSales to Wholesale Customers Sales through Direct to Consumer\\n\\n$\\n\\n11,273 $ 10,335\\n\\n8,522 $ 4,896\\n\\n3,866 $ 3,382\\n\\n3,736 $ 2,695\\n\\n— $ —\\n\\n27,397 $ 21,308\\n\\n1,299 $ 974\\n\\n— $ —\\n\\n28,696 22,282\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n21,608 $\\n\\n—\\n\\n13,418 $\\n\\n— 7,248 $\\n\\n— 6,431 $\\n\\n58 58 $\\n\\n58\\n\\n48,763 $\\n\\n154 2,427 $\\n\\n27 27 $\\n\\n239 51,217\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand businesses in its CASA territory to third-party distributors.\\n\\nYEAR ENDED MAY 31, 2022\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nASIA PACIFIC & LATIN AMERICA\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear Apparel\\n\\n$\\n\\n12,228 $ 5,492\\n\\n7,388 $ 4,527\\n\\n5,416 $ 1,938\\n\\n4,111 $ 1,610\\n\\n— $ —\\n\\n29,143 $ 13,567\\n\\n2,094 $ 103\\n\\n— $ —\\n\\n31,237 13,670\\n\\nEquipment Other\\n\\n633 —\\n\\n564 —\\n\\n193 —\\n\\n234 —\\n\\n— 102\\n\\n1,624 102\\n\\n26 123\\n\\n— (72)\\n\\n1,650 153\\n\\nTOTAL REVENUES Revenues by:\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\nSales to Wholesale Customers Sales through Direct to Consumer Other\\n\\n$\\n\\n9,621 $ 8,732 —\\n\\n8,377 $ 4,102 —\\n\\n4,081 $ 3,466 —\\n\\n3,529 $ 2,426 —\\n\\n— $ — 102\\n\\n25,608 $ 18,726 102\\n\\n1,292 $ 931 123\\n\\n— $ — (72)\\n\\n26,900 19,657 153\\n\\nTOTAL REVENUES\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\n2023 FORM 10-K 83\\n\\nTable of Contents\\n\\nYEAR ENDED MAY 31, 2021\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\"),\n",
       "  Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"ASIA PACIFIC & LATIN AMERICA\\n\\n(1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE BRAND\\n\\nCONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by:\\n\\nFootwear Apparel Equipment\\n\\n$\\n\\n11,644 $ 5,028 507\\n\\n6,970 $ 3,996 490\\n\\n5,748 $ 2,347 195\\n\\n3,659 $ 1,494 190\\n\\n— $ — —\\n\\n28,021 $ 12,865 1,382\\n\\n1,986 $ 104 29\\n\\n— $ — —\\n\\n30,007 12,969 1,411\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n17,179 $\\n\\n—\\n\\n11,456 $\\n\\n— 8,290 $\\n\\n— 5,343 $\\n\\n25 25 $\\n\\n25\\n\\n42,293 $\\n\\n86 2,205 $\\n\\n40 40 $\\n\\n151 44,538\\n\\nRevenues by:\\n\\nSales to Wholesale Customers $\\n\\n10,186 $\\n\\n7,812 $\\n\\n4,513 $\\n\\n3,387 $\\n\\n— $\\n\\n25,898 $\\n\\n1,353 $\\n\\n— $\\n\\n27,251\\n\\nSales through Direct to Consumer Other\\n\\n6,993 —\\n\\n3,644 —\\n\\n3,777 —\\n\\n1,956 —\\n\\n— 25\\n\\n16,370 25\\n\\n766 86\\n\\n— 40\\n\\n17,136 151\\n\\nTOTAL REVENUES\\n\\n$\\n\\n17,179 $\\n\\n11,456 $\\n\\n8,290 $\\n\\n5,343 $\\n\\n25 $\\n\\n42,293 $\\n\\n2,205 $\\n\\n40 $\\n\\n44,538\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand business in Brazil to a third-party distributor.\\n\\nFor the fiscal years ended May 31, 2023, 2022 and 2021, Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment. Converse Other revenues were primarily attributable to licensing businesses. Corporate revenues primarily consisted of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse but managed through the Company's central foreign exchange risk management program.\\n\\nAs of May 31, 2023 and 2022, the Company did not have any contract assets and had an immaterial amount of contract liabilities recorded in Accrued liabilities on the Consolidated Balance Sheets.\\n\\nSALES-RELATED RESERVES\\n\\nAs of May 31, 2023 and 2022, the Company's sales-related reserve balance, which includes returns, post-invoice sales discounts and miscellaneous claims, was $994 million and $1,015 million, respectively, recorded in Accrued liabilities on the Consolidated Balance Sheets. The estimated cost of inventory for expected product returns was $226 million and $194 million as of May 31, 2023 and 2022, respectively, and was recorded in Prepaid expenses and other current assets on the Consolidated Balance Sheets.\\n\\nNOTE 15 — OPERATING SEGMENTS AND RELATED INFORMATION\")],\n",
       " 'answer': \"Nike's revenue last year was $51,217 million.\"}"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "\n",
    "question_answer_chain = create_stuff_documents_chain(llm, prompt)\n",
    "rag_chain = create_retrieval_chain(rds.as_retriever(), question_answer_chain)\n",
    "\n",
    "rag_chain.invoke({\"input\": \"What was nike's revenue last year?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Creating a test set\n",
    "\n",
    "Now that our setup is complete and we have our RAG app to evaluate we need a test set to evaluate against. The ragas library provides a helpful class for generating a synthetic test set given our data as input that we will use here. The output of this generation is a set of `questions`, `contexts`, and `ground_truth`. \n",
    "\n",
    "The questions are generated by an LLM based on slices of context from the provided doc and the ground_truth is determined via a critic LLM. Note there is nothing special about this data itself and you can provide your own `questions` and `ground_truth` for evaluation purposes. When starting a project however, there is often a lack of quality human labeled data to be used for evaluation and a synthetic dataset is a valuable place to start if pre live user/process data (which should be incorporated as an ultimate goal).\n",
    "\n",
    "For more detail see [the docs](https://docs.ragas.io/en/stable/concepts/testset_generation.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "# source: https://docs.ragas.io/en/latest/getstarted/testset_generation.html\n",
    "from ragas.testset.generator import TestsetGenerator\n",
    "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
    "from ragas.run_config import RunConfig\n",
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
    "\n",
    "run_config = RunConfig(\n",
    "    timeout=200,\n",
    "    max_wait=160,\n",
    "    max_retries=3,\n",
    ")\n",
    "\n",
    "# generator with openai models\n",
    "generator_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n",
    "critic_llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "generator = TestsetGenerator.from_langchain(\n",
    "    generator_llm,\n",
    "    critic_llm,\n",
    "    embeddings,\n",
    "    run_config=run_config,\n",
    ")\n",
    "\n",
    "testset = generator.generate_with_langchain_docs(\n",
    "    chunks,\n",
    "    test_size=10,\n",
    "    distributions={\n",
    "        simple: 0.5,\n",
    "        reasoning: 0.25,\n",
    "        multi_context: 0.25\n",
    "    },\n",
    "    run_config=run_config\n",
    ")\n",
    "\n",
    "# save to csv since this can be a time consuming process\n",
    "testset.to_pandas().to_csv(\"resources/new_testset.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation helper functions\n",
    "\n",
    "The following code takes a RetrievalQA chain, testset dataframe, and the metrics to be evaluated and returns a dataframe including the metrics calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from datasets import Dataset\n",
    "from ragas import evaluate\n",
    "from ragas.run_config import RunConfig\n",
    "\n",
    "def parse_contexts(source_docs):\n",
    "    return [doc.page_content for doc in source_docs]\n",
    "\n",
    "def create_evaluation_dataset(chain, testset):\n",
    "    res_set = {\n",
    "        \"question\": [],\n",
    "        \"answer\": [],\n",
    "        \"contexts\": [],\n",
    "        \"ground_truth\": []\n",
    "    }\n",
    "\n",
    "    for _, row in testset.iterrows():\n",
    "        result = chain.invoke({\"input\": row[\"question\"]})\n",
    "\n",
    "        res_set[\"question\"].append(row[\"question\"])\n",
    "        res_set[\"answer\"].append(result[\"answer\"])\n",
    "\n",
    "        contexts = parse_contexts(result[\"context\"])\n",
    "\n",
    "        if not len(contexts):\n",
    "            print(f\"no contexts found for question: {row['question']}\")\n",
    "        res_set[\"contexts\"].append(contexts)\n",
    "        res_set[\"ground_truth\"].append(str(row[\"ground_truth\"]))\n",
    "\n",
    "    return Dataset.from_dict(res_set)\n",
    "\n",
    "def evaluate_dataset(eval_dataset, metrics, llm, embeddings):\n",
    "\n",
    "    run_config = RunConfig(max_retries=1) # see ragas docs for more run_config options\n",
    "\n",
    "    eval_result = evaluate(\n",
    "        eval_dataset,\n",
    "        metrics=metrics,\n",
    "        run_config=run_config,\n",
    "        llm=llm,\n",
    "        embeddings=embeddings\n",
    "    )\n",
    "\n",
    "    eval_df = eval_result.to_pandas()\n",
    "    return eval_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the evaluation data\n",
    "\n",
    "Input: chain to be evaluated and a pregenerated test set<br>\n",
    "Output: dataset formatted for use with ragas evaluation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>contexts</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>evolution_type</th>\n",
       "      <th>metadata</th>\n",
       "      <th>episode_done</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are short-term investments and how are th...</td>\n",
       "      <td>[\"CASH AND EQUIVALENTS Cash and equivalents re...</td>\n",
       "      <td>Short-term investments are highly liquid inves...</td>\n",
       "      <td>simple</td>\n",
       "      <td>[{'source': 'resources/nke-10k-2023.pdf'}]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What are some of the risks and uncertainties a...</td>\n",
       "      <td>['Our NIKE Direct operations, including our re...</td>\n",
       "      <td>Many factors unique to retail operations, some...</td>\n",
       "      <td>simple</td>\n",
       "      <td>[{'source': 'resources/nke-10k-2023.pdf'}]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is NIKE's policy regarding securities ana...</td>\n",
       "      <td>[\"Investors should also be aware that while NI...</td>\n",
       "      <td>NIKE's policy is to not disclose any material ...</td>\n",
       "      <td>simple</td>\n",
       "      <td>[{'source': 'resources/nke-10k-2023.pdf'}]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What are the revenues for the Footwear and App...</td>\n",
       "      <td>['(Dollars in millions, except per share data)...</td>\n",
       "      <td>The revenues for the Footwear and Apparel cate...</td>\n",
       "      <td>simple</td>\n",
       "      <td>[{'source': 'resources/nke-10k-2023.pdf'}]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>How do master netting arrangements impact the ...</td>\n",
       "      <td>[\"The Company records the assets and liabiliti...</td>\n",
       "      <td>The Company records the assets and liabilities...</td>\n",
       "      <td>simple</td>\n",
       "      <td>[{'source': 'resources/nke-10k-2023.pdf'}]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What are short-term investments and how are th...   \n",
       "1  What are some of the risks and uncertainties a...   \n",
       "2  What is NIKE's policy regarding securities ana...   \n",
       "3  What are the revenues for the Footwear and App...   \n",
       "4  How do master netting arrangements impact the ...   \n",
       "\n",
       "                                            contexts  \\\n",
       "0  [\"CASH AND EQUIVALENTS Cash and equivalents re...   \n",
       "1  ['Our NIKE Direct operations, including our re...   \n",
       "2  [\"Investors should also be aware that while NI...   \n",
       "3  ['(Dollars in millions, except per share data)...   \n",
       "4  [\"The Company records the assets and liabiliti...   \n",
       "\n",
       "                                        ground_truth evolution_type  \\\n",
       "0  Short-term investments are highly liquid inves...         simple   \n",
       "1  Many factors unique to retail operations, some...         simple   \n",
       "2  NIKE's policy is to not disclose any material ...         simple   \n",
       "3  The revenues for the Footwear and Apparel cate...         simple   \n",
       "4  The Company records the assets and liabilities...         simple   \n",
       "\n",
       "                                     metadata  episode_done  \n",
       "0  [{'source': 'resources/nke-10k-2023.pdf'}]          True  \n",
       "1  [{'source': 'resources/nke-10k-2023.pdf'}]          True  \n",
       "2  [{'source': 'resources/nke-10k-2023.pdf'}]          True  \n",
       "3  [{'source': 'resources/nke-10k-2023.pdf'}]          True  \n",
       "4  [{'source': 'resources/nke-10k-2023.pdf'}]          True  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testset_df = pd.read_csv(\"resources/testset_15.csv\")\n",
    "testset_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_dataset = create_evaluation_dataset(rag_chain, testset_df)\n",
    "eval_dataset.to_pandas().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate generation metrics\n",
    "Generation metrics quantify how well the RAG app did creating answers to the provided questions (i.e. the G in **R**etrival **A**ugments **G**eneration). We will calculate the generation metrics **faithfulness** and **answer relevancy** for this example.\n",
    "\n",
    "The ragas libary conveniently abstracts the calculation of these metrics so we don't have to write redundant code but please review the following definitions in order to build intuition around what these metrics actually measure.\n",
    "\n",
    "Note: the following examples are paraphrased from the [ragas docs](https://docs.ragas.io/en/stable/concepts/metrics/index.html)\n",
    "\n",
    "------\n",
    "\n",
    "### Faithfulness\n",
    "\n",
    "An answer to a question can be said to be \"faithful\" if the **claims** that are made in the answer **can be inferred** from the **context**.\n",
    "\n",
    "#### Mathematically:\n",
    "\n",
    "$$\n",
    "Faithfullness\\ score = \\frac{Number\\ of\\ claims\\ in\\ the\\ generated\\ answer\\ that\\ can\\ be\\ inferred\\ from\\ the\\ given\\ context}{Total\\ number\\ of\\ claim\\ in\\ the\\ generated\\ answer}\n",
    "$$\n",
    "\n",
    "#### Example process:\n",
    "\n",
    "> Question: Where and when was Einstein born?\n",
    "> \n",
    "> Context: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time\n",
    ">\n",
    "> answer: Einstein was born in Germany on 20th March 1879.\n",
    "\n",
    "Step 1: Use LLM to break generated answer into individual statements.\n",
    "- “Einstein was born in Germany.”\n",
    "- “Einstein was born on 20th March 1879.”\n",
    "\n",
    "Step 2: For each statement use LLM to verify if it can be inferred from the context.\n",
    "- “Einstein was born in Germany.” => yes. \n",
    "- “Einstein was born on 20th March 1879.” => no.\n",
    "\n",
    "Step 3: plug into formula\n",
    "\n",
    "Number of claims inferred from context = 1\n",
    "Total number of claims = 2\n",
    "Faithfulness = 1/2\n",
    "\n",
    "### Answer Relevance\n",
    "\n",
    "An answer can be said to be relevant if it directly addresses the question (intuitively).\n",
    "\n",
    "#### Example process:\n",
    "\n",
    "1. Use an LLM to generate \"hypothetical\" questions to a given answer with the following prompt:\n",
    "\n",
    "    > Generate a question for the given answer.\n",
    "    > answer: [answer]\n",
    "\n",
    "2. Embed the generated \"hypothetical\" questions as vectors.\n",
    "3. Calculate the cosine similarity of the hypothetical questions and the original question, sum those similarities, and divide by n.\n",
    "\n",
    "With data:\n",
    "\n",
    "> Question: Where is France and what is it’s capital?\n",
    "> \n",
    "> answer: France is in western Europe.\n",
    "\n",
    "Step 1 - use LLM to create 'n' variants of question from the generated answer.\n",
    "\n",
    "- “In which part of Europe is France located?”\n",
    "- “What is the geographical location of France within Europe?”\n",
    "- “Can you identify the region of Europe where France is situated?”\n",
    "\n",
    "Step 2 - Calculate the mean cosine similarity between the generated questions and the actual question.\n",
    "\n",
    "## Now let's implement using our helper functions\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd9cabb4b0c448b08cad96d2ef3391a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ragas.metrics import faithfulness, answer_relevancy\n",
    "\n",
    "faithfulness_metrics = evaluate_dataset(eval_dataset, [faithfulness], llm, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72432636d3a44519b57329c66ded9c8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answer_relevancy_metrics = evaluate_dataset(eval_dataset, [answer_relevancy], llm, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.781229</td>\n",
       "      <td>0.938581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.362666</td>\n",
       "      <td>0.085342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.736997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.652778</td>\n",
       "      <td>0.926596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.975230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.994168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       faithfulness  answer_relevancy\n",
       "count     15.000000         15.000000\n",
       "mean       0.781229          0.938581\n",
       "std        0.362666          0.085342\n",
       "min        0.000000          0.736997\n",
       "25%        0.652778          0.926596\n",
       "50%        1.000000          0.975230\n",
       "75%        1.000000          0.994168\n",
       "max        1.000000          1.000000"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_metrics_default = faithfulness_metrics\n",
    "gen_metrics_default[\"answer_relevancy\"] = answer_relevancy_metrics[\"answer_relevancy\"]\n",
    "\n",
    "gen_metrics_default.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating retrieval metrics\n",
    "\n",
    "Retrieval metrics quantify how well the system performed at fetching the best possible context for generation. Like before please review the definitions below to understand what happens under-the-hood when we execute the evaluation code. \n",
    "\n",
    "-----\n",
    "\n",
    "### Context Relevance\n",
    "\n",
    "\"The context is considered relevant to the extent that it exclusively contains information that is needed to answer the question.\"\n",
    "\n",
    "#### Example process:\n",
    "\n",
    "1. Use the following LLM prompt to extract a subset of sentences necessary to answer the question. The context is defined as the formatted search result from the vector database.\n",
    "\n",
    "    > Please extract relevant sentences from\n",
    "    > the provided context that can potentially\n",
    "    > help answer the following `{question}`. If no\n",
    "    > relevant sentences are found, or if you\n",
    "    > believe the question cannot be answered\n",
    "    > from the given context, return the phrase\n",
    "    > \"Insufficient Information\". While extracting candidate sentences you’re not allowed to make any changes to sentences\n",
    "    > from given `{context}`.\n",
    "\n",
    "2. Compute the context relevance score = (number of extracted sentences) / (total number of sentences in context)\n",
    "\n",
    "Moving from the initial paper to the active evaluation library ragas there are a few more insightful metrics to evaluate. From the library [source](https://docs.ragas.io/en/stable/concepts/metrics/index.html) let's introduce `context precision` and `context recall`. \n",
    "\n",
    "### Context recall\n",
    "Context can be said to have high recall if retrieved context aligns with the ground truth answer.\n",
    "\n",
    "#### Mathematically:\n",
    "\n",
    "$$\n",
    "Context\\ recall = \\frac{Ground\\ Truth\\ sentences\\ that\\ can\\ be\\ attributed\\ to\\ context}{Total\\ number\\ of\\ sentences\\ in\\ the\\ ground\\ truth}\n",
    "$$\n",
    "\n",
    "#### Example process:\n",
    "\n",
    "Data:\n",
    "> question: Where is France and what is it’s capital?\n",
    "> ground truth answer: France is in Western Europe and its capital is Paris.\n",
    "> context: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history.\n",
    ">\n",
    "> Note: ground truth answer can be created by critic LLM or with own human labeled data set.\n",
    "\n",
    "Step 1 - use an LLM to break the ground truth down into individual statements:\n",
    "- `France is in Western Europe`\n",
    "- `Its capital is Paris`\n",
    "\n",
    "Step 2 - for each ground truth statement, use an LLM to determine if it can be attributed from the context.\n",
    "- `France is in Western Europe` => yes\n",
    "- `Its capital is Paris` => no\n",
    "\n",
    "\n",
    "Step 3 - plug in to formula\n",
    "\n",
    "context recall = (1 + 0) / 2 = 0.5\n",
    "\n",
    "### Context precision\n",
    "\n",
    "This metrics relates to how chunks are ranked in a response. Ideally the most relevant chunks are at the top.\n",
    "\n",
    "#### Mathematically:\n",
    "\n",
    "$$\n",
    "Context\\ Precision@k = \\frac{precision@k}{total\\ number\\ relevant\\ items\\ in\\ the\\ top\\ k\\ results}\n",
    "$$\n",
    "\n",
    "$$\n",
    "Precision@k = \\frac{true\\ positive@k}{true\\ positives@k + false\\ positives@k}\n",
    "$$\n",
    "\n",
    "#### Example process:\n",
    "\n",
    "Data:\n",
    "> Question: Where is France and what is it’s capital?\n",
    "> \n",
    "> Ground truth: France is in Western Europe and its capital is Paris.\n",
    "> \n",
    "> Context: [ “The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and”, “France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower”]\n",
    "\n",
    "Step 1 - for each chunk use the LLM to check if it's relevant or not to the ground truth answer.\n",
    "\n",
    "Step 2 - for each chunk in the context calculate the precision defined as: ``\n",
    "- `“The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and”` => precision = 0/1 or 0.\n",
    "- `“France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower”` => the precision would be (1) / (1 true positive + 1 false positive) = 0.5. \n",
    "\n",
    "\n",
    "Step 3 - calculate the overall context precision = (0 + 0.5) / 1 = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c076c3dc42cf49cf8d768dec225727d5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ragas.metrics import context_recall, context_precision\n",
    "\n",
    "context_recall_metrics = evaluate_dataset(eval_dataset, [context_recall], llm, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1055dffc473846a3b5f43895485be9a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "context_precision_metrics = evaluate_dataset(eval_dataset, [context_precision], llm, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_recall</th>\n",
       "      <th>context_precision</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.925926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.129099</td>\n",
       "      <td>0.145352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.916667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       context_recall  context_precision\n",
       "count       15.000000          15.000000\n",
       "mean         0.966667           0.925926\n",
       "std          0.129099           0.145352\n",
       "min          0.500000           0.500000\n",
       "25%          1.000000           0.916667\n",
       "50%          1.000000           1.000000\n",
       "75%          1.000000           1.000000\n",
       "max          1.000000           1.000000"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ret_metrics_default = context_recall_metrics\n",
    "ret_metrics_default[\"context_precision\"] = context_precision_metrics[\"context_precision\"]\n",
    "\n",
    "ret_metrics_default.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = ret_metrics_default\n",
    "metrics[\"faithfulness\"] = gen_metrics_default[\"faithfulness\"]\n",
    "metrics[\"answer_relevancy\"] = gen_metrics_default[\"answer_relevancy\"]\n",
    "\n",
    "metrics.to_csv(f\"resources/metrics_{CHUNK_SIZE}_{CHUNK_OVERLAP}.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# All together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_recall</th>\n",
       "      <th>context_precision</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>15.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.925926</td>\n",
       "      <td>0.781229</td>\n",
       "      <td>0.938581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.129099</td>\n",
       "      <td>0.145352</td>\n",
       "      <td>0.362666</td>\n",
       "      <td>0.085342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.736997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0.652778</td>\n",
       "      <td>0.926596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.975230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.994168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       context_recall  context_precision  faithfulness  answer_relevancy\n",
       "count       15.000000          15.000000     15.000000         15.000000\n",
       "mean         0.966667           0.925926      0.781229          0.938581\n",
       "std          0.129099           0.145352      0.362666          0.085342\n",
       "min          0.500000           0.500000      0.000000          0.736997\n",
       "25%          1.000000           0.916667      0.652778          0.926596\n",
       "50%          1.000000           1.000000      1.000000          0.975230\n",
       "75%          1.000000           1.000000      1.000000          0.994168\n",
       "max          1.000000           1.000000      1.000000          1.000000"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "Overall our RAG app showed pretty good performance. All values indicated above 0.6, which from anecdotal experience, is a reasonable lower-bound for performance however obviously higher values are more ideal. It is worth noting that generation metrics can be a bit more hazy in terms of ideal ranges since the LLM evaluation cannot yet capture the way a response feels to a user. For these metrics it's important to make sure they are not severely low however blind optimization to the top can result in a very uncreative chat experience which may or may not be ideal for the intended use case.\n",
    "\n",
    "## Review\n",
    "\n",
    "- we initialized our RAG app with data from a 10k document\n",
    "- generated a testset to evaluate \n",
    "- calculated both retrieval and generation metrics\n",
    "\n",
    "## Next steps\n",
    "\n",
    "Now that we know how to measure our system we can quickly and easily experiment with different techniques with a baseline in place to improve our systems.\n",
    "\n",
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redisvl.index import SearchIndex\n",
    "\n",
    "idx = SearchIndex.from_existing(\n",
    "    index_name,\n",
    "    redis_url=REDIS_URL\n",
    ")\n",
    "\n",
    "idx.delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}