Notebooks/03_BOW_Similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ee7a5c30",
   "metadata": {
    "papermill": {
     "duration": 0.01042,
     "end_time": "2023-02-20T20:26:22.817549",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.807129",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src='../assets/1/lang-pic.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "# 1. Introduction\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.1 NLP series</p>\n",
    "\n",
    "This is the **third in a series of notebooks** covering the **fundamentals of Natural Language Processing (NLP)**. I find that the best way to learn is by teaching others, hence why I am sharing my journey learning this field from scratch. I hope these notebooks can be helpful to you too. \n",
    "\n",
    "NLP series:\n",
    "\n",
    "1. [Tokenization](./01_Tokenization.ipynb)\n",
    "2. [Preprocessing](./02_Pre_Processing.ipynb)\n",
    "3. Bag of Words and Similarity\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/03_BOW_Similarity.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/03_BOW_Similarity.ipynb)\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.2 Outline</p>\n",
    "\n",
    "We have now seen how to tokenize and pre-process text. But to be able to use machine learning, we need to **convert this text into numbers**. In this notebook, we'll see one way to do this via a **basic bag-of-words** and discuss some of its variants. \n",
    "\n",
    "We'll then learn how to measure the **similarity between two documents** through one of the most popular approaches, namely **cosine similarity**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "533b1345",
   "metadata": {
    "papermill": {
     "duration": 0.009552,
     "end_time": "2023-02-20T20:26:22.836571",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.827019",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Bag-of-Words\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.1 The idea</p>\n",
    "\n",
    "After tokenization and pre-processing, we are left with **variable length** sequences of text, but the problem is machine learning algorithms require **fixed length** vectors of numbers. \n",
    "\n",
    "The **simplest** approach to overcome this is by using a **bag-of-words**, which simply **counts how many times each word appears** in a document. It's called a **bag** because the **order of the words is ignored** - we only care about whether a word appeared or not. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/3/basicbow.png' width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "The linguistic reasoning behind this approach is that **similar documents share similar vocabularies**. For example, football articles will often use words like *score*, *pass*, *team* whereas weather reports will use a completely different set of words like *rain*, *sun*, *umbrella*.\n",
    "\n",
    "We might want to **remove stop words** (common words that have little meaning like *the*, *of*, *how*) to make it easier to identify similar documents as these will be in pretty much all documents. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f316e827",
   "metadata": {
    "papermill": {
     "duration": 0.00896,
     "end_time": "2023-02-20T20:26:22.854538",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.845578",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.2 Binary Bag-of-Words</p>\n",
    "\n",
    "For now, we'll focus on the **binary** version of a bag-of-words. This just indicates **whether a word appeared or not**, ignoring word order and word frequency.\n",
    "\n",
    "<center>\n",
    "<img src='../assets/3/binarybow.jpg' width=600>\n",
    "</center>\n",
    "\n",
    "Each **row** in a **binary bag-of-words matrix** corresponds to a **single document** in the corpus. Each **column** corresponds to a **token** in the vocabulary. Note that the order of the tokens isn't important but it does need to be **fixed beforehand** when building the vocabulary.  \n",
    "\n",
    "To **construct** the matrix, we place a 1 in entry (i,j) if and only if the j-th token appears in the i-th document and a 0 otherwise. \n",
    "\n",
    "For a **general** bag-of-words, the (i,j) entry would instead be the **frequency** of the j-th token in the i-th document (but we will see there are better ways to encode frequency later)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b71e9590",
   "metadata": {
    "papermill": {
     "duration": 0.008651,
     "end_time": "2023-02-20T20:26:22.872352",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.863701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. Similarity\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.1 Vector Space Model</p>\n",
    "\n",
    "We have gone from thinking of documents as a sequence of words to **points in a multi-dimensional vector space**. Importantly, the dimension of this space if **fixed**, i.e. each vector has the same length.\n",
    "\n",
    "<center>\n",
    "<img src='../assets/3/unitcube.png' width=400>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "This is very useful because it now allows us to **measure the distances** between these points among other things. Points (documents) that are close together will correspond to documents being **similar** in their vocabularies. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b4bdb58",
   "metadata": {
    "papermill": {
     "duration": 0.008661,
     "end_time": "2023-02-20T20:26:22.891809",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.883148",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.2 Cosine Similarity</p>\n",
    "\n",
    "There are many **metrics** we could use to measure how 'close' two points are. For example, we could consider using the Euclidean distance, Manhattan distance or even Hamming distance. However, if documents in the same corpus have very different lengths, or the vocabulary is extremely large, these metrics become less reliable. \n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\cos(\\theta) = \\frac{a \\cdot b}{\\|a\\| \\|b\\|}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "Instead, in the NLP domain it is much more common to use **Cosine Similarity**. This measures the **cosine of the angle** between any two points (more precisely their vectors starting from the origin). The **closer the score 1**, the smaller the angle between the vectors and the **more similar** the documents are. \n",
    "\n",
    "<br>\n",
    "<center>\n",
    "<img src='../assets/3/cosine-similarity-vectors-original.jpg' width=800>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "Note that the **threshold** used to decide whether two documents are similar will **change depending on the application** and it can be anywhere between 0 and 1. It will be sensitive to how we pre-process our text. Lemmatization and stop word removal can help reduce the size of the vocabulary making it easier to identify similar documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01c445b2",
   "metadata": {
    "papermill": {
     "duration": 0.008747,
     "end_time": "2023-02-20T20:26:22.909611",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.900864",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4. Encoding context\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.1 Drawbacks to Bag-of-Words</p>\n",
    "\n",
    "Whilst using a bag-of-words is a great tool for **simple** NLP applications, it does have a number of drawbacks that we need to be aware about.\n",
    "\n",
    "* There is no way to handle **Out-of-Vocabulary** (OOV) words. If a new word appears in a later document, it will just be dropped.\n",
    "* It creates **sparse matrices** which can be inefficient, although we can overcome this by using a dictionary representation. \n",
    "* It isn't able to capture similarity between **synonyms**. \n",
    "* Word order is lost so words have **no relationship** to each other. For example, \"man eats bread\" is very different to \"bread eats man\" but they would have the same representations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f66b87e",
   "metadata": {
    "papermill": {
     "duration": 0.008656,
     "end_time": "2023-02-20T20:26:22.927304",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.918648",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.2 n-grams</p>\n",
    "\n",
    "One way to get around the problem of losing word order information is to use **n-grams**. This is when we group **chunks of n tokens** together to behave as if they were a single token. \n",
    "\n",
    "A 2-gram (aka **bigram**) would have 2 tokens per chunk, a 3-gram (aka **trigram**) would have 3 tokens per chuck, etc. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/3/8ARA1.png' width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "This helps us capture **some context** that using single tokens wouldn't. The **vocabulary** then becomes the **collection of n-grams** produced. Depending on the application, you might want to use unigrams and bigrams together or just bigrams. You could even filter out bigrams that aren't useful for your application (e.g. only keep highly frequent or noun-noun bigrams).\n",
    "\n",
    "Measuring **similarity** is exactly the **same as before**. However, using n-grams can **significantly increase the size of the vocabulary** making computations slower. There is therefore a tradeoff between contextual information added and increased computational time for modelling. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08eefb3d",
   "metadata": {
    "papermill": {
     "duration": 0.00869,
     "end_time": "2023-02-20T20:26:22.945178",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.936488",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5. Application\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.1 Bag-of-Words using sklearn</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "490536c8",
   "metadata": {
    "papermill": {
     "duration": 0.008643,
     "end_time": "2023-02-20T20:26:22.962846",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.954203",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Import the **libraries**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e03333dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:22.983236Z",
     "iopub.status.busy": "2023-02-20T20:26:22.982428Z",
     "iopub.status.idle": "2023-02-20T20:26:35.799727Z",
     "shell.execute_reply": "2023-02-20T20:26:35.798343Z"
    },
    "papermill": {
     "duration": 12.83109,
     "end_time": "2023-02-20T20:26:35.802954",
     "exception": false,
     "start_time": "2023-02-20T20:26:22.971864",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020ffe04",
   "metadata": {
    "papermill": {
     "duration": 0.008675,
     "end_time": "2023-02-20T20:26:35.820878",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.812203",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Define the **corpus**. Here we use some of the top news stories from 2022."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1d5171ff",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:35.840812Z",
     "iopub.status.busy": "2023-02-20T20:26:35.840144Z",
     "iopub.status.idle": "2023-02-20T20:26:35.845872Z",
     "shell.execute_reply": "2023-02-20T20:26:35.844509Z"
    },
    "papermill": {
     "duration": 0.018568,
     "end_time": "2023-02-20T20:26:35.848428",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.829860",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# A corpus containing a collection of sentences\n",
    "corpus = [\n",
    "    \"Inflation surges around the world.\",\n",
    "    \"The Omicron coronavirus variant spreads.\",\n",
    "    \"World population exceeds 8 billion.\",\n",
    "    \"AI predicts protein structures.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06faa991",
   "metadata": {
    "papermill": {
     "duration": 0.008491,
     "end_time": "2023-02-20T20:26:35.866245",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.857754",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use **sklearn's CountVectorizer** to create a bag-of-words matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "34ad1ed5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:35.886244Z",
     "iopub.status.busy": "2023-02-20T20:26:35.885823Z",
     "iopub.status.idle": "2023-02-20T20:26:35.891048Z",
     "shell.execute_reply": "2023-02-20T20:26:35.889655Z"
    },
    "papermill": {
     "duration": 0.018309,
     "end_time": "2023-02-20T20:26:35.893527",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.875218",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize vectorizer\n",
    "vectorizer = CountVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84d33b01",
   "metadata": {
    "papermill": {
     "duration": 0.009016,
     "end_time": "2023-02-20T20:26:35.911481",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.902465",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The `.fit_transform` method learns a **vocabulary** from the corpus and returns the **bag-of-words matrix**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a999f08e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:35.931879Z",
     "iopub.status.busy": "2023-02-20T20:26:35.930755Z",
     "iopub.status.idle": "2023-02-20T20:26:35.940526Z",
     "shell.execute_reply": "2023-02-20T20:26:35.939390Z"
    },
    "papermill": {
     "duration": 0.022631,
     "end_time": "2023-02-20T20:26:35.943147",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.920516",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fit vectorizer to corpus\n",
    "bow = vectorizer.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfac95f",
   "metadata": {
    "papermill": {
     "duration": 0.00858,
     "end_time": "2023-02-20T20:26:35.960725",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.952145",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can see the **vocabulary dictionary** mapping using the `.vocabulary_` method. We could also use the `.get_feature_names_out()` to see just the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bef9eb24",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:35.980664Z",
     "iopub.status.busy": "2023-02-20T20:26:35.979635Z",
     "iopub.status.idle": "2023-02-20T20:26:35.990144Z",
     "shell.execute_reply": "2023-02-20T20:26:35.988945Z"
    },
    "papermill": {
     "duration": 0.022887,
     "end_time": "2023-02-20T20:26:35.992481",
     "exception": false,
     "start_time": "2023-02-20T20:26:35.969594",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'inflation': 5,\n",
       " 'surges': 12,\n",
       " 'around': 1,\n",
       " 'the': 13,\n",
       " 'world': 15,\n",
       " 'omicron': 6,\n",
       " 'coronavirus': 3,\n",
       " 'variant': 14,\n",
       " 'spreads': 10,\n",
       " 'population': 7,\n",
       " 'exceeds': 4,\n",
       " 'billion': 2,\n",
       " 'ai': 0,\n",
       " 'predicts': 8,\n",
       " 'protein': 9,\n",
       " 'structures': 11}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View vocabulary\n",
    "vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02d4c868",
   "metadata": {
    "papermill": {
     "duration": 0.008647,
     "end_time": "2023-02-20T20:26:36.010599",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.001952",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The vectorizer **output** is a **compressed sparse row matrix**, which is done to **improve memory efficiency**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "74d658c6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:36.031969Z",
     "iopub.status.busy": "2023-02-20T20:26:36.031524Z",
     "iopub.status.idle": "2023-02-20T20:26:36.038445Z",
     "shell.execute_reply": "2023-02-20T20:26:36.037332Z"
    },
    "papermill": {
     "duration": 0.020547,
     "end_time": "2023-02-20T20:26:36.040891",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.020344",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x16 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 18 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7619b661",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:36.061208Z",
     "iopub.status.busy": "2023-02-20T20:26:36.060685Z",
     "iopub.status.idle": "2023-02-20T20:26:36.068220Z",
     "shell.execute_reply": "2023-02-20T20:26:36.067275Z"
    },
    "papermill": {
     "duration": 0.020542,
     "end_time": "2023-02-20T20:26:36.070763",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.050221",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 5)\t1\n",
      "  (0, 12)\t1\n",
      "  (0, 1)\t1\n",
      "  (0, 13)\t1\n",
      "  (0, 15)\t1\n",
      "  (1, 13)\t1\n",
      "  (1, 6)\t1\n",
      "  (1, 3)\t1\n",
      "  (1, 14)\t1\n",
      "  (1, 10)\t1\n",
      "  (2, 15)\t1\n",
      "  (2, 7)\t1\n",
      "  (2, 4)\t1\n",
      "  (2, 2)\t1\n",
      "  (3, 0)\t1\n",
      "  (3, 8)\t1\n",
      "  (3, 9)\t1\n",
      "  (3, 11)\t1\n"
     ]
    }
   ],
   "source": [
    "print(bow)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b295dec3",
   "metadata": {
    "papermill": {
     "duration": 0.009087,
     "end_time": "2023-02-20T20:26:36.089417",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.080330",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To convert the sparse matrix into a **dense matrix**, we call the `.toarray()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3a511976",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:36.109856Z",
     "iopub.status.busy": "2023-02-20T20:26:36.109404Z",
     "iopub.status.idle": "2023-02-20T20:26:36.117204Z",
     "shell.execute_reply": "2023-02-20T20:26:36.115995Z"
    },
    "papermill": {
     "duration": 0.020823,
     "end_time": "2023-02-20T20:26:36.119671",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.098848",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1],\n",
       "       [0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0],\n",
       "       [0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],\n",
       "       [1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dense matrix representation\n",
    "bow.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d68bd14",
   "metadata": {
    "papermill": {
     "duration": 0.009037,
     "end_time": "2023-02-20T20:26:36.138627",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.129590",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Notice how sklearn lower-cased and **tokenized the corpus for us**. Next we will do the same using our own **custom tokenizer**, which will give us **more control** over how the text is pre-processed. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9422a0a",
   "metadata": {
    "papermill": {
     "duration": 0.00903,
     "end_time": "2023-02-20T20:26:36.157123",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.148093",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.2 Custom tokenizer using spacy</p>\n",
    "\n",
    "To do this, we need to define our custom tokenizer as a **function** that given a document, **returns a list of tokens**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0344fd70",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:36.177808Z",
     "iopub.status.busy": "2023-02-20T20:26:36.177395Z",
     "iopub.status.idle": "2023-02-20T20:26:37.046385Z",
     "shell.execute_reply": "2023-02-20T20:26:37.045101Z"
    },
    "papermill": {
     "duration": 0.882901,
     "end_time": "2023-02-20T20:26:37.049558",
     "exception": false,
     "start_time": "2023-02-20T20:26:36.166657",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load english language model\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Define custom tokenizer (remove stop words and punctuation and apply lemmatization)\n",
    "def custom_tokenizer(doc):\n",
    "    return [t.lemma_ for t in nlp(doc) if (not t.is_punct) and (not t.is_stop)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de078322",
   "metadata": {
    "papermill": {
     "duration": 0.009094,
     "end_time": "2023-02-20T20:26:37.068537",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.059443",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The tokenizer is then passed as a **callback function** inside the count vectorizer. We also set binary equal to true to produce a **binary** bag-of-words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4bf12d49",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.089399Z",
     "iopub.status.busy": "2023-02-20T20:26:37.088945Z",
     "iopub.status.idle": "2023-02-20T20:26:37.143032Z",
     "shell.execute_reply": "2023-02-20T20:26:37.141496Z"
    },
    "papermill": {
     "duration": 0.068061,
     "end_time": "2023-02-20T20:26:37.146147",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.078086",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Pass tokenizer as callback function to countvectorizer\n",
    "vectorizer = CountVectorizer(tokenizer=custom_tokenizer, binary=True)\n",
    "\n",
    "# Fit vectorizer to corpus\n",
    "bow = vectorizer.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68804981",
   "metadata": {
    "papermill": {
     "duration": 0.009161,
     "end_time": "2023-02-20T20:26:37.164922",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.155761",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can view the resulting **vocabulary** and matrix the same way as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7cd0259b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.185752Z",
     "iopub.status.busy": "2023-02-20T20:26:37.185295Z",
     "iopub.status.idle": "2023-02-20T20:26:37.193573Z",
     "shell.execute_reply": "2023-02-20T20:26:37.192393Z"
    },
    "papermill": {
     "duration": 0.021798,
     "end_time": "2023-02-20T20:26:37.196260",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.174462",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'inflation': 5,\n",
       " 'surge': 12,\n",
       " 'world': 14,\n",
       " 'omicron': 6,\n",
       " 'coronavirus': 3,\n",
       " 'variant': 13,\n",
       " 'spread': 10,\n",
       " 'population': 7,\n",
       " 'exceed': 4,\n",
       " '8': 0,\n",
       " 'billion': 2,\n",
       " 'ai': 1,\n",
       " 'predict': 8,\n",
       " 'protein': 9,\n",
       " 'structure': 11}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Vocabulary\n",
    "vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ecd503d7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.217653Z",
     "iopub.status.busy": "2023-02-20T20:26:37.217245Z",
     "iopub.status.idle": "2023-02-20T20:26:37.225666Z",
     "shell.execute_reply": "2023-02-20T20:26:37.224301Z"
    },
    "papermill": {
     "duration": 0.02231,
     "end_time": "2023-02-20T20:26:37.228355",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.206045",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1],\n",
       "       [0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0],\n",
       "       [1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1],\n",
       "       [0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dense matrix representation\n",
    "bow.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90c17cd4",
   "metadata": {
    "papermill": {
     "duration": 0.009533,
     "end_time": "2023-02-20T20:26:37.248180",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.238647",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The sparse matrix can be **sliced and indexed** like a normal array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fb22e3b9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.269425Z",
     "iopub.status.busy": "2023-02-20T20:26:37.268998Z",
     "iopub.status.idle": "2023-02-20T20:26:37.277379Z",
     "shell.execute_reply": "2023-02-20T20:26:37.276052Z"
    },
    "papermill": {
     "duration": 0.021918,
     "end_time": "2023-02-20T20:26:37.279871",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.257953",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (1, 3)\t1\n",
      "  (2, 0)\t1\n",
      "  (2, 2)\t1\n",
      "  (3, 1)\t1\n"
     ]
    }
   ],
   "source": [
    "# Sparse slice\n",
    "print(bow[:,0:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8451de04",
   "metadata": {
    "papermill": {
     "duration": 0.009506,
     "end_time": "2023-02-20T20:26:37.299823",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.290317",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.3 Document Similarity</p>\n",
    "\n",
    "Here we will measure the **cosine similarity** between the documents in our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b67402d3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.323363Z",
     "iopub.status.busy": "2023-02-20T20:26:37.322930Z",
     "iopub.status.idle": "2023-02-20T20:26:37.329317Z",
     "shell.execute_reply": "2023-02-20T20:26:37.327933Z"
    },
    "papermill": {
     "duration": 0.021032,
     "end_time": "2023-02-20T20:26:37.331991",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.310959",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cosine similarity using numpy\n",
    "def cosine_sim(a,b):\n",
    "    return np.dot(a,b)/(np.linalg.norm(a)*np.linalg.norm(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5c6703e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.354543Z",
     "iopub.status.busy": "2023-02-20T20:26:37.354134Z",
     "iopub.status.idle": "2023-02-20T20:26:37.361942Z",
     "shell.execute_reply": "2023-02-20T20:26:37.360577Z"
    },
    "papermill": {
     "duration": 0.021783,
     "end_time": "2023-02-20T20:26:37.364614",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.342831",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Omicron coronavirus variant spreads.\n",
      "AI predicts protein structures.\n",
      "Similarity score: 0.000\n"
     ]
    }
   ],
   "source": [
    "# Similarity between two documents\n",
    "print(corpus[1])\n",
    "print(corpus[3])\n",
    "print(f'Similarity score: {cosine_sim(bow[1].toarray().squeeze(),bow[3].toarray().squeeze()):.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fabe75b8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.386779Z",
     "iopub.status.busy": "2023-02-20T20:26:37.386367Z",
     "iopub.status.idle": "2023-02-20T20:26:37.394173Z",
     "shell.execute_reply": "2023-02-20T20:26:37.392634Z"
    },
    "papermill": {
     "duration": 0.022003,
     "end_time": "2023-02-20T20:26:37.397017",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.375014",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inflation surges around the world.\n",
      "World population exceeds 8 billion.\n",
      "Similarity score: 0.258\n"
     ]
    }
   ],
   "source": [
    "# Similarity between two documents\n",
    "print(corpus[0])\n",
    "print(corpus[2])\n",
    "print(f'Similarity score: {cosine_sim(bow[0].toarray().squeeze(),bow[2].toarray().squeeze()):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78427adb",
   "metadata": {
    "papermill": {
     "duration": 0.009663,
     "end_time": "2023-02-20T20:26:37.416844",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.407181",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can also use sklearn's `cosine_similarity`. This calculates all the **pairwise similarities** and returns the result in a matrix indexed by the documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "138b7b2f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.439305Z",
     "iopub.status.busy": "2023-02-20T20:26:37.438897Z",
     "iopub.status.idle": "2023-02-20T20:26:37.449551Z",
     "shell.execute_reply": "2023-02-20T20:26:37.448323Z"
    },
    "papermill": {
     "duration": 0.025535,
     "end_time": "2023-02-20T20:26:37.452364",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.426829",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.         0.         0.25819889 0.        ]\n",
      " [0.         1.         0.         0.        ]\n",
      " [0.25819889 0.         1.         0.        ]\n",
      " [0.         0.         0.         1.        ]]\n"
     ]
    }
   ],
   "source": [
    "# cosine_similarity takes either array-likes or sparse matrices\n",
    "print(cosine_similarity(bow))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2206adf0",
   "metadata": {
    "papermill": {
     "duration": 0.009759,
     "end_time": "2023-02-20T20:26:37.472221",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.462462",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.4 n-grams</p>\n",
    "\n",
    "Finally, we will build a bag-of-words matrix using **n-grams**. To do this, we can pass the `ngram_range` parameter in countvectorizer. It takes in a tuple, with the **first entry** indicating the **minimum** chunk size and the **second entry** indicating the **maximum** chunk size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f854df42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.494203Z",
     "iopub.status.busy": "2023-02-20T20:26:37.493768Z",
     "iopub.status.idle": "2023-02-20T20:26:37.531349Z",
     "shell.execute_reply": "2023-02-20T20:26:37.530248Z"
    },
    "papermill": {
     "duration": 0.051787,
     "end_time": "2023-02-20T20:26:37.534216",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.482429",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary: 27\n",
      "{'inflation': 11, 'surge': 21, 'world': 25, 'inflation surge': 12, 'surge world': 22, 'Omicron': 4, 'coronavirus': 7, 'variant': 23, 'spread': 19, 'Omicron coronavirus': 5, 'coronavirus variant': 8, 'variant spread': 24, 'population': 13, 'exceed': 9, '8': 0, 'billion': 6, 'world population': 26, 'population exceed': 14, 'exceed 8': 10, '8 billion': 1, 'AI': 2, 'predict': 15, 'protein': 17, 'structure': 20, 'AI predict': 3, 'predict protein': 16, 'protein structure': 18}\n"
     ]
    }
   ],
   "source": [
    "# Unigrams and bigrams with ngram_range=(1,2)\n",
    "vectorizer = CountVectorizer(tokenizer=custom_tokenizer, lowercase=False, binary=True, ngram_range=(1,2))\n",
    "\n",
    "# Fit vectorizer to corpus\n",
    "unibigrams = vectorizer.fit_transform(corpus)\n",
    "\n",
    "# Print vocabulary size\n",
    "print(f'Size of vocabulary: {len(vectorizer.get_feature_names_out())}')\n",
    "\n",
    "# Print vocabulary\n",
    "print(vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7d07fa58",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:26:37.557046Z",
     "iopub.status.busy": "2023-02-20T20:26:37.556623Z",
     "iopub.status.idle": "2023-02-20T20:26:37.593702Z",
     "shell.execute_reply": "2023-02-20T20:26:37.592525Z"
    },
    "papermill": {
     "duration": 0.052004,
     "end_time": "2023-02-20T20:26:37.596791",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.544787",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary: 12\n",
      "{'inflation surge': 5, 'surge world': 9, 'Omicron coronavirus': 2, 'coronavirus variant': 3, 'variant spread': 10, 'world population': 11, 'population exceed': 6, 'exceed 8': 4, '8 billion': 0, 'AI predict': 1, 'predict protein': 7, 'protein structure': 8}\n"
     ]
    }
   ],
   "source": [
    "# Only bigrams with ngram_range=(2,2)\n",
    "vectorizer = CountVectorizer(tokenizer=custom_tokenizer, lowercase=False, binary=True, ngram_range=(2,2))\n",
    "\n",
    "# Fit vectorizer to corpus\n",
    "bigrams = vectorizer.fit_transform(corpus)\n",
    "\n",
    "# Print vocabulary size\n",
    "print(f'Size of vocabulary: {len(vectorizer.get_feature_names_out())}')\n",
    "\n",
    "# Print vocabulary\n",
    "print(vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74533f8e",
   "metadata": {
    "papermill": {
     "duration": 0.009863,
     "end_time": "2023-02-20T20:26:37.617101",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.607238",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 6. Conclusion\n",
    "\n",
    "In this notebook, we saw how to convert **sequences of text** to **vectors of numbers** by using a basic **bag-of-words**, a process sometimes called **vectorization**.  Whilst this can be used for simple applications, it has a number of **limitations** including **losing word order information**. This can be partially resolved by using **n-grams** but it comes at the cost of **increasing** the size of our vocabulary. \n",
    "\n",
    "We also investigated how to measure document similarity using the popular **cosine similarity** metric. We will build on these ideas in future notebooks. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beccc842",
   "metadata": {
    "papermill": {
     "duration": 0.009907,
     "end_time": "2023-02-20T20:26:37.637827",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.627920",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**References:**\n",
    "* [NLP demystified](https://www.nlpdemystified.org/)\n",
    "\n",
    "### Coming UP\n",
    "#### [4. TF-IDF and Document Search](./04_TFIDF_DocSearch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88b9ae45",
   "metadata": {
    "papermill": {
     "duration": 0.010003,
     "end_time": "2023-02-20T20:26:37.658095",
     "exception": false,
     "start_time": "2023-02-20T20:26:37.648092",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 28.842881,
   "end_time": "2023-02-20T20:26:40.760365",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-02-20T20:26:11.917484",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}