assignment4.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tamnhu2913/Applied-Text-Mining-in-Python/blob/main/assignment4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "b562fcdae0cf6bf26c71c079ad6296fb",
          "grade": false,
          "grade_id": "cell-9aeed736a0e899b5",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "K-80OcdOADBy"
      },
      "source": [
        "# Assignment 4 - Document Similarity & Topic Modelling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "a45a724e7fe9eda42e35d96e051bcaa4",
          "grade": false,
          "grade_id": "cell-a932c63a4c180b33",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "4fDA7ZxcADB1"
      },
      "source": [
        "## Part 1 - Document Similarity\n",
        "\n",
        "For the first part of this assignment, you will complete the functions `doc_to_synsets` and `similarity_score` which will be used by `document_path_similarity` to find the path similarity between two documents.\n",
        "\n",
        "The following functions are provided:\n",
        "* **`convert_tag:`** converts the tag given by `nltk.pos_tag` to a tag used by `wordnet.synsets`. You will need to use this function in `doc_to_synsets`.\n",
        "* **`document_path_similarity:`** computes the symmetrical path similarity between two documents by finding the synsets in each document using `doc_to_synsets`, then computing similarities using `similarity_score`.\n",
        "\n",
        "You will need to finish writing the following functions:\n",
        "* **`doc_to_synsets:`** returns a list of synsets in document. This function should first tokenize and part of speech tag the document using `nltk.word_tokenize` and `nltk.pos_tag`. Then it should find each tokens corresponding synset using `wn.synsets(token, wordnet_tag)`. The first synset match should be used. If there is no match, that token is skipped.\n",
        "* **`similarity_score:`** returns the normalized similarity score of a list of synsets (s1) onto a second list of synsets (s2). For each synset in s1, find the synset in s2 with the largest similarity value. Sum all of the largest similarity values together and normalize this value by dividing it by the number of largest similarity values found. Be careful with data types, which should be floats. Missing values should be ignored.\n",
        "\n",
        "Once doc_to_synsets and similarity_score have been completed, submit to the autograder which will run a test to check that these functions are running correctly.\n",
        "\n",
        "*Do not modify the functions `convert_tag` and `document_path_similarity`.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "e4e34c4781de9b9d593a31c58fea606c",
          "grade": false,
          "grade_id": "cell-2c29841f8b54b400",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "_OlX_BI4ADB3"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "import numpy as np\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')\n",
        "from nltk.corpus import wordnet as wn\n",
        "import pandas as pd\n",
        "nltk.data.path.append(\"assets/\")\n",
        "\n",
        "def convert_tag(tag):\n",
        "    \"\"\"Convert the tag given by nltk.pos_tag to the tag used by wordnet.synsets\"\"\"\n",
        "\n",
        "    tag_dict = {'N': 'n', 'J': 'a', 'R': 'r', 'V': 'v'}\n",
        "    try:\n",
        "        return tag_dict[tag[0]]\n",
        "    except KeyError:\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "654d281deb3c2923c393335ccb2e789e",
          "grade": false,
          "grade_id": "cell-22d836339ad6be8a",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "7oS_NMzAADB5"
      },
      "outputs": [],
      "source": [
        "def doc_to_synsets(doc):\n",
        "    \"\"\"\n",
        "    Returns a list of synsets in document.\n",
        "\n",
        "    Tokenizes and tags the words in the document doc.\n",
        "    Then finds the first synset for each word/tag combination.\n",
        "    If a synset is not found for that combination it is skipped.\n",
        "\n",
        "    Args:\n",
        "        doc: string to be converted\n",
        "\n",
        "    Returns:\n",
        "        list of synsets\n",
        "\n",
        "    Example:\n",
        "        doc_to_synsets('Fish are friends.')\n",
        "        Out: [Synset('fish.n.01'), Synset('be.v.01'), Synset('friend.n.01')]\n",
        "    \"\"\"\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    tokens = nltk.word_tokenize(doc)\n",
        "    pos_tags = nltk.pos_tag(tokens)\n",
        "    syns = []\n",
        "    for token, tag in pos_tags:\n",
        "      cv_tag = convert_tag(tag)\n",
        "      wntag = wn.synsets(token, cv_tag)\n",
        "      if wntag:\n",
        "        syns.append(wntag[0])\n",
        "    return syns\n",
        "\n",
        "\n",
        "def similarity_score(s1, s2):\n",
        "    \"\"\"\n",
        "    Calculate the normalized similarity score of s1 onto s2\n",
        "\n",
        "    For each synset in s1, finds the synset in s2 with the largest similarity value.\n",
        "    Sum of all of the largest similarity values and normalize this value by dividing it by the\n",
        "    number of largest similarity values found.\n",
        "\n",
        "    Args:\n",
        "        s1, s2: list of synsets from doc_to_synsets\n",
        "\n",
        "    Returns:\n",
        "        normalized similarity score of s1 onto s2\n",
        "\n",
        "    Example:\n",
        "        synsets1 = doc_to_synsets('I like cats')\n",
        "        synsets2 = doc_to_synsets('I like dogs')\n",
        "        similarity_score(synsets1, synsets2)\n",
        "        Out: 0.7333333333333333\n",
        "    \"\"\"\n",
        "\n",
        "def similarity_score(s1, s2):\n",
        "  score = []\n",
        "  for syn1 in s1:\n",
        "    s1_score = [syn1.path_similarity(syn2) for syn2 in s2]\n",
        "    if s1_score:\n",
        "      score.append(np.nanmax(s1_score))\n",
        "\n",
        "  if score:\n",
        "    return np.mean(score)\n",
        "  else:\n",
        "    return 0.0\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "bfc6773bd020f674a6501b97e9e20079",
          "grade": false,
          "grade_id": "cell-314b4a30aa06ad7e",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "IMrQ14z1ADB6"
      },
      "outputs": [],
      "source": [
        "def document_path_similarity(doc1, doc2):\n",
        "    \"\"\"Finds the symmetrical similarity between doc1 and doc2\"\"\"\n",
        "\n",
        "    synsets1 = doc_to_synsets(doc1)\n",
        "    synsets2 = doc_to_synsets(doc2)\n",
        "\n",
        "    return (similarity_score(synsets1, synsets2) + similarity_score(synsets2, synsets1)) / 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "ff03c992760f7041ff7c19303da36d04",
          "grade": true,
          "grade_id": "cell-8513adbb001100a6",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "lLNC2ejuADB6"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "510e481c575da27a1f9b30e18e205322",
          "grade": false,
          "grade_id": "cell-e0e4fb9c6ee002af",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "pIhA8Y1XADB7"
      },
      "source": [
        "`paraphrases` is a DataFrame which contains the following columns: `Quality`, `D1`, and `D2`.\n",
        "\n",
        "`Quality` is an indicator variable which indicates if the two documents `D1` and `D2` are paraphrases of one another (1 for paraphrase, 0 for not paraphrase)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "bb713a63ffb7b1b3471c7a1e99725201",
          "grade": false,
          "grade_id": "cell-4c40d18c6c90bcf9",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "CLRHUKaFADB7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "187218b8-2e37-4b74-ac53-fb257143c61e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Quality                                                 D1  \\\n",
              "0        1  Ms Stewart, the chief executive, was not expec...   \n",
              "1        1  After more than two years' detention under the...   \n",
              "2        1  \"It still remains to be seen whether the reven...   \n",
              "3        0  And it's going to be a wild ride,\" said Allan ...   \n",
              "4        1  The cards are issued by Mexico's consulates to...   \n",
              "\n",
              "                                                  D2  \n",
              "0  Ms Stewart, 61, its chief executive officer an...  \n",
              "1  After more than two years in detention by the ...  \n",
              "2  \"It remains to be seen whether the revenue rec...  \n",
              "3  Now the rest is just mechanical,\" said Allan H...  \n",
              "4  The card is issued by Mexico's consulates to i...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9e748023-3fb5-4ef4-b726-e07d82febb4e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Quality</th>\n",
              "      <th>D1</th>\n",
              "      <th>D2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Ms Stewart, the chief executive, was not expec...</td>\n",
              "      <td>Ms Stewart, 61, its chief executive officer an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>After more than two years' detention under the...</td>\n",
              "      <td>After more than two years in detention by the ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>\"It still remains to be seen whether the reven...</td>\n",
              "      <td>\"It remains to be seen whether the revenue rec...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>And it's going to be a wild ride,\" said Allan ...</td>\n",
              "      <td>Now the rest is just mechanical,\" said Allan H...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>The cards are issued by Mexico's consulates to...</td>\n",
              "      <td>The card is issued by Mexico's consulates to i...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9e748023-3fb5-4ef4-b726-e07d82febb4e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9e748023-3fb5-4ef4-b726-e07d82febb4e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9e748023-3fb5-4ef4-b726-e07d82febb4e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-14fef303-3df6-47e4-8465-71f9d7dc7e50\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-14fef303-3df6-47e4-8465-71f9d7dc7e50')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-14fef303-3df6-47e4-8465-71f9d7dc7e50 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "paraphrases",
              "summary": "{\n  \"name\": \"paraphrases\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"Quality\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"D1\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"Ms Stewart, the chief executive, was not expected to attend.\",\n          \"Sun was the lone major vendor to see its shipments decline, falling 2.9 percent to 59,692 units.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"D2\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"Ms Stewart, 61, its chief executive officer and chairwoman, did not attend.\\n\",\n          \"IBM (NYSE: IBM)  was the fastest-growing vendor, with sales jumping 37 percent to 220,000 units.\\n\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "# Use this dataframe for questions most_similar_docs and label_accuracy\n",
        "paraphrases = pd.read_csv('assets/paraphrases.csv')\n",
        "paraphrases.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "dc3742e572649c3961a1400794bf7080",
          "grade": false,
          "grade_id": "cell-bd634ad070710c0a",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "-nlC6AVDADB8"
      },
      "source": [
        "___\n",
        "\n",
        "### most_similar_docs\n",
        "\n",
        "Using `document_path_similarity`, find the pair of documents in paraphrases which has the maximum similarity score.\n",
        "\n",
        "*This function should return a tuple `(D1, D2, similarity_score)`*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "53240536e14a8ea515062fba69ee6816",
          "grade": false,
          "grade_id": "cell-a9043a8e87723e7f",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "WFis9uBKADB8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "212b26fb-ac73-4ee9-a056-003dad67a400"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('\"Indeed, Iran should be put on notice that efforts to try to remake Iraq in their image will be aggressively put down,\" he said.',\n",
              " '\"Iran should be on notice that attempts to remake Iraq in Iran\\'s image will be aggressively put down,\" he said.\\n',\n",
              " 0.9590643274853801)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "def most_similar_docs():\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "  data = paraphrases.copy()\n",
        "  data['score'] = data.apply(lambda x: document_path_similarity(x['D1'], x['D2']), axis=1)\n",
        "  max_value = data.loc[data['score'].idxmax(), ['D1', 'D2', 'score']].tolist()\n",
        "  return tuple(max_value)\n",
        "\n",
        "most_similar_docs()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "8ef9ca58e161295ada11fe2e80a877d0",
          "grade": true,
          "grade_id": "cell-3fedb973b11cff11",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "kFo4ueQpADB9"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "920f6dd60584558a367d6d6cc96b2d40",
          "grade": false,
          "grade_id": "cell-324e2e400c533ee8",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "M1e9j2L2ADB9"
      },
      "source": [
        "### label_accuracy\n",
        "\n",
        "Provide labels for the twenty pairs of documents by computing the similarity for each pair using `document_path_similarity`. Let the classifier rule be that if the score is greater than 0.75, label is paraphrase (1), else label is paraphrase (0). Report accuracy of the classifier using scikit-learn's accuracy_score.\n",
        "\n",
        "*This function should return a float.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "679249da013c7ed60aa06dd8b8630997",
          "grade": false,
          "grade_id": "cell-9e32dbbdb446b28d",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "yH9eJwfOADB9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa2cdd4a-4897-4e66-b168-e1bf67b384f0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "def label_accuracy():\n",
        "  from sklearn.metrics import accuracy_score\n",
        "  data = paraphrases.copy()\n",
        "  data['score'] = data.apply(lambda x: document_path_similarity(x['D1'], x['D2']), axis =1)\n",
        "  data['label'] = np.where(data['score'] > 0.75, 1, 0)\n",
        "  return accuracy_score(data['Quality'], data['label'])\n",
        "\n",
        "label_accuracy()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "f83172ce251c4665a2d20e210f333568",
          "grade": true,
          "grade_id": "cell-38683a0be95791b4",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "JmxkOnH3ADB9"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "b44e3f41d8229552249a94004bd682b4",
          "grade": false,
          "grade_id": "cell-69b6f071d2a015d1",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "w1KwV7qdADB9"
      },
      "source": [
        "## Part 2 - Topic Modelling\n",
        "\n",
        "For the second part of this assignment, you will use Gensim's LDA (Latent Dirichlet Allocation) model to model topics in `newsgroup_data`. You will first need to finish the code in the cell below by using gensim.models.ldamodel.LdaModel constructor to estimate LDA model parameters on the corpus, and save to the variable `ldamodel`. Extract 10 topics using `corpus` and `id_map`, and with `passes=25` and `random_state=34`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "233c1b01c469bc03afff6a117d5859bb",
          "grade": false,
          "grade_id": "cell-badd89f1cf3388f3",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "AFKL40IbADB-"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "import gensim\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "# Load the list of documents\n",
        "with open('assets/newsgroups', 'rb') as f:\n",
        "    newsgroup_data = pickle.load(f)\n",
        "\n",
        "# Use CountVectorizor to find three letter tokens, remove stop_words,\n",
        "# remove tokens that don't appear in at least 20 documents,\n",
        "# remove tokens that appear in more than 20% of the documents\n",
        "vect = CountVectorizer(min_df=20, max_df=0.2, stop_words='english',\n",
        "                       token_pattern='(?u)\\\\b\\\\w\\\\w\\\\w+\\\\b')\n",
        "# Fit and transform\n",
        "X = vect.fit_transform(newsgroup_data)\n",
        "\n",
        "# Convert sparse matrix to gensim corpus.\n",
        "corpus = gensim.matutils.Sparse2Corpus(X, documents_columns=False)\n",
        "\n",
        "# Mapping from word IDs to words (To be used in LdaModel's id2word parameter)\n",
        "id_map = dict((v, k) for k, v in vect.vocabulary_.items())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "9428fc4a1c574c41e90c5f9035b188c9",
          "grade": false,
          "grade_id": "cell-335faff9e0486431",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "bpLCjro5ADB-"
      },
      "outputs": [],
      "source": [
        "# Use the gensim.models.ldamodel.LdaModel constructor to estimate\n",
        "# LDA model parameters on the corpus, and save to the variable `ldamodel`\n",
        "\n",
        "ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=10,\n",
        "                                           id2word=id_map, passes=25, random_state=34)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "08aa0097d5b8ea6e1f242bea0a1c2f1f",
          "grade": false,
          "grade_id": "cell-8761473b2afdb2a0",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "TGShQUiYADB-"
      },
      "source": [
        "### lda_topics\n",
        "\n",
        "Using `ldamodel`, find a list of the 10 topics and the most significant 10 words in each topic. This should be structured as a list of 10 tuples where each tuple takes on the form:\n",
        "\n",
        "`(9, '0.068*\"space\" + 0.036*\"nasa\" + 0.021*\"science\" + 0.020*\"edu\" + 0.019*\"data\" + 0.017*\"shuttle\" + 0.015*\"launch\" + 0.015*\"available\" + 0.014*\"center\" + 0.013*\"information\"')`\n",
        "\n",
        "for example.\n",
        "\n",
        "*This function should return a list of tuples.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "9635c4874774a2fe8b6ba666622bf618",
          "grade": false,
          "grade_id": "cell-3626921bb392f4cf",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "B5wFbCCcADB-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86a8b6e6-9325-471f-dc4c-1e171617e56d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(0,\n",
              "  '0.056*\"edu\" + 0.043*\"com\" + 0.033*\"thanks\" + 0.022*\"mail\" + 0.021*\"know\" + 0.020*\"does\" + 0.014*\"info\" + 0.012*\"monitor\" + 0.010*\"looking\" + 0.010*\"don\"'),\n",
              " (1,\n",
              "  '0.024*\"ground\" + 0.018*\"current\" + 0.018*\"just\" + 0.013*\"want\" + 0.013*\"use\" + 0.011*\"using\" + 0.011*\"used\" + 0.010*\"power\" + 0.010*\"speed\" + 0.010*\"output\"'),\n",
              " (2,\n",
              "  '0.061*\"drive\" + 0.042*\"disk\" + 0.033*\"scsi\" + 0.030*\"drives\" + 0.028*\"hard\" + 0.028*\"controller\" + 0.027*\"card\" + 0.020*\"rom\" + 0.018*\"floppy\" + 0.017*\"bus\"'),\n",
              " (3,\n",
              "  '0.023*\"time\" + 0.015*\"atheism\" + 0.014*\"list\" + 0.013*\"left\" + 0.012*\"alt\" + 0.012*\"faq\" + 0.012*\"probably\" + 0.011*\"know\" + 0.011*\"send\" + 0.010*\"months\"'),\n",
              " (4,\n",
              "  '0.025*\"car\" + 0.016*\"just\" + 0.014*\"don\" + 0.014*\"bike\" + 0.012*\"good\" + 0.011*\"new\" + 0.011*\"think\" + 0.010*\"year\" + 0.010*\"cars\" + 0.010*\"time\"'),\n",
              " (5,\n",
              "  '0.030*\"game\" + 0.027*\"team\" + 0.023*\"year\" + 0.017*\"games\" + 0.016*\"play\" + 0.012*\"season\" + 0.012*\"players\" + 0.012*\"win\" + 0.011*\"hockey\" + 0.011*\"good\"'),\n",
              " (6,\n",
              "  '0.017*\"information\" + 0.014*\"help\" + 0.014*\"medical\" + 0.012*\"new\" + 0.012*\"use\" + 0.012*\"000\" + 0.012*\"research\" + 0.011*\"university\" + 0.010*\"number\" + 0.010*\"program\"'),\n",
              " (7,\n",
              "  '0.022*\"don\" + 0.021*\"people\" + 0.018*\"think\" + 0.017*\"just\" + 0.012*\"say\" + 0.011*\"know\" + 0.011*\"does\" + 0.011*\"good\" + 0.010*\"god\" + 0.009*\"way\"'),\n",
              " (8,\n",
              "  '0.034*\"use\" + 0.023*\"apple\" + 0.020*\"power\" + 0.016*\"time\" + 0.015*\"data\" + 0.015*\"software\" + 0.012*\"pin\" + 0.012*\"memory\" + 0.012*\"simms\" + 0.011*\"port\"'),\n",
              " (9,\n",
              "  '0.068*\"space\" + 0.036*\"nasa\" + 0.021*\"science\" + 0.020*\"edu\" + 0.019*\"data\" + 0.017*\"shuttle\" + 0.015*\"launch\" + 0.015*\"available\" + 0.014*\"center\" + 0.014*\"sci\"')]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "def lda_topics():\n",
        "  return ldamodel.print_topics(num_topics = 10, num_words=10)\n",
        "\n",
        "lda_topics()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "5b00589df439c4ea31976a1c50a4abce",
          "grade": true,
          "grade_id": "cell-be3408e13d501ecb",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "oNFesXqgADB-"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "955ee10e095bd549182a9d18177d84b8",
          "grade": false,
          "grade_id": "cell-5cb2ba78a730107f",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "WYHnHGYbADB-"
      },
      "source": [
        "### topic_distribution\n",
        "\n",
        "For the new document `new_doc`, find the topic distribution. Remember to use vect.transform on the the new doc, and Sparse2Corpus to convert the sparse matrix to gensim corpus.\n",
        "\n",
        "*This function should return a list of tuples, where each tuple is `(#topic, probability)`*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "178e83cfb938733a4ce0dfc9d397a15b",
          "grade": false,
          "grade_id": "cell-57807620ac95b62a",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "rvdhVuLWADB-"
      },
      "outputs": [],
      "source": [
        "new_doc = [\"\\n\\nIt's my understanding that the freezing will start to occur because \\\n",
        "of the\\ngrowing distance of Pluto and Charon from the Sun, due to it's\\nelliptical orbit. \\\n",
        "It is not due to shadowing effects. \\n\\n\\nPluto can shadow Charon, and vice-versa.\\n\\nGeorge \\\n",
        "Krumins\\n-- \"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "f407a0d5d3d8ffe64a10489b1a2f71c3",
          "grade": false,
          "grade_id": "cell-2efe69c14930ff13",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "CzbmESINADB_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7a2ed095-0c70-45ea-eb39-0db3b2389537"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(0, 0.020003108),\n",
              " (1, 0.020003324),\n",
              " (2, 0.020001281),\n",
              " (3, 0.49674848),\n",
              " (4, 0.020004038),\n",
              " (5, 0.020004129),\n",
              " (6, 0.020002972),\n",
              " (7, 0.020002645),\n",
              " (8, 0.020003129),\n",
              " (9, 0.34322688)]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "def topic_distribution():\n",
        "  new_X = vect.transform(new_doc)\n",
        "  doc_corpus = gensim.matutils.Sparse2Corpus(new_X, documents_columns=False)\n",
        "  topics = ldamodel.get_document_topics(doc_corpus)\n",
        "  return list(topics)[0]\n",
        "\n",
        "topic_distribution()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "525d35a1a77b065ac55b4800e53f1efa",
          "grade": true,
          "grade_id": "cell-9a7e35944a145c49",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "1Yk8bOlCADB_"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "markdown",
          "checksum": "b4d7fdb7d4d08f321b4b0199a59e867d",
          "grade": false,
          "grade_id": "cell-edd0c66c43ce9b3c",
          "locked": true,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "Eu6Ha0u_ADCA"
      },
      "source": [
        "### topic_names\n",
        "\n",
        "From the list of the following given topics, assign topic names to the topics you found. If none of these names best matches the topics you found, create a new 1-3 word \"title\" for the topic.\n",
        "\n",
        "Topics: Health, Science, Automobiles, Politics, Government, Travel, Computers & IT, Sports, Business, Society & Lifestyle, Religion, Education.\n",
        "\n",
        "*This function should return a list of 10 strings.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "dfdbbdac5cb54200701c3e5a11c179d7",
          "grade": false,
          "grade_id": "cell-06fc668c4104c2d2",
          "locked": false,
          "schema_version": 3,
          "solution": true,
          "task": false
        },
        "id": "5_5QT0IKADCB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c273002b-0d57-4c4c-e152-b016fd106dba"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Health',\n",
              " 'Automobiles',\n",
              " 'Politics',\n",
              " 'Computers & IT',\n",
              " 'Sports',\n",
              " 'Travel',\n",
              " 'Business',\n",
              " 'Religion',\n",
              " 'Education',\n",
              " 'Society & Lifestyle']"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "def topic_names():\n",
        "  return [\"Health\", \"Automobiles\", \"Politics\", \"Computers & IT\", \"Sports\",\n",
        "            \"Travel\", \"Business\", \"Religion\", \"Education\", \"Society & Lifestyle\"]\n",
        "\n",
        "topic_names()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "cell_type": "code",
          "checksum": "c7d78951c158c0911f860154707c68e5",
          "grade": true,
          "grade_id": "cell-78ea383cc25da8da",
          "locked": true,
          "points": 1,
          "schema_version": 3,
          "solution": false,
          "task": false
        },
        "id": "L7KY7gcrADCB"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "coursera": {
      "course_slug": "python-text-mining",
      "graded_item_id": "2qbcK",
      "launcher_item_id": "pi9Sh",
      "part_id": "kQiwX"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "vscode": {
      "interpreter": {
        "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
      }
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}