movie_data_graph.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This demo contains the following:\n",
    "\n",
    "* Setting up Python environment - importing libraries and first look at the raw dataset \n",
    "\n",
    "* Import dataset to ArangoDB\n",
    "\n",
    "* Preprocessing raw data\n",
    "\n",
    "    * Using ArangoQL\n",
    "    \n",
    "    * Connecting with Python using PyArango\n",
    "\n",
    "* Data exploration with the features of ArangoDB.\n",
    "    \n",
    "    * Graph visualization\n",
    "    \n",
    "    * ArangoSearch example\n",
    "    \n",
    "    * K-shortest path example\n",
    "    \n",
    "    * Pruned search\n",
    "\n",
    "* Machine Learning tasks\n",
    "    \n",
    "    * Movie similarity based on plots using Tensorflow. \n",
    "    \n",
    "    * Genre classification based on plots using -\n",
    "    \n",
    "        * scikit-learn\n",
    "        \n",
    "        * Tensorflow\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are working with Movie data scraped from Wikipedia: [link](https://www.kaggle.com/jrobischon/wikipedia-movie-plots)\n",
    "\n",
    "The dataset contains descriptions of 34,886 movies from around the world. Column descriptions are listed below:\n",
    "\n",
    "1. Release Year - Year in which the movie was released\n",
    "2. Title - Movie title\n",
    "3. Origin/Ethnicity - Origin of movie (i.e. American, Bollywood, Tamil, etc.)\n",
    "4. Director - Director(s) (comma separated, null values)\n",
    "5. Cast - Main actor and actresses (comma separated, null values)\n",
    "6. Genre - Movie Genre(s) (unknown values)\n",
    "7. Wiki Page - URL of the Wikipedia page from which the plot description was scraped\n",
    "8. Plot - Long form description of movie plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"wiki_movie_plots_deduped.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Release Year</th>\n",
       "      <th>Title</th>\n",
       "      <th>Origin/Ethnicity</th>\n",
       "      <th>Director</th>\n",
       "      <th>Cast</th>\n",
       "      <th>Genre</th>\n",
       "      <th>Wiki Page</th>\n",
       "      <th>Plot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>18279</td>\n",
       "      <td>1934</td>\n",
       "      <td>Get Your Man</td>\n",
       "      <td>British</td>\n",
       "      <td>George King</td>\n",
       "      <td>Dorothy Boyd, Sebastian Shaw</td>\n",
       "      <td>comedy</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Get_Your_Man_(19...</td>\n",
       "      <td>A determined young woman sets up an elaborate ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2131</td>\n",
       "      <td>1936</td>\n",
       "      <td>Satan Met a Lady</td>\n",
       "      <td>American</td>\n",
       "      <td>William Dieterle</td>\n",
       "      <td>Bette Davis, Warren William, Alison Skipworth</td>\n",
       "      <td>comedy, drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Satan_Met_a_Lady</td>\n",
       "      <td>Private detective Ted Shane returns to work wi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21818</td>\n",
       "      <td>1984</td>\n",
       "      <td>Next of Kin</td>\n",
       "      <td>Canadian</td>\n",
       "      <td>Atom Egoyan</td>\n",
       "      <td>Patrick Tierney, Arsinée Khanjian</td>\n",
       "      <td>drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Next_of_Kin_(198...</td>\n",
       "      <td>Twenty-three-year-old Peter Foster is an only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26942</td>\n",
       "      <td>2013</td>\n",
       "      <td>Goliyon Ki Rasleela Ram-Leela</td>\n",
       "      <td>Bollywood</td>\n",
       "      <td>Sanjay Leela Bhansali</td>\n",
       "      <td>Ranveer Singh, Deepika Padukone, Richa Chadda,...</td>\n",
       "      <td>romance/drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Goliyon_Ki_Rasle...</td>\n",
       "      <td>In the fictional Gujarati village Ranjaar, inf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29264</td>\n",
       "      <td>1962</td>\n",
       "      <td>Bale Pandiya</td>\n",
       "      <td>Tamil</td>\n",
       "      <td>B. R. Panthulu</td>\n",
       "      <td>Sivaji Ganesan, M. R. Radha, Devika</td>\n",
       "      <td>unknown</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bale_Pandiya_(19...</td>\n",
       "      <td>Pandiya is a young man who leads a troubled li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31056</td>\n",
       "      <td>2011</td>\n",
       "      <td>Seedan</td>\n",
       "      <td>Tamil</td>\n",
       "      <td>Subramaniya Siva</td>\n",
       "      <td>NaN</td>\n",
       "      <td>drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Seedan</td>\n",
       "      <td>Mahalakshmi (Ananya) is a servant at the resid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32925</td>\n",
       "      <td>1965</td>\n",
       "      <td>Sword of the Beast</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Gosha, HideoHideo Gosha</td>\n",
       "      <td>Mikijiro Hira</td>\n",
       "      <td>action, drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Sword_of_the_Beast</td>\n",
       "      <td>Gennosuke is a rebel samurai on the run, havin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>417</td>\n",
       "      <td>1921</td>\n",
       "      <td>Tol'able David</td>\n",
       "      <td>American</td>\n",
       "      <td>Henry King</td>\n",
       "      <td>Richard Barthelmess</td>\n",
       "      <td>drama</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Tol%27able_David</td>\n",
       "      <td>David Kinemon, youngest son of West Virginia t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33425</td>\n",
       "      <td>2004</td>\n",
       "      <td>Ultraman: The Next</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Kazuya Konaka</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Ultraman_(2004_f...</td>\n",
       "      <td>First Lieutenant Shunichi Maki of the Japan Ai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2668</td>\n",
       "      <td>1939</td>\n",
       "      <td>She Married a Cop</td>\n",
       "      <td>American</td>\n",
       "      <td>Sidney Salkow</td>\n",
       "      <td>Jean Parker, Phil Regan</td>\n",
       "      <td>comedy</td>\n",
       "      <td>https://en.wikipedia.org/wiki/She_Married_a_Cop</td>\n",
       "      <td>A couple of cops, Jimmy Duffy and partner Joe,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Release Year                          Title Origin/Ethnicity  \\\n",
       "18279          1934                   Get Your Man          British   \n",
       "2131           1936               Satan Met a Lady         American   \n",
       "21818          1984                    Next of Kin         Canadian   \n",
       "26942          2013  Goliyon Ki Rasleela Ram-Leela        Bollywood   \n",
       "29264          1962                   Bale Pandiya            Tamil   \n",
       "31056          2011                         Seedan            Tamil   \n",
       "32925          1965             Sword of the Beast         Japanese   \n",
       "417            1921                 Tol'able David         American   \n",
       "33425          2004             Ultraman: The Next         Japanese   \n",
       "2668           1939              She Married a Cop         American   \n",
       "\n",
       "                      Director  \\\n",
       "18279              George King   \n",
       "2131          William Dieterle   \n",
       "21818              Atom Egoyan   \n",
       "26942    Sanjay Leela Bhansali   \n",
       "29264           B. R. Panthulu   \n",
       "31056         Subramaniya Siva   \n",
       "32925  Gosha, HideoHideo Gosha   \n",
       "417                 Henry King   \n",
       "33425            Kazuya Konaka   \n",
       "2668             Sidney Salkow   \n",
       "\n",
       "                                                    Cast          Genre  \\\n",
       "18279                       Dorothy Boyd, Sebastian Shaw         comedy   \n",
       "2131       Bette Davis, Warren William, Alison Skipworth  comedy, drama   \n",
       "21818                  Patrick Tierney, Arsinée Khanjian          drama   \n",
       "26942  Ranveer Singh, Deepika Padukone, Richa Chadda,...  romance/drama   \n",
       "29264                Sivaji Ganesan, M. R. Radha, Devika        unknown   \n",
       "31056                                                NaN          drama   \n",
       "32925                                      Mikijiro Hira  action, drama   \n",
       "417                                  Richard Barthelmess          drama   \n",
       "33425                                                NaN        unknown   \n",
       "2668                             Jean Parker, Phil Regan         comedy   \n",
       "\n",
       "                                               Wiki Page  \\\n",
       "18279  https://en.wikipedia.org/wiki/Get_Your_Man_(19...   \n",
       "2131      https://en.wikipedia.org/wiki/Satan_Met_a_Lady   \n",
       "21818  https://en.wikipedia.org/wiki/Next_of_Kin_(198...   \n",
       "26942  https://en.wikipedia.org/wiki/Goliyon_Ki_Rasle...   \n",
       "29264  https://en.wikipedia.org/wiki/Bale_Pandiya_(19...   \n",
       "31056               https://en.wikipedia.org/wiki/Seedan   \n",
       "32925   https://en.wikipedia.org/wiki/Sword_of_the_Beast   \n",
       "417       https://en.wikipedia.org/wiki/Tol%27able_David   \n",
       "33425  https://en.wikipedia.org/wiki/Ultraman_(2004_f...   \n",
       "2668     https://en.wikipedia.org/wiki/She_Married_a_Cop   \n",
       "\n",
       "                                                    Plot  \n",
       "18279  A determined young woman sets up an elaborate ...  \n",
       "2131   Private detective Ted Shane returns to work wi...  \n",
       "21818  Twenty-three-year-old Peter Foster is an only ...  \n",
       "26942  In the fictional Gujarati village Ranjaar, inf...  \n",
       "29264  Pandiya is a young man who leads a troubled li...  \n",
       "31056  Mahalakshmi (Ananya) is a servant at the resid...  \n",
       "32925  Gennosuke is a rebel samurai on the run, havin...  \n",
       "417    David Kinemon, youngest son of West Virginia t...  \n",
       "33425  First Lieutenant Shunichi Maki of the Japan Ai...  \n",
       "2668   A couple of cops, Jimmy Duffy and partner Joe,...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that columns like `Cast` (also `Director` and `Genre`) contain multiple values that might be separated by a comma, space or slash etc. It will require some preprocessing. \n",
    "\n",
    "First we will learn how to import the data into ArangoDB, preprocess it and build a knowledge graph from it for better interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import data to ArangoDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create new databse:\n",
    "\n",
    "    db._createDatabase(\"arangoml\", {}, [{ username: \"root\", passwd: \"\", active: true}])\n",
    "\n",
    "ArangoDB Import data:\n",
    "1. Go to the directory that contains the dataset.\n",
    "2. Open terminal and write the following command:\n",
    "\n",
    "        arangoimport --file \"wiki_movie_plots_deduped.csv\" --type csv --server.database arangoml --create-collection --collection \"movies\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocess dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using ArangoQL\n",
    "\n",
    "1. We want store different columns like cast, director etc. as documents in collections as raw data is highly unstructured. But it requires some processing first. For example, if we want to store all Casts in a 'cast' collection, we first need to process the original data (which ideally should contain comma separated cast members) as it contains unwanted characters and stopwords. We handle them and extract unique Actors/Actresses from the raw dataset in following way:\n",
    "\n",
    "        let casts_data = (\n",
    "        for i in movies\n",
    "            filter i['Cast'] != null\n",
    "            let casts = substitute(\n",
    "                i['Cast'], \n",
    "                [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','Cast: ','.'],\n",
    "                ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', '','']\n",
    "            )\n",
    "            for j in split(casts, \",\")\n",
    "                let nj = substitute(trim(j),[' '],['_'])\n",
    "                filter trim(j)!=''\n",
    "                return distinct nj)\n",
    "\n",
    "        for i in casts_data\n",
    "            insert {'_key':i} in cast options {ignoreErrors: true}\n",
    "        \n",
    "    We can execute same query for Director, Origin, Genre columns.\n",
    "    Director:\n",
    "    \n",
    "         let directors_data = (\n",
    "         for i in movies\n",
    "             filter i['Director'] != null\n",
    "             let director = substitute(\n",
    "                 i['Director'], \n",
    "                 [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','Director: ','Directors: ','.'],\n",
    "                 ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', '', '','']\n",
    "             )\n",
    "             for j in split(director, \",\")\n",
    "                 let nj = substitute(trim(j),[' '],['_'])\n",
    "                 filter trim(j)!=''\n",
    "                 return distinct nj)\n",
    "\n",
    "         for i in directors_data\n",
    "             insert {'_key':i} in director options {ignoreErrors: true}\n",
    "             \n",
    "     Origin:\n",
    "            \n",
    "            let origin_data = (\n",
    "             for i in movies\n",
    "                 filter i['Origin'] != null\n",
    "                 let origin = substitute(\n",
    "                     i['Origin'], \n",
    "                        [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','-','_',' ','.'],\n",
    "                        ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', ',', ',', ',','']\n",
    "                        )\n",
    "                 )\n",
    "                 for j in split(origin, \",\")\n",
    "                     let nj = substitute(trim(j),[' '],['_'])\n",
    "                     filter trim(j)!=''\n",
    "                     return distinct nj)\n",
    "\n",
    "             for i in origin_data\n",
    "                 insert {'_key':i} in origin options {ignoreErrors: true}\n",
    "             \n",
    "     Genre:\n",
    "              \n",
    "             let genre_data = (\n",
    "             for i in movies\n",
    "                 filter i['Genre'] != null\n",
    "                 let genre = substitute(\n",
    "                     i['Genre'], \n",
    "                    [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','-','_',' ','.'],\n",
    "                    ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', ',', ',', ',','']\n",
    "                    )\n",
    "                 for j in split(genre, \",\")\n",
    "                     let nj = substitute(trim(j),[' '],['_'])\n",
    "                     filter trim(j)!=''\n",
    "                     return distinct nj)\n",
    "\n",
    "             for i in genre_data\n",
    "                 insert {'_key':i} in genre options {ignoreErrors: true}\n",
    "         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. We create a 'movie' collection that will store specific info about movies like Release data, Title, Plot. Along with this, we also add an edge between the moveis and its corresponding cast members, director(s), origin and genre (that we created from previous queries). To insert data into 'movie' collection, execute the following query:\n",
    "\n",
    "            for i in movies\n",
    "                let id_split = split(i['Wiki Page'],\"/\")\n",
    "                let id = substitute(id_split[length(id_split)-1],[\"#\"],[\"_\"])\n",
    "                insert {_key:id, year:i['Release Year'], title:i['Title'], plot:i['Plot']} \n",
    "                    into movie \n",
    "                    options { overwrite: true, ignoreErrors: true }\n",
    "\n",
    "\n",
    "    \n",
    "    For adding edge with Casts/Director append the following query to the above query:\n",
    "    \n",
    "            let casts = substitute(\n",
    "                i['Cast'], \n",
    "                [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','Director: ','Directors: ','Cast: ','.'],\n",
    "                ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', '', '', '','']\n",
    "                )\n",
    "            for j in split(casts, \",\")\n",
    "                let nj = substitute(trim(j),[' '],['_'])\n",
    "                filter trim(j)!=''\n",
    "                insert {_from: concat(\"movie/\",id), _to:concat(\"cast/\",nj), label:'had as a cast'} \n",
    "                    into conn \n",
    "                    options {ignoreErrors: true}\n",
    "\n",
    "    Similarly for adding edge with Genre/Origin:\n",
    "    \n",
    "        for i in movies\n",
    "            let id_split = split(i['Wiki Page'],\"/\")\n",
    "            let id = substitute(id_split[length(id_split)-1],[\"#\"],[\"_\"])\n",
    "            let genre = substitute(\n",
    "                i['Genre'], \n",
    "                [\"'\",']','[','\"','\\r\\n',')','(','; ',' and ',' & ','/','-','_',' ','.'],\n",
    "                ['', '', '',', ', ', ', '', '', ', ', ', ', ', ', ', ', ',', ',', ',','']\n",
    "                )\n",
    "            for j in split(genre, \",\")\n",
    "                let nj = substitute(trim(j),[' '],['_'])\n",
    "                filter trim(j)!=''\n",
    "                insert {_from: concat(\"movie/\",id), _to:concat(\"genre/\",nj), label:'genre'} \n",
    "                    into conn \n",
    "                    options {ignoreErrors: true}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Python "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to do insert node and edges is by using Python. For this, we connect with ArangoDB using PyArango. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyArango.connection import Connection\n",
    "conn = Connection(username=\"root\", password=\"\")\n",
    "db = conn[\"arangoml\"]\n",
    "def exec(db, aql):\n",
    "\toutput = db.AQLQuery(aql, rawResults=True, batchSize=1000)\n",
    "\treturn np.array(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we just need to use `exec()` and provide database variable `db` with corresponding `aql` query for execution. It’s that easy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create graph named `movies` in ArangoDB with the all the node collections and the edge collection created in the previous section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ex2](screenshots/pic3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ex2](screenshots/pic2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we can clearly see the connections with the descriptions, we perform graph exploration techniques that are available in ArangoDB for answering different types of research questions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search movies containing given phrase in its plot\n",
    "We do this by using new feature in ArangoDB 3.5 called ArangoSearch. We use `PHRASE` function to search for documents containing an exact sentence or phrase. To know how it works, refer to [this](https://www.arangodb.com/arangodb-training-center/search/arangosearch/) blog.\n",
    "\n",
    "We link the view named `search_` with the `movie` collection to index `Plot` column and execute the following query. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    for i in search_\n",
    "        SEARCH PHRASE(i.Plot,'batman and robin', 'text_en')\n",
    "        SORT TFIDF(i) desc\n",
    "        limit 5\n",
    "        return [i.Title, i['Release Year']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search with specific Genre combinations\n",
    "We use another new feature `K_SHORTEST_PATHS` ([details](https://www.arangodb.com/docs/stable/aql/graphs-kshortest-paths.html)) for this. It allows you to query not just a shortest path between two documents in an ArangoDB database, but the list of paths between them, sorted by the length/weights. Here we try to find shortest paths between two different genres `comedy` and `horror`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    FOR p IN ANY K_SHORTEST_PATHS 'genre/comedy' TO 'genre/horror'\n",
    "      GRAPH 'movies'\n",
    "          LIMIT 3\n",
    "          RETURN [p.vertices[*]._key]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s do a similar search with `war` and `horror`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    FOR p IN ANY K_SHORTEST_PATHS 'genre/war' TO 'genre/horror'\n",
    "      GRAPH 'movies'\n",
    "          LIMIT 3\n",
    "          RETURN [p.vertices[*]._key]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found that there is just one movie titled `Below` in the database (as the shortest path is 3) which is about `war` + `horror`. But the other two outputs just connects movies through their origin. Other outputs are simply connected through their `American` origin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using pruned traversal on graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we look for `American action` movies using pruning ([detail](https://www.arangodb.com/docs/stable/aql/graphs-traversals.html#pruning)) on `Genre` edges during graph traversal. It improves query performance and reduces the amount of overhead generated by the query.\n",
    "\n",
    "    FOR v, e, p IN 1..3 ANY 'origin/American' GRAPH 'movies'\n",
    "          PRUNE e.label == 'genre'\n",
    "          FILTER v._key=='action'\n",
    "          LIMIT 5\n",
    "          RETURN p.vertices[1]._key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without the PRUNE command, if we execute the above query, we get the same results in ~5 minutes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to perform mainly two ML tasks:\n",
    "\n",
    "1. Movie similarity based on plots - using Tensorflow. \n",
    "    - Content-based recommendation of movies.\n",
    "2. Genre classification based on plots - using scikit-learn and Tensorflow. \n",
    "    - Predicting appropriate genres for data with null/unknown values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Movie recommendation based on plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm_notebook\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "from nltk import sent_tokenize\n",
    "from scipy import spatial\n",
    "from operator import itemgetter\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply basic regex tools to clean movie plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_plot(text_list):\n",
    "    clean_list = []\n",
    "    for sent in text_list:\n",
    "        sent = re.sub('[%s]' % re.escape(\"\"\"!\"#$%&'()*+,-.:;<=>?@[\\]^`{|}~\"\"\"), '',sent)\n",
    "        sent = sent.replace('[]','')\n",
    "        sent = re.sub('\\d+',' ',sent)\n",
    "        sent = sent.lower()\n",
    "        clean_list.append(sent)\n",
    "    return clean_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find plot embeddings: (takes some time ~ 5 minutes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c21c6295d924f13a35f0c7d14342449",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=34886), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "plot_emb_list = []\n",
    "with tf.Graph().as_default():\n",
    "    embed = hub.Module(\"https://tfhub.dev/google/universal-sentence-encoder/2\")\n",
    "    messages = tf.placeholder(dtype=tf.string, shape=[None])\n",
    "    output = embed(messages)\n",
    "    with tf.Session() as session:\n",
    "        session.run([tf.global_variables_initializer(), tf.tables_initializer()])\n",
    "        for plot in tqdm_notebook(df['Plot']):\n",
    "            sent_list = sent_tokenize(plot)\n",
    "            clean_sent_list = clean_plot(sent_list)\n",
    "            sent_embed = session.run(output, feed_dict={messages: clean_sent_list})\n",
    "            plot_emb_list.append(sent_embed.mean(axis=0).reshape(1,512))            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['embeddings'] = plot_emb_list\n",
    "df.to_pickle('./df_embed.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def similar_movie(movie_name,topn=5):\n",
    "    plot = df[df['Title']==movie_name]['Plot'].values[0]\n",
    "    with tf.Graph().as_default():\n",
    "        embed = hub.Module(\"https://tfhub.dev/google/universal-sentence-encoder/2\")\n",
    "        messages = tf.placeholder(dtype=tf.string, shape=[None])\n",
    "        output = embed(messages)\n",
    "        with tf.Session() as session:\n",
    "            session.run([tf.global_variables_initializer(), tf.tables_initializer()])\n",
    "            sent_list = sent_tokenize(plot)\n",
    "            clean_sent_list = clean_plot(sent_list)\n",
    "            sent_embed2 = (session.run(output, feed_dict={messages: clean_sent_list})).mean(axis=0).reshape(1,512)\n",
    "            similarities, titles = [],[movie_name]\n",
    "            for tensor,title in zip(df['embeddings'],df['Title']):\n",
    "                if title not in titles:\n",
    "                    cos_sim = 1 - spatial.distance.cosine(sent_embed2,tensor)\n",
    "                    similarities.append((title,cos_sim))\n",
    "                    titles.append(title)\n",
    "            return sorted(similarities,key=itemgetter(1),reverse=True)[1:topn+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('Batman v Superman: Dawn of Justice', 0.933032214641571),\n",
       " ('Superman Returns', 0.9193736910820007),\n",
       " ('Superman: Unbound', 0.9133195281028748),\n",
       " ('Superman IV: The Quest for Peace', 0.9110839366912842),\n",
       " ('Justice League: The Flashpoint Paradox', 0.9093491435050964),\n",
       " ('Man of Steel', 0.9017841219902039),\n",
       " ('Justice League', 0.8930277824401855),\n",
       " ('Megamind', 0.8920153379440308),\n",
       " ('Superman III', 0.8913161754608154),\n",
       " ('Hulk', 0.8826043605804443)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similar_movie('Superman',10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that based on the plots, these are the top movies recommended by the model that are similar to “batman” movie. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Genre Prediction based on plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Using simpler tools (scikit-learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.metrics import f1_score, accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use Tokenizer and remove unneccessary symbols/expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9bdc6fa7b8474efd804b6b24136b562d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=34886), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "new_plots = []\n",
    "for plot in tqdm_notebook(df['Plot']):\n",
    "    sent_list = sent_tokenize(plot)\n",
    "    clean_sent_list = clean_plot(sent_list)\n",
    "    new_plots.append(clean_sent_list[0])\n",
    "df_new = df.copy()\n",
    "df_new['clean plot'] = new_plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split data into train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = df_new[df_new['Genre']!='unknown'][['Title','clean plot','Genre']]\n",
    "test_df = df_new[df_new['Genre']=='unknown'][['Title','clean plot','Genre']]\n",
    "train_df['genre_new'] = [x.replace(' ',',').replace('_',',').replace('-',',').split(',') for x in train_df['Genre'].values]\n",
    "test_df['genre_new'] = [x.replace(' ',',').replace('_',',').replace('-',',').split(',') for x in test_df['Genre'].values]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove stopwords from plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# function to remove stopwords\n",
    "def remove_stopwords(text):\n",
    "    no_stopword_text = [w for w in text.split() if not w in stop_words]\n",
    "    return ' '.join(no_stopword_text)\n",
    "train_df['clean_plot_new'] = train_df['clean plot'].apply(lambda x: remove_stopwords(x))\n",
    "test_df['clean_plot_new'] = test_df['clean plot'].apply(lambda x: remove_stopwords(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply binarizer for multi-label classification for Genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "multilabel_binarizer = MultiLabelBinarizer()\n",
    "multilabel_binarizer.fit(train_df['genre_new'])\n",
    "\n",
    "# transform target variable\n",
    "y = multilabel_binarizer.transform(train_df['genre_new'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find embeddings of plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=10000)\n",
    "xtrain, xval, ytrain, yval = train_test_split(train_df['clean_plot_new'], y, test_size=0.2, random_state=9)\n",
    "xtrain_tfidf = tfidf_vectorizer.fit_transform(xtrain)\n",
    "xval_tfidf = tfidf_vectorizer.transform(xval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define Logistic Regression model and train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneVsRestClassifier(estimator=LogisticRegression(C=1.0, class_weight=None,\n",
       "                                                 dual=False, fit_intercept=True,\n",
       "                                                 intercept_scaling=1,\n",
       "                                                 l1_ratio=None, max_iter=100,\n",
       "                                                 multi_class='warn',\n",
       "                                                 n_jobs=None, penalty='l2',\n",
       "                                                 random_state=None,\n",
       "                                                 solver='warn', tol=0.0001,\n",
       "                                                 verbose=0, warm_start=False),\n",
       "                    n_jobs=None)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "clf = OneVsRestClassifier(lr)\n",
    "clf.fit(xtrain_tfidf, ytrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As Logistic regression is rather simpler model and data is complicated, we modify threshold for predition probabilities from 0.5 to 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.12619336920673493\n",
      "F1-score: 0.3614946739559263\n"
     ]
    }
   ],
   "source": [
    "y_pred_prob = clf.predict_proba(xval_tfidf)\n",
    "y_pred_new = (y_pred_prob >= 0.2).astype(int)\n",
    "print(\"Accuracy:\" ,accuracy_score(yval, y_pred_new))\n",
    "print(\"F1-score:\" ,f1_score(yval, y_pred_new, average=\"micro\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make predictions ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def infer_tags(q):\n",
    "    q_vec = tfidf_vectorizer.transform([q])\n",
    "    q_pred = clf.predict(q_vec)\n",
    "    return multilabel_binarizer.inverse_transform(q_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie:\t\t The Nanny Diaries\n",
      "Predicted genre:  [('drama',)]\n",
      "Actual genre:  ['comedy', 'drama']\n",
      "Movie:\t\t Sasural\n",
      "Predicted genre:  [('drama',)]\n",
      "Actual genre:  ['family', 'drama']\n",
      "Movie:\t\t You Can't Cheat an Honest Man\n",
      "Predicted genre:  [('comedy',)]\n",
      "Actual genre:  ['comedy']\n",
      "Movie:\t\t A Mother's Story\n",
      "Predicted genre:  [('drama',)]\n",
      "Actual genre:  ['drama']\n",
      "Movie:\t\t Bad Company\n",
      "Predicted genre:  [('drama',)]\n",
      "Actual genre:  ['drama']\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "while i<5: \n",
    "    k = xval.sample(1).index[0]\n",
    "    if infer_tags(xval[k])!=[()]:\n",
    "        print(\"Movie:\\t\\t\",train_df['Title'].ix[k])\n",
    "        print(\"Predicted genre: \", infer_tags(xval[k]))\n",
    "        print(\"Actual genre: \",train_df['genre_new'].ix[k])\n",
    "        i+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "xtest_tfidf = tfidf_vectorizer.transform(test_df['clean_plot_new'])\n",
    "y_test_pred_prob = clf.predict_proba(xtest_tfidf)\n",
    "y_test_pred_new = (y_test_pred_prob >= 0.2).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie:\t\t\t The Bellboy\n",
      "Predicted genre:\t [('comedy',)]\n",
      "Movie:\t\t\t Vaigasi Poranthachu\n",
      "Predicted genre:\t [('drama',)]\n",
      "Movie:\t\t\t Avan (dubbed from Hindi)\n",
      "Predicted genre:\t [('drama',)]\n",
      "Movie:\t\t\t Ek Ruka Hua Faisla\n",
      "Predicted genre:\t [('drama',)]\n",
      "Movie:\t\t\t Anubandham\n",
      "Predicted genre:\t [('drama',)]\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "while i<5: \n",
    "    k = test_df['clean plot'].sample(1).index[0]\n",
    "    pred = infer_tags(test_df['clean plot'].ix[k])\n",
    "    if pred!=[()]:\n",
    "        print(\"Movie:\\t\\t\\t\",test_df['Title'].ix[k])\n",
    "        print(\"Predicted genre:\\t\", pred)\n",
    "        i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Using Deep Learning (Tensorflow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, Embedding, Input\n",
    "from keras.layers import Conv1D, GlobalMaxPool1D, Dropout, concatenate\n",
    "from keras.preprocessing import text, sequence\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MultiLabelBinarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# new_plots = []\n",
    "# for plot in tqdm_notebook(df['Plot']):\n",
    "#     sent_list = sent_tokenize(plot)\n",
    "#     clean_sent_list = clean_plot(sent_list)\n",
    "#     new_plots.append(clean_sent_list[0])\n",
    "# df_new = df.copy()\n",
    "# df_new['clean plot'] = new_plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply similar preprocessing as previous case: Remove unnecessary symbols/expressions and stopwords from `clean plot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_df = df_new[df_new['Genre']!='unknown'][['Title','clean plot','Genre']]\n",
    "# test_df = df_new[df_new['Genre']=='unknown'][['Title','clean plot','Genre']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from nltk.corpus import stopwords\n",
    "# stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# # function to remove stopwords\n",
    "# def remove_stopwords(text):\n",
    "#     no_stopword_text = [w for w in text.split() if not w in stop_words]\n",
    "#     return ' '.join(no_stopword_text)\n",
    "# train_df['clean_plot_new'] = train_df['clean plot'].apply(lambda x: remove_stopwords(x))\n",
    "# test_df['clean_plot_new'] = test_df['clean plot'].apply(lambda x: remove_stopwords(x))\n",
    "# train_df['genre_new'] = [x.replace(' ',',').replace('_',',').replace('-',',').split(',') for x in train_df['Genre'].values]\n",
    "# test_df['genre_new'] = [x.replace(' ',',').replace('_',',').replace('-',',').split(',') for x in test_df['Genre'].values]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxlen = 200\n",
    "max_features = 20000\n",
    "encoder = MultiLabelBinarizer()\n",
    "encoder.fit_transform(train_df['genre_new'])\n",
    "y_train = encoder.transform(train_df['genre_new'])\n",
    "y_test = encoder.transform(test_df['genre_new'])\n",
    "num_classes = len(encoder.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train tokenizer on movie plots of training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = text.Tokenizer(num_words=max_features)\n",
    "tokenizer.fit_on_texts(list(train_df['clean_plot_new']))\n",
    "# train data\n",
    "list_tokenized_train = tokenizer.texts_to_sequences(train_df['clean_plot_new'])\n",
    "X_t = sequence.pad_sequences(list_tokenized_train, maxlen=200)\n",
    "# test data\n",
    "list_tokenized_test = tokenizer.texts_to_sequences(test_df['clean_plot_new'])\n",
    "X_te = sequence.pad_sequences(list_tokenized_test, maxlen=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define 1D-CNN Model here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(conv_layers = 2, max_dilation_rate = 3):\n",
    "    embed_size = 128\n",
    "    inp = Input(shape=(maxlen, ))\n",
    "    x = Embedding(max_features, embed_size)(inp)\n",
    "    x = Dropout(0.25)(x)\n",
    "    x = Conv1D(2*embed_size, \n",
    "                   kernel_size = 3)(x)\n",
    "    prefilt_x = Conv1D(2*embed_size, \n",
    "                   kernel_size = 3)(x)\n",
    "    out_conv = []\n",
    "    for dilation_rate in range(max_dilation_rate):\n",
    "        x = prefilt_x\n",
    "        for i in range(3):\n",
    "            x = Conv1D(32*2**(i), \n",
    "                       kernel_size = 3, \n",
    "                       dilation_rate = dilation_rate+1)(x)    \n",
    "        out_conv += [Dropout(0.5)(GlobalMaxPool1D()(x))]\n",
    "    x = concatenate(out_conv, axis = -1)    \n",
    "    x = Dense(50, activation=\"relu\")(x)\n",
    "    x = Dropout(0.1)(x)\n",
    "    x = Dense(num_classes, activation=\"sigmoid\")(x)\n",
    "    model = Model(inputs=inp, outputs=x)\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 200)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 200, 128)     2560000     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 200, 128)     0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 198, 256)     98560       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, 196, 256)     196864      conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, 194, 32)      24608       conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_6 (Conv1D)               (None, 192, 32)      24608       conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_9 (Conv1D)               (None, 190, 32)      24608       conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_4 (Conv1D)               (None, 192, 64)      6208        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_7 (Conv1D)               (None, 188, 64)      6208        conv1d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_10 (Conv1D)              (None, 184, 64)      6208        conv1d_9[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_5 (Conv1D)               (None, 190, 128)     24704       conv1d_4[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_8 (Conv1D)               (None, 184, 128)     24704       conv1d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_11 (Conv1D)              (None, 178, 128)     24704       conv1d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 128)          0           conv1d_5[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 128)          0           conv1d_8[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 128)          0           conv1d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 128)          0           global_max_pooling1d_1[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 128)          0           global_max_pooling1d_2[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 128)          0           global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 384)          0           dropout_2[0][0]                  \n",
      "                                                                 dropout_3[0][0]                  \n",
      "                                                                 dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 50)           19250       concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 50)           0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 915)          46665       dropout_5[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 3,087,899\n",
      "Trainable params: 3,087,899\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = build_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 23042 samples, validate on 5761 samples\n",
      "Epoch 1/5\n",
      "23042/23042 [==============================] - 163s 7ms/step - loss: 5.4872 - acc: 0.1992 - val_loss: 7.2422 - val_acc: 0.2036\n",
      "Epoch 2/5\n",
      "23042/23042 [==============================] - 111s 5ms/step - loss: 4.6120 - acc: 0.2324 - val_loss: 7.4824 - val_acc: 0.2036\n",
      "Epoch 3/5\n",
      "23042/23042 [==============================] - 109s 5ms/step - loss: 4.3383 - acc: 0.2326 - val_loss: 7.7074 - val_acc: 0.2036\n",
      "Epoch 4/5\n",
      "23042/23042 [==============================] - 107s 5ms/step - loss: 4.0725 - acc: 0.2326 - val_loss: 7.9147 - val_acc: 0.2036\n",
      "Epoch 5/5\n",
      "23042/23042 [==============================] - 118s 5ms/step - loss: 3.8266 - acc: 0.2326 - val_loss: 8.0666 - val_acc: 0.2036\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1723d1dd8>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "epochs = 5\n",
    "\n",
    "file_path=\"weights.hdf5\"\n",
    "\n",
    "checkpoint = ModelCheckpoint(file_path, monitor='val_loss', verbose=1, save_best_only=True, mode='min')\n",
    "early = EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=4)\n",
    "\n",
    "callbacks_list = [checkpoint, early] \n",
    "model.fit(X_t, y_train, \n",
    "          batch_size=batch_size, \n",
    "          epochs=epochs, \n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Movie:\t\t The Great Train Robbery\n",
      "Predicted genre:  ('crime', 'drama', 'western')\n",
      "Actual genre:  ['western']\n",
      "\n",
      "Movie:\t\t The Suburbanite\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['comedy']\n",
      "\n",
      "Movie:\t\t Dream of a Rarebit Fiend\n",
      "Predicted genre:  ('', '/', 'adventure', 'animated', 'animation', 'comedy', 'drama', 'family', 'fantasy', 'horror', 'short')\n",
      "Actual genre:  ['short']\n",
      "\n",
      "Movie:\t\t From Leadville to Aspen: A Hold-Up in the Rockies\n",
      "Predicted genre:  ('comedy', 'drama', 'film')\n",
      "Actual genre:  ['short', 'action/crime', 'western']\n",
      "\n",
      "Movie:\t\t Kathleen Mavourneen\n",
      "Predicted genre:  ('', 'adventure', 'animated', 'animation', 'comedy', 'drama', 'family', 'fantasy', 'horror', 'short')\n",
      "Actual genre:  ['short', 'film']\n",
      "\n",
      "Movie:\t\t Daniel Boone\n",
      "Predicted genre:  ('', 'animated', 'animation', 'comedy', 'drama', 'family', 'fantasy', 'horror', 'short')\n",
      "Actual genre:  ['biographical']\n",
      "\n",
      "Movie:\t\t How Brown Saw the Baseball Game\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['comedy']\n",
      "\n",
      "Movie:\t\t Laughing Gas\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['comedy']\n",
      "\n",
      "Movie:\t\t The Adventures of Dollie\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t The Black Viper\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t A Calamitous Elopement\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['comedy']\n",
      "\n",
      "Movie:\t\t The Call of the Wild\n",
      "Predicted genre:  ('', 'comedy', 'drama', 'family', 'horror')\n",
      "Actual genre:  ['adventure']\n",
      "\n",
      "Movie:\t\t A Christmas Carol\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t The Fight for Freedom\n",
      "Predicted genre:  ('action', 'crime', 'drama', 'western')\n",
      "Actual genre:  ['western']\n",
      "\n",
      "Movie:\t\t At the Altar\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t A Drunkard's Reformation\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t The Golden Louis\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t An Arcadian Maid\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n",
      "\n",
      "Movie:\t\t Hemlock Hoax, the Detective\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['comedy']\n",
      "\n",
      "Movie:\t\t The House with Closed Shutters\n",
      "Predicted genre:  ('comedy', 'drama')\n",
      "Actual genre:  ['drama']\n"
     ]
    }
   ],
   "source": [
    "y_pred_new = (y_pred >= 0.5).astype(int)\n",
    "y_pred_genre = multilabel_binarizer.inverse_transform(y_pred_new)\n",
    "for ind,i in enumerate(train_df.index[:20]):\n",
    "    print(\"\\nMovie:\\t\\t\",train_df['Title'].ix[i])\n",
    "    print(\"Predicted genre: \", y_pred_genre[ind])\n",
    "    print(\"Actual genre: \",train_df['genre_new'].ix[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this model performs much better than the previous one. The `Genre`s of a movie is identifiable using the plot summaries provided in the data. We can use these predictions in place of `unknown` or missing Genre informatiosn of a movie."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}