RNN_Twitter_Sentiment_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter US Airline Sentiment Analysis with an RNN\n",
    "### Analyze how travelers in February 2015 expressed their feelings on Twitter\n",
    "\n",
    "In this notebook, we implement a recurrent neural network to analyze how travelers in February 2015 expressed their feelings on Twitter. Using an RNN rather than a strictly feedforward network is more accurate since we can include information about the *sequence* of words. \n",
    "\n",
    "A sentiment analysis job about the problems of each major U.S. airline. Twitter data was scraped from February of 2015 and contributors were asked to first classify positive, negative, and neutral tweets, followed by categorizing negative reasons (such as \"late flight\" or \"rude service\").\n",
    "\n",
    "In this notebook, our goal is to identify whether a tweet is negative or non-negative (positive or neutral)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Architecture\n",
    "\n",
    "The architecture for this network is shown below.\n",
    "\n",
    "<img src=\"assets/network_diagram.png\" width=40%>\n",
    "\n",
    ">**First, we'll pass in words to an embedding layer.** We need an embedding layer because we have tens of thousands of words, so we'll need a more efficient representation for our input data than one-hot encoded vectors. \n",
    "\n",
    ">**After input words are passed to an embedding layer, the new embeddings will be passed to LSTM cells.** The LSTM cells will add *recurrent* connections to the network and give us the ability to include information about the *sequence* of words in the movie review data. \n",
    "\n",
    ">**Finally, the LSTM outputs will go to a sigmoid output layer.** We're using a sigmoid function because we have two classes (0 is negative and 1 is non-negative) and a sigmoid will output predicted sentiment values between 0-1. \n",
    "\n",
    "We don't care about the sigmoid outputs except for the **very last one**; we can ignore the rest. We'll calculate the loss by comparing the output at the last time step and the training label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Load in and visualize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>570306133677760513</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cairdin</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:35:52 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>570301130888122368</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:59 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>570301083672813571</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>yvonnalynn</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:48 -0800</td>\n",
       "      <td>Lets Play</td>\n",
       "      <td>Central Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>570301031407624196</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.7033</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:36 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>570300817074462722</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica and it's a really big bad thing...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:14:45 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             tweet_id airline_sentiment  airline_sentiment_confidence  \\\n",
       "0  570306133677760513           neutral                        1.0000   \n",
       "1  570301130888122368          positive                        0.3486   \n",
       "2  570301083672813571           neutral                        0.6837   \n",
       "3  570301031407624196          negative                        1.0000   \n",
       "4  570300817074462722          negative                        1.0000   \n",
       "\n",
       "  negativereason  negativereason_confidence         airline  \\\n",
       "0            NaN                        NaN  Virgin America   \n",
       "1            NaN                     0.0000  Virgin America   \n",
       "2            NaN                        NaN  Virgin America   \n",
       "3     Bad Flight                     0.7033  Virgin America   \n",
       "4     Can't Tell                     1.0000  Virgin America   \n",
       "\n",
       "  airline_sentiment_gold        name negativereason_gold  retweet_count  \\\n",
       "0                    NaN     cairdin                 NaN              0   \n",
       "1                    NaN    jnardino                 NaN              0   \n",
       "2                    NaN  yvonnalynn                 NaN              0   \n",
       "3                    NaN    jnardino                 NaN              0   \n",
       "4                    NaN    jnardino                 NaN              0   \n",
       "\n",
       "                                                text tweet_coord  \\\n",
       "0                @VirginAmerica What @dhepburn said.         NaN   \n",
       "1  @VirginAmerica plus you've added commercials t...         NaN   \n",
       "2  @VirginAmerica I didn't today... Must mean I n...         NaN   \n",
       "3  @VirginAmerica it's really aggressive to blast...         NaN   \n",
       "4  @VirginAmerica and it's a really big bad thing...         NaN   \n",
       "\n",
       "               tweet_created tweet_location               user_timezone  \n",
       "0  2015-02-24 11:35:52 -0800            NaN  Eastern Time (US & Canada)  \n",
       "1  2015-02-24 11:15:59 -0800            NaN  Pacific Time (US & Canada)  \n",
       "2  2015-02-24 11:15:48 -0800      Lets Play  Central Time (US & Canada)  \n",
       "3  2015-02-24 11:15:36 -0800            NaN  Pacific Time (US & Canada)  \n",
       "4  2015-02-24 11:14:45 -0800            NaN  Pacific Time (US & Canada)  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/Tweets.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only use the data in the `text` and `airline_sentiment` column. we also only use 14000 data rows of the total 14640 data rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reviews = np.array(data['text'])[:14000]\n",
    "labels = np.array(data['airline_sentiment'])[:14000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'@AmericanAir we have 8 ppl so we need 2 know how many seats are on the next flight. Plz put us on standby for 4 people on the next flight?'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['text'].loc[14639]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'neutral'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['airline_sentiment'].loc[14639]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the distribution of the airline sentiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'negative': 8679, 'neutral': 3017, 'positive': 2304})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "Counter(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Data pre-processing\n",
    "The first step when building a neural network model is getting the data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit.\n",
    "\n",
    "Here are the processing steps, we'll want to take:\n",
    ">* We'll want to get rid of periods and extraneous punctuation.\n",
    "* We'll want to remove web address, twitter id, and digit. \n",
    "\n",
    "First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "punctuation = '!\"#$%&\\'()*+,-./:;<=>?[\\\\]^_`{|}~'\n",
    "\n",
    "# get rid of punctuation\n",
    "all_reviews = 'separator'.join(reviews)\n",
    "all_reviews = all_reviews.lower()\n",
    "all_text = ''.join([c for c in all_reviews if c not in punctuation])\n",
    "\n",
    "# split by new lines and spaces\n",
    "reviews_split = all_text.split('separator')\n",
    "all_text = ' '.join(reviews_split)\n",
    "\n",
    "# create a list of words\n",
    "words = all_text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we remove web address, twitter id, and digit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get rid of web address, twitter id, and digit\n",
    "new_reviews = []\n",
    "for review in reviews_split:\n",
    "    review = review.split()\n",
    "    new_text = []\n",
    "    for word in review:\n",
    "        if (word[0] != '@') & ('http' not in word) & (~word.isdigit()):\n",
    "            new_text.append(word)\n",
    "    new_reviews.append(new_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding the words\n",
    "The embedding lookup requires that we pass in integers to our network. The easiest way to do this is to create dictionaries that map the words in the vocabulary to integers. Then, we can convert each of our reviews into integers so they can be passed into the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Build a dictionary that maps words to integers\n",
    "counts = Counter(words)\n",
    "vocab = sorted(counts, key=counts.get, reverse=True)\n",
    "vocab_to_int = {word: ii for ii, word in enumerate(vocab, 1)}\n",
    "\n",
    "## use the dict to tokenize each review in reviews_split\n",
    "## store the tokenized reviews in reviews_ints\n",
    "reviews_ints = []\n",
    "for review in new_reviews:\n",
    "    reviews_ints.append([vocab_to_int[word] for word in review])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print out the number of unique words in the vocabulary and the contents of the first, tokenized review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words:  16727\n",
      "\n",
      "Tokenized review: \n",
      " [[57, 213]]\n"
     ]
    }
   ],
   "source": [
    "# stats about vocabulary\n",
    "print('Unique words: ', len((vocab_to_int)))  # should ~ 74000+\n",
    "print()\n",
    "\n",
    "# print tokens in first review\n",
    "print('Tokenized review: \\n', reviews_ints[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding the labels\n",
    "As mentioned before, our goal is to identify whether a tweet is negative or non-negative (positive or neutral). Our labels are \"positive\", \"negative\", or \"neutral. To use these labels in our network, we need to convert them to 0 and 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 1=positive, 1=neutral, 0=negative label conversion\n",
    "encoded_labels = []\n",
    "for label in labels:\n",
    "    if label == 'neutral':\n",
    "        encoded_labels.append(1)\n",
    "    elif label == 'negative':\n",
    "        encoded_labels.append(0)\n",
    "    else:\n",
    "        encoded_labels.append(1)\n",
    "\n",
    "encoded_labels = np.asarray(encoded_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding sequences\n",
    "To deal with both short and very long reviews, we'll pad or truncate all our reviews to a specific length. For reviews shorter than some `seq_length`, we'll pad with 0s. For reviews longer than `seq_length`, we can truncate them to the first `seq_length` words. A good `seq_length`, in this case, is 30, because the maximum review length from the data is 32.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_features(reviews_ints, seq_length):\n",
    "    ''' Return features of review_ints, where each review is padded with 0's \n",
    "        or truncated to the input seq_length.\n",
    "    '''\n",
    "    \n",
    "    # getting the correct rows x cols shape\n",
    "    features = np.zeros((len(reviews_ints), seq_length), dtype=int)\n",
    "\n",
    "    # for each review, I grab that review and \n",
    "    for i, row in enumerate(reviews_ints):\n",
    "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0 446]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0]]\n"
     ]
    }
   ],
   "source": [
    "# Test implementation!\n",
    "\n",
    "seq_length = 30\n",
    "\n",
    "features = pad_features(reviews_ints, seq_length=seq_length)\n",
    "\n",
    "## test statements \n",
    "assert len(features)==len(reviews_ints), \"The features should have as many rows as reviews.\"\n",
    "assert len(features[0])==seq_length, \"Each feature row should contain seq_length values.\"\n",
    "\n",
    "# print first 10 values of the first 30 batches \n",
    "print(features[:10,:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Training, validation, and test\n",
    "With our data in nice shape, we'll split it into training, validation, and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\t\tFeature Shapes:\n",
      "Train set: \t\t(11200, 30) \n",
      "Validation set: \t(1400, 30) \n",
      "Test set: \t\t(1400, 30)\n"
     ]
    }
   ],
   "source": [
    "split_frac = 0.8\n",
    "\n",
    "## split data into training, validation, and test data (features and labels, x and y)\n",
    "\n",
    "split_idx = int(len(features)*split_frac)\n",
    "train_x, remaining_x = features[:split_idx], features[split_idx:]\n",
    "train_y, remaining_y = encoded_labels[:split_idx], encoded_labels[split_idx:]\n",
    "\n",
    "test_idx = int(len(remaining_x)*0.5)\n",
    "val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]\n",
    "val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]\n",
    "\n",
    "## print out the shapes of the resultant feature data\n",
    "print(\"\\t\\t\\tFeature Shapes:\")\n",
    "print(\"Train set: \\t\\t{}\".format(train_x.shape), \n",
    "      \"\\nValidation set: \\t{}\".format(val_x.shape),\n",
    "      \"\\nTest set: \\t\\t{}\".format(test_x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoaders and Batching\n",
    "After creating training, test, and validation data, we can create DataLoaders for this data by following two steps:\n",
    "1. Create a known format for accessing our data, using [TensorDataset](https://pytorch.org/docs/stable/data.html#) which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.\n",
    "2. Create DataLoaders and batch our training, validation, and test Tensor datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# create Tensor datasets\n",
    "train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))\n",
    "test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))\n",
    "\n",
    "# dataloaders\n",
    "batch_size = 50\n",
    "\n",
    "# make sure the SHUFFLE the training data\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\n",
    "valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample input size:  torch.Size([50, 30])\n",
      "Sample input: \n",
      " tensor([[    0,     0,     0,  ..., 14227,     5,   104],\n",
      "        [    0,     0,     0,  ...,    24,   418,   771],\n",
      "        [    0,     0,     0,  ...,     7,     4,   141],\n",
      "        ...,\n",
      "        [    0,     0,     0,  ...,    58,     5,   276],\n",
      "        [    0,     0,     0,  ...,    33,  4191,    35],\n",
      "        [    0,     0,     0,  ...,    81,   864, 10467]])\n",
      "\n",
      "Sample label size:  torch.Size([50])\n",
      "Sample label: \n",
      " tensor([0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1,\n",
      "        0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1,\n",
      "        0, 0])\n"
     ]
    }
   ],
   "source": [
    "# obtain one batch of training data\n",
    "dataiter = iter(train_loader)\n",
    "sample_x, sample_y = dataiter.next()\n",
    "\n",
    "print('Sample input size: ', sample_x.size()) # batch_size, seq_length\n",
    "print('Sample input: \\n', sample_x)\n",
    "print()\n",
    "print('Sample label size: ', sample_y.size()) # batch_size\n",
    "print('Sample label: \\n', sample_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Sentiment Network with PyTorch\n",
    "Below is where we'll define the network.\n",
    "\n",
    "<img src=\"assets/network_diagram.png\" width=40%>\n",
    "\n",
    "The layers are as follows:\n",
    "1. An [embedding layer](https://pytorch.org/docs/stable/nn.html#embedding) that converts our word tokens (integers) into embeddings of a specific size.\n",
    "2. An [LSTM layer](https://pytorch.org/docs/stable/nn.html#lstm) defined by a hidden_state size and number of layers\n",
    "3. A fully-connected output layer that maps the LSTM layer outputs to a desired output_size\n",
    "4. A sigmoid activation layer which turns all outputs into a value 0-1; return **only the last sigmoid output** as the output of this network.\n",
    "\n",
    "### The Embedding Layer\n",
    "\n",
    "We need to add an [embedding layer](https://pytorch.org/docs/stable/nn.html#embedding) because there are 16500+ words in our vocabulary. It is massively inefficient to one-hot encode that many classes. So, instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table. \n",
    "\n",
    "\n",
    "### The LSTM Layer(s)\n",
    "\n",
    "We'll create an [LSTM](https://pytorch.org/docs/stable/nn.html#lstm) to use in our recurrent network, which takes in an input_size, a hidden_dim, a number of layers, a dropout probability (for dropout between multiple layers), and a batch_first parameter.\n",
    "\n",
    "Most of the time, the network will have better performance with more layers; between 2-3. Adding more layers allows the network to learn really complex relationships.\n",
    "\n",
    "Note: `init_hidden` should initialize the hidden and cell state of an lstm layer to all zeros, and move those state to GPU, if available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU available, training on CPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class SentimentRNN(nn.Module):\n",
    "    \"\"\"\n",
    "    The RNN model that will be used to perform Sentiment analysis.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):\n",
    "        \"\"\"\n",
    "        Initialize the model by setting up the layers.\n",
    "        \"\"\"\n",
    "        super(SentimentRNN, self).__init__()\n",
    "\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_dim = hidden_dim\n",
    "        \n",
    "        # embedding and LSTM layers\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, \n",
    "                            dropout=drop_prob, batch_first=True)\n",
    "        \n",
    "        # dropout layer\n",
    "        self.dropout = nn.Dropout(0.3)\n",
    "        \n",
    "        # linear and sigmoid layers\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "        self.sig = nn.Sigmoid()\n",
    "        \n",
    "\n",
    "    def forward(self, x, hidden):\n",
    "        \"\"\"\n",
    "        Perform a forward pass of our model on some input and hidden state.\n",
    "        \"\"\"\n",
    "        batch_size = x.size(0)\n",
    "\n",
    "        # embeddings and lstm_out\n",
    "        x = x.long()\n",
    "        embeds = self.embedding(x)\n",
    "        lstm_out, hidden = self.lstm(embeds, hidden)\n",
    "    \n",
    "        # stack up lstm outputs\n",
    "        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        # dropout and fully-connected layer\n",
    "        out = self.dropout(lstm_out)\n",
    "        out = self.fc(out)\n",
    "        # sigmoid function\n",
    "        sig_out = self.sig(out)\n",
    "        \n",
    "        # reshape to be batch_size first\n",
    "        sig_out = sig_out.view(batch_size, -1)\n",
    "        sig_out = sig_out[:, -1] # get last batch of labels\n",
    "        \n",
    "        # return last sigmoid output and hidden state\n",
    "        return sig_out, hidden\n",
    "    \n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        ''' Initializes hidden state '''\n",
    "        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,\n",
    "        # initialized to zero, for hidden state and cell state of LSTM\n",
    "        weight = next(self.parameters()).data\n",
    "        \n",
    "        if (train_on_gpu):\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),\n",
    "                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())\n",
    "        else:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())\n",
    "        \n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate the network\n",
    "Here, we'll instantiate the network. First up, defining the hyperparameters.\n",
    "\n",
    "* `vocab_size`: Size of our vocabulary or the range of values for our input, word tokens.\n",
    "* `output_size`: Size of our desired output; the number of class scores we want to output (negative/non-negative).\n",
    "* `embedding_dim`: Number of columns in the embedding lookup table; size of our embeddings.\n",
    "* `hidden_dim`: Number of units in the hidden layers of our LSTM cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.\n",
    "* `n_layers`: Number of LSTM layers in the network. Typically between 1-3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SentimentRNN(\n",
      "  (embedding): Embedding(16728, 200)\n",
      "  (lstm): LSTM(200, 128, num_layers=2, batch_first=True, dropout=0.5)\n",
      "  (dropout): Dropout(p=0.3)\n",
      "  (fc): Linear(in_features=128, out_features=1, bias=True)\n",
      "  (sig): Sigmoid()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model w/ hyperparams\n",
    "vocab_size = len(vocab_to_int)+1 # +1 for the 0 padding + our word tokens\n",
    "output_size = 1\n",
    "embedding_dim = 200\n",
    "hidden_dim = 128\n",
    "n_layers = 2\n",
    "\n",
    "net = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Below is the typical training code. We'll use a cross entropy loss, which is designed to work with a single Sigmoid output. [BCELoss](https://pytorch.org/docs/stable/nn.html#bceloss), or **Binary Cross Entropy Loss**, applies cross entropy loss to a single value between 0 and 1. We also have some data and training hyparameters:\n",
    "\n",
    "* `lr`: Learning rate for our optimizer.\n",
    "* `epochs`: Number of times to iterate through the training dataset.\n",
    "* `clip`: The maximum gradient value to clip at (to prevent exploding gradients)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# loss and optimization functions\n",
    "lr=0.001\n",
    "\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/10... Step: 100... Loss: 0.506128... Val Loss: 0.492362\n",
      "Epoch: 1/10... Step: 200... Loss: 0.570771... Val Loss: 0.438589\n",
      "Epoch: 2/10... Step: 300... Loss: 0.258980... Val Loss: 0.426052\n",
      "Epoch: 2/10... Step: 400... Loss: 0.325954... Val Loss: 0.414884\n",
      "Epoch: 3/10... Step: 500... Loss: 0.312811... Val Loss: 0.425056\n",
      "Epoch: 3/10... Step: 600... Loss: 0.314457... Val Loss: 0.448399\n",
      "Epoch: 4/10... Step: 700... Loss: 0.178827... Val Loss: 0.505257\n",
      "Epoch: 4/10... Step: 800... Loss: 0.127423... Val Loss: 0.463786\n",
      "Epoch: 5/10... Step: 900... Loss: 0.103283... Val Loss: 0.574637\n",
      "Epoch: 5/10... Step: 1000... Loss: 0.085955... Val Loss: 0.731613\n",
      "Epoch: 5/10... Step: 1100... Loss: 0.106801... Val Loss: 0.582090\n",
      "Epoch: 6/10... Step: 1200... Loss: 0.120974... Val Loss: 0.697028\n",
      "Epoch: 6/10... Step: 1300... Loss: 0.048913... Val Loss: 0.828894\n",
      "Epoch: 7/10... Step: 1400... Loss: 0.033686... Val Loss: 0.819520\n",
      "Epoch: 7/10... Step: 1500... Loss: 0.145023... Val Loss: 0.793312\n",
      "Epoch: 8/10... Step: 1600... Loss: 0.035300... Val Loss: 0.858518\n",
      "Epoch: 8/10... Step: 1700... Loss: 0.209686... Val Loss: 0.894603\n",
      "Epoch: 9/10... Step: 1800... Loss: 0.104914... Val Loss: 0.863970\n",
      "Epoch: 9/10... Step: 1900... Loss: 0.008456... Val Loss: 1.026853\n",
      "Epoch: 9/10... Step: 2000... Loss: 0.015449... Val Loss: 0.905517\n",
      "Epoch: 10/10... Step: 2100... Loss: 0.003005... Val Loss: 1.109386\n",
      "Epoch: 10/10... Step: 2200... Loss: 0.022017... Val Loss: 0.904183\n"
     ]
    }
   ],
   "source": [
    "# training params\n",
    "\n",
    "epochs = 10 \n",
    "\n",
    "counter = 0\n",
    "print_every = 100\n",
    "clip=5 # gradient clipping\n",
    "\n",
    "# move model to GPU, if available\n",
    "if(train_on_gpu):\n",
    "    net.cuda()\n",
    "\n",
    "net.train()\n",
    "# train for some number of epochs\n",
    "for e in range(epochs):\n",
    "    # initialize hidden state\n",
    "    h = net.init_hidden(batch_size)\n",
    "\n",
    "    # batch loop\n",
    "    for inputs, labels in train_loader:\n",
    "        counter += 1\n",
    "\n",
    "        if(train_on_gpu):\n",
    "            inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "        # Creating new variables for the hidden state, otherwise\n",
    "        # we'd backprop through the entire training history\n",
    "        h = tuple([each.data for each in h])\n",
    "\n",
    "        # zero accumulated gradients\n",
    "        net.zero_grad()\n",
    "\n",
    "        # get the output from the model\n",
    "        output, h = net(inputs, h)\n",
    "\n",
    "        # calculate the loss and perform backprop\n",
    "        loss = criterion(output.squeeze(), labels.float())\n",
    "        loss.backward()\n",
    "        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n",
    "        nn.utils.clip_grad_norm_(net.parameters(), clip)\n",
    "        optimizer.step()\n",
    "\n",
    "        # loss stats\n",
    "        if counter % print_every == 0:\n",
    "            # Get validation loss\n",
    "            val_h = net.init_hidden(batch_size)\n",
    "            val_losses = []\n",
    "            net.eval()\n",
    "            for inputs, labels in valid_loader:\n",
    "\n",
    "                # Creating new variables for the hidden state, otherwise\n",
    "                # we'd backprop through the entire training history\n",
    "                val_h = tuple([each.data for each in val_h])\n",
    "\n",
    "                if(train_on_gpu):\n",
    "                    inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "                output, val_h = net(inputs, val_h)\n",
    "                val_loss = criterion(output.squeeze(), labels.float())\n",
    "\n",
    "                val_losses.append(val_loss.item())\n",
    "\n",
    "            net.train()\n",
    "            print(\"Epoch: {}/{}...\".format(e+1, epochs),\n",
    "                  \"Step: {}...\".format(counter),\n",
    "                  \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                  \"Val Loss: {:.6f}\".format(np.mean(val_losses)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing\n",
    "\n",
    "We'll see how our trained model performs on all of our defined test_data, above. We'll calculate the average loss and accuracy over the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.894\n",
      "Test accuracy: 0.815\n"
     ]
    }
   ],
   "source": [
    "# Get test data loss and accuracy\n",
    "\n",
    "test_losses = [] # track loss\n",
    "num_correct = 0\n",
    "\n",
    "# init hidden state\n",
    "h = net.init_hidden(batch_size)\n",
    "\n",
    "net.eval()\n",
    "# iterate over test data\n",
    "for inputs, labels in test_loader:\n",
    "\n",
    "    # Creating new variables for the hidden state, otherwise\n",
    "    # we'd backprop through the entire training history\n",
    "    h = tuple([each.data for each in h])\n",
    "\n",
    "    if(train_on_gpu):\n",
    "        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "    \n",
    "    # get predicted outputs\n",
    "    output, h = net(inputs, h)\n",
    "    \n",
    "    # calculate loss\n",
    "    test_loss = criterion(output.squeeze(), labels.float())\n",
    "    test_losses.append(test_loss.item())\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze())  # rounds to the nearest integer\n",
    "    \n",
    "    # compare predictions to true label\n",
    "    correct_tensor = pred.eq(labels.float().view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())\n",
    "    num_correct += np.sum(correct)\n",
    "\n",
    "\n",
    "# -- stats! -- ##\n",
    "# avg test loss\n",
    "print(\"Test loss: {:.3f}\".format(np.mean(test_losses)))\n",
    "\n",
    "# accuracy over all test data\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print(\"Test accuracy: {:.3f}\".format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Creating a predict function\n",
    "We'll write a predict function that takes in a trained net, a plain text_review, and a sequence length, and prints out a custom statement for a non-negative or negative review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# negative test review\n",
    "test_review = \"@AmericanAir you have my money, you change my flight, and don't answer your phones! Any other suggestions so I can make my commitment??\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5, 22, 11, 367, 5, 126, 11, 8, 10, 85, 335, 21, 922, 93, 194, 1550, 44, 3, 34, 125, 11, 2888]]\n"
     ]
    }
   ],
   "source": [
    "def tokenize_review(test_review):\n",
    "    test_review = test_review.lower() # lowercase\n",
    "    # get rid of punctuation\n",
    "    test_text = ''.join([c for c in test_review if c not in punctuation])\n",
    "\n",
    "    # splitting by spaces\n",
    "    test_words = test_text.split()\n",
    "    \n",
    "    # get rid of web address, twitter id, and digit\n",
    "    new_text = []\n",
    "    for word in test_words:\n",
    "        if (word[0] != '@') & ('http' not in word) & (~word.isdigit()):\n",
    "            new_text.append(word)\n",
    "\n",
    "    # tokens\n",
    "    test_ints = []\n",
    "    test_ints.append([vocab_to_int[word] for word in new_text])\n",
    "\n",
    "    return test_ints\n",
    "\n",
    "# test code and generate tokenized review\n",
    "test_ints = tokenize_review(test_review)\n",
    "print(test_ints)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   0    0    0    0    0    0    0    0    5   22   11  367    5  126\n",
      "    11    8   10   85  335   21  922   93  194 1550   44    3   34  125\n",
      "    11 2888]]\n"
     ]
    }
   ],
   "source": [
    "# test sequence padding\n",
    "seq_length=30\n",
    "features = pad_features(test_ints, seq_length)\n",
    "\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 30])\n"
     ]
    }
   ],
   "source": [
    "# test conversion to tensor and pass into your model\n",
    "feature_tensor = torch.from_numpy(features)\n",
    "print(feature_tensor.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict(net, test_review, sequence_length=30):\n",
    "    \n",
    "    net.eval()\n",
    "    \n",
    "    # tokenize review\n",
    "    test_ints = tokenize_review(test_review)\n",
    "    \n",
    "    # pad tokenized sequence\n",
    "    seq_length=sequence_length\n",
    "    features = pad_features(test_ints, seq_length)\n",
    "    \n",
    "    # convert to tensor to pass into your model\n",
    "    feature_tensor = torch.from_numpy(features)\n",
    "    \n",
    "    batch_size = feature_tensor.size(0)\n",
    "    \n",
    "    # initialize hidden state\n",
    "    h = net.init_hidden(batch_size)\n",
    "    \n",
    "    if(train_on_gpu):\n",
    "        feature_tensor = feature_tensor.cuda()\n",
    "    \n",
    "    # get the output from the model\n",
    "    output, h = net(feature_tensor, h)\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze()) \n",
    "    # printing output value, before rounding\n",
    "    print('Prediction value, pre-rounding: {:.6f}'.format(output.item()))\n",
    "    \n",
    "    # print custom response\n",
    "    if(pred.item()==1):\n",
    "        print(\"Non-negative review detected.\")\n",
    "    else:\n",
    "        print(\"Negative review detected.\")\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "seq_length = 30 # good to use the length that was trained on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.000080\n",
      "Negative review detected.\n"
     ]
    }
   ],
   "source": [
    "# call function on negative review\n",
    "test_review_neg = \"@AmericanAir you have my money, you change my flight, and don't answer your phones! Any other suggestions so I can make my commitment??\"\n",
    "predict(net, test_review_neg, seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.999499\n",
      "Non-negative review detected.\n"
     ]
    }
   ],
   "source": [
    "# call function on positive review\n",
    "test_review_pos = \"@AmericanAir thank you we got on a different flight to Chicago.\"\n",
    "predict(net, test_review_pos, seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.988350\n",
      "Non-negative review detected.\n"
     ]
    }
   ],
   "source": [
    "# call function on neutral review\n",
    "test_review_neu = \"@AmericanAir i need someone to help me out\"\n",
    "predict(net, test_review_neu, seq_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a trained model and a predict function, you can pass in _any_ kind of text and this model will predict whether the text has a positive or negative sentiment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}