asgn02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "a7dae92407ae88ef7b05311c26297b24",
     "grade": false,
     "grade_id": "cell-8e42a94857e3371d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Document Analysis Assignment 2: Youtube Video Engagement Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Your Information\n",
    "Please fill in the following information:\n",
    "\n",
    "- **Name**:    [You Li]\n",
    "\n",
    "- **Uni id**:  [u6430173]\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "adc73f6a85b91c39e1e1e0a67b51b328",
     "grade": false,
     "grade_id": "cell-a279c7f2b6d01db8",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Overview\n",
    "The engagement of a Youtube video is a number between 0 and 1, and it can be defined as the percentage of the video that its audience watches in average. \n",
    "For more information please refer to this recent paper and its presentation:\n",
    "```\n",
    "S. Wu, M.-A. Rizoiu, & L. Xie, \"Beyond Views: Measuring and Predicting Engagement in Online Videos, \" in Proc. International AAAI Conference on Web and Social Media (ICWSM ’18), Stanford, CA, USA, 2018.\n",
    "```\n",
    "[link to paper](https://arxiv.org/pdf/1709.02541.pdf)\n",
    "[paper presentation](http://www.rizoiu.eu/documents/research/presentations/WU_ICWSM-2018_slides.pdf)\n",
    "\n",
    "Your task in this assignment is to predict a Youtube video's engagement by using only the textual infomation related to it (i.e. its description, its author name also called *channelTitle*, and the *videoTitle*).\n",
    "Predicting a continuous variable is called a **regression** task, very similar to the classification tasks presented in the lectures.\n",
    "Techniques such as Linear Regression, SVM, kNN and tree-based methods have been adapted for regression and are readily implemented in packages such as [`scikit-learn`](http://scikit-learn.org/). \n",
    "\n",
    "To complete this assignment, you will need to follow the following steps:\n",
    "\n",
    "1. Read and preprocess the training and testing dataset in order to extract text data.\n",
    "2. Train and apply regression models and compare their performance.\n",
    "3. **(optional, but recommended)** tune your models' hyper-parameters to improve their performance.\n",
    "4. Generate your result file in the required format.\n",
    "5. After solving the coding part, you need to answer three questions in written part.\n",
    "\n",
    "(**hint: you can reuse code given in the tutorial**)\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The dataset provided on Wattle contains two folders: **train** and **test**. \n",
    "Each folder contains the videos in the training set and the testing set respectively.\n",
    "Each video is represented as a document and it is described as a json file, entitled `[id].json`. \n",
    "See an example below: \n",
    "```\n",
    "{\n",
    "   \"channelTitle\":\"K KIRK PRODUCTIONS\",\n",
    "   \"description\":\"New video from Reggie Records Starring FOE Lil Reggie & PESO.SOUNDCLOUD: FOE LIL REGGIE INSTAGRAM: true_stvr\",\n",
    "   \"engagementScore\":0.792,\n",
    "   \"videoTitle\":\"FOE Lil Reggie ft. PESO-Keep It Going (OFFICIAL VIDEO)\"\n",
    "}\n",
    "```\n",
    "In the training set, the response variable is in the field called `engagementScore` -- i.e. the engagement score of the video.\n",
    "Your task is to predict the `engagementScore` for each document in the testing set. \n",
    "\n",
    "## Evaluation\n",
    "\n",
    "The performance of your prediction will be evaluated automatically on Kaggle using the Root Mean Square Error (RMSE) performance measure.\n",
    "RMSE measures the deviation between your predictions and the ground truth, and [it is defined as](http://statweb.stanford.edu/~susan/courses/s60/split/node60.html):\n",
    "\n",
    "$$\n",
    "    \\operatorname {RMSE} ={\\sqrt {\\frac {\\sum _{i=1}^{n}({\\hat {y}}_{i}-y_{i})^{2}}{n}}}.\n",
    "$$\n",
    "\n",
    "where $n$ is the number of data points, $y_i$ is the true engagement score for video $i$ and $\\hat{y}_i$ is the predicted engagement score for video $i$.\n",
    "\n",
    "Your score will be computed using a lower bound and an upper bound, which will be shown on the Kaggle leader board. \n",
    "Achieving a RMSE score equal to the lower bound amount to a grade of zero, while achieving the upper bound amounts to the full points (here 7 points, see score distribution here below).\n",
    "Consequently, your score for this competition task will be calculated based on:\n",
    "\n",
    "$$\n",
    "    \\operatorname{Your\\_Score} = \\frac{Lower\\_Bound - Your\\_Performance}{Lower\\_Bound - Upper\\_Bound} * 7\n",
    "$$\n",
    "\n",
    "Notes about the lower bound and upper bounds predictors:\n",
    "\n",
    "* The **lower bound** is the performance obtained by a regressor that always makes the *random* guess (i.e. one that always predicts the mean engagement score in the training set).\n",
    "* The **upper bound** \"in-house\" regressor was built in a couple of hours by one of your fellow students, on the same dataset that you were given.\n",
    "No exotic tricks were used for this regressor, and it is possible to obtain better results than this.\n",
    "If you obtain a better performance than the upper bound, then you will have a grade higher than 7 points for the coding part. This can be useful to compensate for any lost points for the written questions.\n",
    "Note however, that the total grade of this assignment is capped at 10 marks.\n",
    "\n",
    "Note that here \"min\" and \"max\" refer to performance, and RMSE needs to be minimized, i.e. max performance RMSE is lower than min performance RMSE.\n",
    "\n",
    "## For the Kaggle competition\n",
    "\n",
    "- Join the competition [here](https://www.kaggle.com/t/f31b296a987c46dca752753eb6fde2b3)\n",
    "- Before submitting the result, first go to **team** menu and change your **team name** as **your university id**.\n",
    "- You need to upload the generated result file to Kaggle. The result file should be in the following format\n",
    "```\n",
    "id,engagementScore\n",
    "abcdefgh,0.01\n",
    "hijk1234,0.02\n",
    "lmno5678,0.3\n",
    "...\n",
    "```\n",
    "- Note that you are only allowed to upload **5 copies** of your results to Kaggle per day. Make every upload count, and don't waste your opportunities!\n",
    "- You should use cross-validation instead of relying on the public set - this is what the daily limit is for!\n",
    "- For detailed submission instructions, check the end of this notebook.\n",
    "\n",
    "Score distribution (total 10 points):\n",
    "\n",
    "- Kaggle competition: 7 points\n",
    "- Written part Q1: 1 point\n",
    "- Written part Q2: 1 point\n",
    "- Written part Q3: 1 point\n",
    "\n",
    "After completion, please rename this notebook to **`your_uid.ipynb` (e.g. `u6000001.ipynb`)** and submit this file to Wattle. Do not upload any other files to Wattle except this notebook file.\n",
    "\n",
    "**Note:** you need to fill in the cells below with your code. Failure to provide your code nullifies your Kaggle grade (meaning you get zero for the coding part)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "e5ffa31c0127c094add8bfb261f3fd94",
     "grade": false,
     "grade_id": "cell-a46ba2805e2b46f6",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## 1. Loading dataset and preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "440a0dc19d1b1eb1b5402a9120126bee",
     "grade": true,
     "grade_id": "cell-d8b7796e3cd82200",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.set_printoptions(suppress=True)\n",
    "# 1. Read and preprocess the training and testing dataset in order to extract text data.\n",
    "# 2. Train and apply regression models and compare their performance.\n",
    "# 3. **(optional, but recommended)** tune your models' hyper-parameters to improve their performance.\n",
    "# 4. Generate your result file in the required format.\n",
    "# 5. After solving the coding part, you need to answer three questions in written part.\n",
    "# import matplotlib.pyplot as plt\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import pandas as pd\n",
    "from collections import namedtuple\n",
    "from io import open\n",
    "# import os, io\n",
    "import json\n",
    "## A document class with following attributes\n",
    "## id: document id\n",
    "## category: category of document\n",
    "## text: body of documment\n",
    "# Doc = namedtuple('Doc', 'id category text')\n",
    "Doc = namedtuple('Doc', 'id channelTitle description engagementScore videoTitle')\n",
    "\n",
    "def read_doc(doc_path, encoding):\n",
    "    '''\n",
    "        reads a document from path\n",
    "        input:\n",
    "            - doc_path : path of document\n",
    "            - encoding: encoding\n",
    "        output: =>\n",
    "            - doc: instance of Doc namedtuple\n",
    "    '''\n",
    "    category, _id = tuple(doc_path.split('/')[-2:])\n",
    "    _id = _id.split('.')[0]\n",
    "    \n",
    "    fp = open(doc_path, 'r', encoding = encoding)\n",
    "    json_data = json.load(fp)\n",
    "    fp.close()\n",
    "    ct = json_data['channelTitle']\n",
    "    ds = json_data['description']\n",
    "    if category == \"train\":\n",
    "        es = json_data['engagementScore']\n",
    "    else: es = 0\n",
    "    vt= json_data['videoTitle']\n",
    "#     print(json_data)\n",
    "    return Doc(id = _id, channelTitle=ct ,description=ds , engagementScore=es , videoTitle=vt )\n",
    "\n",
    "def read_dataset(path, encoding = \"ISO-8859-1\"):\n",
    "    '''\n",
    "        reads multiple documents from path\n",
    "        input:\n",
    "            - doc_path : path of document\n",
    "            - encoding: encoding\n",
    "        output: =>\n",
    "            - docs: instances of Doc namedtuple returned as generator\n",
    "    '''\n",
    "    for doc_path in glob.glob(path + '/*'):\n",
    "        yield read_doc(doc_path, encoding = encoding)\n",
    "\n",
    "def read_as_df(path, encoding = \"ISO-8859-1\"):\n",
    "    '''\n",
    "        reads multiple documents from path\n",
    "        input:\n",
    "            - doc_path : path of document\n",
    "            - encoding: encoding\n",
    "        output: =>\n",
    "            - docs: dataframe equivalent of doc Namedtuple\n",
    "    '''\n",
    "    dataset = list(read_dataset(path, encoding))\n",
    "    return pd.DataFrame(dataset, columns = Doc._fields)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## specify the path to the dataset two_newsgroup\n",
    "path_to_train_dataset = './dataset/train'\n",
    "path_to_test_dataset = './dataset/test'\n",
    "## TODO stop\n",
    "train_dataset = read_as_df(path_to_train_dataset)\n",
    "print(\"Number of rows and columns of the dataset: {}\".format(train_dataset.shape))\n",
    "# print(\"The first five documents:\")\n",
    "# train_dataset.head()\n",
    "test_dataset = read_as_df(path_to_test_dataset)\n",
    "print(\"Number of rows and columns of the dataset: {}\".format(test_dataset.shape))\n",
    "# print(\"The first five documents:\")\n",
    "test_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "## Stemmer\n",
    "stemmer = SnowballStemmer(\"english\")\n",
    "stopwords_en = set(stopwords.words('english'))\n",
    "stopwords_en.update([\"n\", \"www\", \"http\", \"https\", \"nhttps\", \"com\", \"r\", \"v\", \"ft\"])\n",
    "# remove the unecessaty words\n",
    "## now build a custom tokenizer based on these\n",
    "__tokenization_pattern = r'''(?x)          # set flag to allow verbose regexps\n",
    "        \\$?\\d+(?:\\.\\d+)?%?  # currency and percentages, e.g. $12.40, 82%\n",
    "      | (?:[A-Z]\\.)+        # abbreviations, e.g. U.S.A.\n",
    "      | (?:\\\\x\\w\\w)+        # ignore unicode\n",
    "      | \\w+(?:-\\w+)*        # words with optional internal hyphens\n",
    "      | \\.\\.\\.              # ellipsis\n",
    "      | [][.,;\"'?():_`-]    # these are separate tokens; includes ], [\n",
    "    '''\n",
    "\n",
    "tokenizer = nltk.tokenize.regexp.RegexpTokenizer(__tokenization_pattern)\n",
    "\n",
    "def preprocessor(text):\n",
    "    '''\n",
    "        turns text into tokens after tokenization, stemming, stop words removal\n",
    "        imput:\n",
    "            - text: document to process\n",
    "        output: =>\n",
    "            - tokens: list of tokens after tokenization, stemming, stop words removal\n",
    "    '''\n",
    "    stems = []\n",
    "    tokens = tokenizer.tokenize(text.encode('utf-8').lower())\n",
    "\n",
    "    for token in tokens:\n",
    "        if token.isalpha() and token not in stopwords_en:\n",
    "            stems.append(str(stemmer.stem(token)))\n",
    "    return stems\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset['cttk'] = train_dataset['channelTitle'].apply(preprocessor)\n",
    "train_dataset['dstk'] = train_dataset['description'].apply(preprocessor)\n",
    "train_dataset['vttk'] = train_dataset['videoTitle'].apply(preprocessor)\n",
    "# train_dataset.head()\n",
    "\n",
    "test_dataset['cttk'] = test_dataset['channelTitle'].apply(preprocessor)\n",
    "test_dataset['dstk'] = test_dataset['description'].apply(preprocessor)\n",
    "test_dataset['vttk'] = test_dataset['videoTitle'].apply(preprocessor)\n",
    "# test_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import chain\n",
    "import nltk\n",
    "import matplotlib.pyplot as plt\n",
    "train_dist = nltk.FreqDist(chain(*train_dataset['dstk']))\n",
    "print(train_dist.most_common(10))\n",
    "\n",
    "plt.figure(figsize=(15, 6))  # the size you want\n",
    "train_dist.plot(50,cumulative=False)\n",
    "\n",
    "test_dist = nltk.FreqDist(chain(*test_dataset['dstk']))\n",
    "print(test_dist.most_common(10))\n",
    "\n",
    "plt.figure(figsize=(15, 6))  # the size you want\n",
    "test_dist.plot(50,cumulative=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_train_tokens = set(chain(*train_dataset['cttk'] + train_dataset['dstk'] + train_dataset['vttk']))\n",
    "print(\"number of tokens in the train dataset: {}\".format(len(all_train_tokens)))\n",
    "all_test_tokens = set(chain(*test_dataset['cttk'] + test_dataset['dstk'] + test_dataset['vttk']))\n",
    "print(\"number of tokens in the test dataset: {}\".format(len(all_test_tokens)))\n",
    "\n",
    "train_dataset['features'] = (train_dataset['cttk'] + train_dataset['dstk'] + train_dataset['vttk']).apply(set)\n",
    "test_dataset['features'] = (test_dataset['cttk'] + test_dataset['dstk'] + test_dataset['vttk']).apply(set)\n",
    "# test_dataset.head()\n",
    "test_dataset['tokens'] = test_dataset['cttk'] + test_dataset['dstk'] + test_dataset['vttk']\n",
    "test_dataset.head()\n",
    "\n",
    "train_dataset['tokens'] = train_dataset['cttk'] + train_dataset['dstk'] + train_dataset['vttk']\n",
    "train_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "06b8a5ad959b8703fc48fde4385d4901",
     "grade": false,
     "grade_id": "cell-eac84e796f175379",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## 2. Apply regression models and select models "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "478aeecb1457b978e0e37288dfbdfe21",
     "grade": true,
     "grade_id": "cell-b367bb77894c52bd",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn import datasets, linear_model, decomposition, svm, neighbors\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "import math\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# train_vec = bow_vectorizer.fit_transform(train_dataset.tokens)\n",
    "# train_ct_vec = bow_vectorizer.fit_transform(train_dataset.cttk)\n",
    "# train_ds_vec = bow_vectorizer.fit_transform(train_dataset.dstk)\n",
    "# train_vt_vec = bow_vectorizer.fit_transform(train_dataset.vttk)\n",
    "\n",
    "# test_vec = bow_vectorizer.fit_transform(test_dataset.tokens)\n",
    "# test_ct_vec = bow_vectorizer.fit_transform(test_dataset.cttk)\n",
    "# test_ds_vec = bow_vectorizer.fit_transform(test_dataset.dstk)\n",
    "# test_vt_vec = bow_vectorizer.fit_transform(test_dataset.vttk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_vectorizer = CountVectorizer(lowercase = False, \n",
    "                                     tokenizer = lambda x: x, # because we already have tokens available\n",
    "                                     stop_words = None, ## stop words removal already done from NLTK\n",
    "                                     max_features = 8000, ## pick top 5K words by frequency\n",
    "                                     ngram_range = (1, 3), ## we want unigrams now\n",
    "                                     binary = False) ## we want as binary/boolean features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lnrreg.fit(train_ct_vec, train_ds_vec, train_vt_vec, train_dataset['engagementScore'],[0.5, 0.3, 0.2])\n",
    "# lnrreg.fit(train_vec, train_dataset['engagementScore'])\n",
    "\n",
    "train_X, test_X, train_y, test_y = train_test_split(train_dataset['tokens'], train_dataset['engagementScore'], test_size=0.2)\n",
    "# test_dataset['engagementScore'] = lnrreg.predict(test_vec)\n",
    "# breg = linear_model.BayesianRidge()\n",
    "# breg.fit(train_X, train_y)\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('train_vec',  bow_vectorizer),\n",
    "    ('tfidf',  TfidfTransformer(sublinear_tf=True)),\n",
    "#     ('linear-regression',  linear_model.LinearRegression()) ])\n",
    "#     ('svm', svm.SVR()) ])\n",
    "#     ('lnrsvm', svm.LinearSVR()) ])\n",
    "#     ('en', linear_model.ElasticNet()) ])\n",
    "    ('knn', neighbors.KNeighborsRegressor(n_neighbors=10, weights='distance'))])\n",
    "\n",
    "pipeline.fit(train_X, train_y)\n",
    "test_pred = pipeline.predict(test_X)\n",
    "test_dataset['engagementScore'] = pipeline.predict(test_dataset['tokens'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(suppress=True)\n",
    "# test_pred = 1/(1+np.exp(-test_pred))\n",
    "print(test_pred)\n",
    "\n",
    "rmse = math.sqrt(mean_squared_error(test_y, test_pred))\n",
    "print(rmse)\n",
    "score = (0.28341-rmse)/(0.28341-0.22695)*7\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np.set_printoptions(suppress=True)\n",
    "# test_dataset['engagementScore'] = test_dataset['engagementScore'].apply(lambda x: 1/(1+math.exp(-x)))\n",
    "# test_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "e082fa0daec5f04c057acdc34bf97264",
     "grade": false,
     "grade_id": "cell-2e842d8f71e623c8",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## 3. Generate your result file for Kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "18278902f596214d5fdc2e5d85409fc2",
     "grade": true,
     "grade_id": "cell-6d4fff9a78082be9",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def write_output(output_file):\n",
    "    with open(output_file, 'w') as output:\n",
    "        output.write(u'id,engagementScore\\n')  \n",
    "#       read topic doc\n",
    "        for i in range(0, len(test_dataset['id'])):\n",
    "            index = test_dataset['id'][i]\n",
    "            egscr = test_dataset['engagementScore'][i]\n",
    "            output.write(u'{},{}\\n'.format(index,egscr))\n",
    "#     print(\"finished writing in output.csv\")\n",
    "           \n",
    "output_file = 'output.csv'\n",
    "write_output(output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d523ac9424255d84feafcfecb483578e",
     "grade": false,
     "grade_id": "cell-5675e41756e1913b",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## 4. Written part\n",
    "\n",
    "Answer briefly and concisely the following questions, based on your implementation from parts 1, 2 and 3.\n",
    "Provide answers using bullet list with 2~3 items.\n",
    "Check [this](https://sourceforge.net/p/jupiter/wiki/markdown_syntax/#md_ex_lists) if you are not familiar with markdown syntax.\n",
    "Each questions is worth 1 mark (10%) of the total grade for the ML assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "94fb6816b2497dab7b0a82b31d5e42ce",
     "grade": false,
     "grade_id": "cell-bc43b15ad6c6c09e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "### Question 1 (1pt)\n",
    " \n",
    "List the methods that you have tested and why. \n",
    "Why did you end up not using them and what was the reason they did not provide high performances?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "9812c1e1601975b42c065d06aa143663",
     "grade": true,
     "grade_id": "cell-5796bfbe4fb99745",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "* Methods have tested using train dataset with partition of 0.2 (This may different from Kaggle result): \n",
    "    * Linear regression: rmse around 0.26\n",
    "        * This is the basic regression model, and the performance is not as good as others since the training model may underfitting.\n",
    "    * ElasticNet: rmse around 0.28\n",
    "        * This linear regression model trained with L1 and L2 prior as regularizer, it also combined the regularization properties of Ridge. It would be useful if ct, des and vt were separately trained. Since the tokens were connected, hence this could not deliver the best performance.\n",
    "    * SVR: rmse around 0.24\n",
    "        * Ths model was chosen for further experiments. The preformance may not as well might result from overfitting.\n",
    "    * KNeighborsRegressor: rmse around 0.23\n",
    "        * This method was chosen for futeher experiments.\n",
    "    * BayesianBridge: rmse around 0.26\n",
    "        * estimates a probabilistic model of the regression problem, and the result is spherical Gaussian distributed. Hence its not suiteble for this question."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "8df7bbd009fb97ad8cc4f5bad2bad52a",
     "grade": false,
     "grade_id": "cell-1990ba6d5b6c9edd",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "### Question 2 (1pt)\n",
    "\n",
    "How did you select the best performing method? How did you tune model hyper-parameters?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "503bc062d547dbe71518a43a8bc4d331",
     "grade": true,
     "grade_id": "cell-b00b94e1856fc087",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "* The best performing method was selected based on the performance on RMSE, lower square error could prove that the estimation performance is near the real engagementScore. SVM, KNN regression has lowest square error as well as the most reliable result on both Kaggle score and the training dataset, and the experences were mainly run on these two regression methods.\n",
    "\n",
    "* Hyper-parameters were tuned in ``bow_vectorizer``. \n",
    "    * tokenizer and stop_words were done in the pre processing step, unicode and english stopwords were removed, so they were set as ``False``.\n",
    "    * For ``max_features``, high value may lead to over fitting and low value may result for underfitting. Since the amount of commenly used etyma is around 300, so this could explain the performance on ``max_features = 300`` is pretty good. Based on the experence performance which were held by setting the ``ngran_range`` to (1, 5); (1, 3) and change the ``max_features`` from 1 to 100,000 using dichotomy, suitable value for ``max_feature`` could be less than 500.\n",
    "    * Increasing the value of ``ngram_range`` could provide more combination of english words. Similarly, hold the `max_feature` and update `ngram_range`, the sutable value is (1, 3)\n",
    "\n",
    "* Take the example of `SVMRegressor`, according to sklearn documentation, changing the value of `kernel`, `degree`, and `max_iter` instead of using `__init__` helped improve the performance. Such experiments also held for `KNeighborsRegressor`. Different values for TfidfTransformer were also tested.\n",
    "\n",
    "\n",
    "```python \n",
    "class sklearn.svm.SVR(kernel=’rbf’, degree=3, gamma=’auto’, coef0=0.0, tol=0.001, C=1.0, epsilon=0.1, shrinking=True, cache_size=200, verbose=False, max_iter=-1)[source]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "bc159a8663c231e2c053e4b546363795",
     "grade": false,
     "grade_id": "cell-245eb08a972a07b6",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "### Question 3 (1pt)\n",
    "\n",
    "The target variable was in the range [0, 1], however most regressors output values in $(-\\infty, +\\infty)$.\n",
    "How did you solve this issue?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "8259c7de1c552c3608a0dab7693da71c",
     "grade": true,
     "grade_id": "cell-2c4be9c366cc4c7a",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "* While using prediction by LinearRegrassion() method, the regressor original output was ranged between $(-\\infty, +\\infty)$. This could be solved by using standard logistic function.\n",
    "$$ z = log (\\frac{p}{1-p}) $$\n",
    "$$ \\frac{p}{1-p} = e^z$$\n",
    "$$ p = \\frac{1}{1 + e^{-z}}$$\n",
    "* in python, this could be calculated by following\n",
    "```python\n",
    "test_dataset['engagementScore'] = test_dataset['engagementScore'].apply(lambda x: 1/(1+math.exp(-x)))\n",
    "```\n",
    "* Overall, logistic regression maps the point x in dimensional feature space to $(0, 1)$ ranged value. Using simply $z = p$ or $z = log p$ is invalid, because $e^z \\in (0, \\infty)$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "e2ad8ccec2933e27d20e2f7cbef12c75",
     "grade": false,
     "grade_id": "cell-c2ebd75ee43d1be0",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Upload output file to Kaggle competition site\n",
    "\n",
    "Once you generate `output.csv` file, you can upload your result on Kaggle competition site. To upload and evaluate your result\n",
    "\n",
    "1. Go to Kaggle competition site: [Click here](https://www.kaggle.com/t/f31b296a987c46dca752753eb6fde2b3).\n",
    "1. Sign up for Kaggle if you do not have an account. Go back to the [original kaggle page](https://www.kaggle.com/t/f31b296a987c46dca752753eb6fde2b3).\n",
    "1. Before submitting the result, first go to `team` menu and change **your team name as your university id**.\n",
    "![ChangeUID](images/changeuid.png)\n",
    "1. Time to submit your own result. Click `submit predictions` in the menu, you may need to agree the competition rules before submitting your result.\n",
    "1. Upload your output csv file, you can write additional description of your submission in the description box.\n",
    "    Note that you are only allowed to submit **3 results per day**. Do not upload an arbitrary result and think which algorithm or parser will perform the best.\n",
    "1. If your output format is correct, the system will generate your score automatically.\n",
    "1. Go to `Leaderboard` menu. The leaderboard will show the current score of the other students.\n",
    "![Leaderboard](images/leaderboard.png)\n",
    "\n",
    "\n",
    "Note that you can check all of your submission from `my submission` menu. Please select one best performing submission before the assignment due. The selected submission will be used to measure the performance of *hidden* test case (see below for details).\n",
    "![Check](images/check.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}