Machine Learning Landscape.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Scikit-Learn is a fast-changing library and generates API warnings all over\n",
    "# the place - disable for now so it doesn't clutter our workspace\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST Dataset\n",
    "\n",
    "The MNIST dataset is a set of 70,000 28x28 images of handwritten digits (0-9), with the first 60,000 images as the training set and the remaining 10,000 as the test set (i.e. the dataset has been shuffled for us already.)\n",
    "\n",
    "Let's download the dataset, pick a random sample and take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "\n",
    "mnist = fetch_mldata('MNIST original', data_home='/scratch/scikit_learn_data')\n",
    "X, y = mnist[\"data\"], mnist[\"target\"]\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "\n",
    "some_digit = X[12345]\n",
    "\n",
    "plt.imshow(some_digit.reshape(28,28), cmap=matplotlib.cm.binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split into training/testing sets.\n",
    "\n",
    "For this classification task, we'll try to classify each digit as a 5/not-5. Before doing anything else, set some test data aside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# No. of training samples\n",
    "m = 60000\n",
    "\n",
    "X_train, X_test, y_train, y_test = X[:m], X[m:], y[:m], y[m:]\n",
    "shuffle_index = np.random.permutation(m)\n",
    "X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]\n",
    "\n",
    "y_train_5 = (y_train == 5)\n",
    "y_test_5 = (y_test == 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stochastic Gradient Descent Classifier\n",
    "\n",
    "Train a SGDClassifier on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "classifier = SGDClassifier(random_state=33)\n",
    "classifier.fit(X_train, y_train_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see how it did on our randomly chosen digit\n",
    "classifier.predict([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model by doing a K-fold cross-validation\n",
    "\n",
    "Let's see how the model does in terms of accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(classifier, X_train, y_train_5, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy is not the only game in town. Let's look at the Confusion Matrix to get a little more insight.\n",
    "\n",
    "![Confusion Matrix](https://raw.githubusercontent.com/vineetbansal/MLLandscape/master/confusion_matrix.png \"Confusion Matrix\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_train_pred = cross_val_predict(classifier, X_train, y_train_5, cv=3)\n",
    "confusion_matrix(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "print(precision_score(y_train_5, y_train_pred))\n",
    "print(recall_score(y_train_5, y_train_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Thresholds\n",
    "\n",
    "The SGDClassifier computes \"scores\" for each prediction internally that we can look at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores = cross_val_predict(classifier, X_train, y_train_5, cv=3, method=\"decision_function\")\n",
    "y_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a response vector and a score vector for predictions, we can ask Scikit-Learn to compute precision/recall values for **all** possible thresholds on the predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision-Recall curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)\n",
    "plt.plot(recalls, precisions)\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve, roc_auc_score\n",
    "fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.show()\n",
    "\n",
    "print(roc_auc_score(y_train_5, y_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiclass Classification\n",
    "\n",
    "Scikit-Learn detects when you try to use a binary classification algo for a multi-class classification task, and it automatically runs a *one-versus-all* classifier - in this case, creating 10 binary classifiers, getting their decision scores, and selecting the class with the highest score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier.fit(X_train, y_train)\n",
    "classifier.predict([some_digit])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_val_score(classifier, X_train, y_train, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Scaling\n",
    "Let's apply feature scaling (subtract mean and divide by variance) on each of the training samples. *Why would it make a difference in the MNIST case?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train.astype('float'))\n",
    "cross_val_score(classifier, X_train_scaled, y_train, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = cross_val_predict(classifier, X_train_scaled, y_train, cv=3)\n",
    "conf_mx = confusion_matrix(y_train, y_train_pred)\n",
    "conf_mx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the confusion matrix to see where most of the misclassifications are happening."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sums = conf_mx.sum(axis=1, keepdims=True)\n",
    "norm_conf_mx = conf_mx / row_sums\n",
    "np.fill_diagonal(norm_conf_mx, 0)\n",
    "plt.matshow(norm_conf_mx, cmap=plt.cm.gray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "X_test_scaled = scaler.fit_transform(X_test.astype('float'))\n",
    "y_pred_test = classifier.predict(X_test_scaled)\n",
    "accuracy_score(y_test, y_pred_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}