pytorch-from-scratch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# PyTorch from scratch\n",
        "\n",
        "This uses a super-simple __linear regression__ model as the frame for walking from manually doing everything from good-old Python to using `numpy` to using `pytorch` on a GPU. It's based on this [excellent tutorial on pytorch](https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this interesting equation we will train a linear regression model against some synthetic data. \n",
        "\n",
        "$$\n",
        "y = a + b x + \\epsilon\n",
        "$$\n",
        "\n",
        "We will start from a plain Python program (using only the `math` library), progress to a `numpy` implementation before showing how to do the same thing using `pytorch`. The goal is to give you visibility into some of the things that different libraries abstract away for you while simultaneously ensuring that you understand what those abstractions are. We will also have some fun looking at how libraries like [sympy](https://www.sympy.org/en/index.html) can be used to help with some of the more tedious aspects of the math."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generating synthetic data to train against\n",
        "\n",
        "Next, let's generate some synthetic data for an equation where `a = 1` and `b = 2`. The synthetic data will sprinkle in some Gaussian noise into the synthetic values. Let's begin by importing `numpy` and initializing the random number generator with a fixed seed value so that our results are reproducible in the future:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "np.random.seed(12)\n",
        "\n"
      ],
      "outputs": [],
      "execution_count": 73,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:24.717Z",
          "iopub.execute_input": "2020-04-14T03:54:24.761Z",
          "iopub.status.idle": "2020-04-14T03:54:24.803Z",
          "shell.execute_reply": "2020-04-14T03:54:22.097Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generate a matrix of size [100, 1] of random numbers between 0 and 1 using the [random.rand() function](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.rand.html). Note that I'm using a naming convention in this notebook where variables prefixed with `d` represent __data__:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dx = np.random.rand(100, 1)\n",
        "dx[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 74,
          "data": {
            "text/plain": [
              "array([[0.15416284],\n",
              "       [0.7400497 ],\n",
              "       [0.26331502],\n",
              "       [0.53373939],\n",
              "       [0.01457496]])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 74,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.186Z",
          "iopub.status.busy": "2020-04-14T03:54:24.886Z",
          "iopub.execute_input": "2020-04-14T03:54:24.928Z",
          "iopub.status.idle": "2020-04-14T03:54:24.994Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's compute y values and sprinkle in the random noise. Marvel at the beautiful vector math and the compact notation."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dy = 1 + 2 * dx + .1 * np.random.randn(100, 1)\n",
        "dy[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 75,
          "data": {
            "text/plain": [
              "array([[1.44909846],\n",
              "       [2.31132976],\n",
              "       [1.67375343],\n",
              "       [2.23112508],\n",
              "       [0.98301043]])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 75,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.240Z",
          "iopub.status.busy": "2020-04-14T03:54:25.058Z",
          "iopub.execute_input": "2020-04-14T03:54:25.101Z",
          "iopub.status.idle": "2020-04-14T03:54:25.178Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generate a sequence from 0 to 99 that will represent indices into our matrix:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "idx = np.arange(100)\n",
        "idx"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 76,
          "data": {
            "text/plain": [
              "array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n",
              "       17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
              "       34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
              "       51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,\n",
              "       68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,\n",
              "       85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 76,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.286Z",
          "iopub.status.busy": "2020-04-14T03:54:25.246Z",
          "iopub.execute_input": "2020-04-14T03:54:25.286Z",
          "iopub.status.idle": "2020-04-14T03:54:25.352Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now shuffle so that the indices are in a random order:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.shuffle(idx)\n",
        "idx"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 77,
          "data": {
            "text/plain": [
              "array([53, 11, 36, 19, 60, 13, 29,  5,  1, 86, 26, 87,  4, 90, 84, 67, 22,\n",
              "       44, 76,  6, 54, 45, 37, 72, 62, 98, 33, 35, 27, 14, 96, 21, 43, 78,\n",
              "       61, 64, 80, 46, 40, 92, 99, 85, 81, 83, 49, 17, 58, 77, 28, 10, 39,\n",
              "       94, 16, 25, 89, 71, 65, 88,  0, 79,  7, 70, 95, 23, 12, 31, 30, 18,\n",
              "       52, 24,  8, 41,  3, 73, 56, 20, 55, 75, 97, 38, 50, 74,  2, 66, 47,\n",
              "       15, 42, 63, 57, 69, 91, 82, 34, 48, 59, 68, 51, 32, 93,  9])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 77,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.329Z",
          "iopub.status.busy": "2020-04-14T03:54:25.412Z",
          "iopub.execute_input": "2020-04-14T03:54:25.451Z",
          "iopub.status.idle": "2020-04-14T03:54:25.522Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Split the indices for the training and validation dataset such that first 80 random indices are used for training and the last 20 random indices are used for validation."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "train_idx = idx[:80]\n",
        "train_idx"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 78,
          "data": {
            "text/plain": [
              "array([53, 11, 36, 19, 60, 13, 29,  5,  1, 86, 26, 87,  4, 90, 84, 67, 22,\n",
              "       44, 76,  6, 54, 45, 37, 72, 62, 98, 33, 35, 27, 14, 96, 21, 43, 78,\n",
              "       61, 64, 80, 46, 40, 92, 99, 85, 81, 83, 49, 17, 58, 77, 28, 10, 39,\n",
              "       94, 16, 25, 89, 71, 65, 88,  0, 79,  7, 70, 95, 23, 12, 31, 30, 18,\n",
              "       52, 24,  8, 41,  3, 73, 56, 20, 55, 75, 97, 38])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 78,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.376Z",
          "iopub.status.busy": "2020-04-14T03:54:25.581Z",
          "iopub.execute_input": "2020-04-14T03:54:25.620Z",
          "iopub.status.idle": "2020-04-14T03:54:25.686Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "val_idx = idx[80:]\n",
        "val_idx"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 79,
          "data": {
            "text/plain": [
              "array([50, 74,  2, 66, 47, 15, 42, 63, 57, 69, 91, 82, 34, 48, 59, 68, 51,\n",
              "       32, 93,  9])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 79,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.421Z",
          "iopub.status.busy": "2020-04-14T03:54:25.751Z",
          "iopub.execute_input": "2020-04-14T03:54:25.793Z",
          "iopub.status.idle": "2020-04-14T03:54:25.859Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now generate the training and validation datasets. Notice how compact this is!"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dx_train, dy_train = dx[train_idx], dy[train_idx]\n",
        "dx_val, dy_val = dx[val_idx], dy[val_idx]"
      ],
      "outputs": [],
      "execution_count": 80,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.460Z",
          "iopub.status.busy": "2020-04-14T03:54:25.924Z",
          "iopub.execute_input": "2020-04-14T03:54:25.966Z",
          "iopub.status.idle": "2020-04-14T03:54:26.010Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now plot these two datasets side-by-side so we can see the data:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from bokeh.plotting import figure, show\n",
        "from bokeh.layouts import row\n",
        "from bokeh.io import output_notebook\n",
        "output_notebook()\n",
        "\n",
        "plot_train = figure(title=\"Training dataset\", x_axis_label='x', y_axis_label='y', plot_height=400, plot_width=400)\n",
        "plot_train.scatter(dx_train.flatten(), dy_train.flatten(), marker=\"square\", fill_color=\"blue\")\n",
        "\n",
        "plot_validate = figure(title=\"Training dataset\", x_axis_label='x', y_axis_label='y', plot_height=400, plot_width=400)\n",
        "plot_validate.scatter(dx_val.flatten(), dy_val.flatten(), marker=\"square\", fill_color=\"red\")\n",
        "\n",
        "show(row(plot_train, plot_validate))"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'bokeh'",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-81-0f87538e8acc>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mbokeh\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplotting\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mfigure\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mshow\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mbokeh\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayouts\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mrow\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mbokeh\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mio\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0moutput_notebook\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0moutput_notebook\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'bokeh'"
          ]
        }
      ],
      "execution_count": 81,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.499Z",
          "iopub.status.busy": "2020-04-14T03:54:26.075Z",
          "iopub.execute_input": "2020-04-14T03:54:26.137Z",
          "iopub.status.idle": "2020-04-14T03:54:26.223Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Gradient descent\n",
        "\n",
        "The loss function that we will use is the [mean square error]() loss function represented by the following equation:\n",
        "\n",
        "$$\n",
        "MSE=\\frac{1}{N}\\sum_{i=1}^N (y_i-\\hat y_i)^2\n",
        "$$\n",
        "\n",
        "Recall that $\\hat y_i$ is computed by the general linear regression equation:\n",
        "\n",
        "$$\n",
        "\\hat y_i = a - b x_i\n",
        "$$\n",
        "\n",
        "which yields:\n",
        "\n",
        "$$\n",
        "MSE=\\frac{1}{N}\\sum_{i=1}^N (y_i - a - b x_i)^2 \n",
        "$$"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next we need to compute the gradients, which are a partial derivative of each of the two variables that we are trying to learn: `a` and `b`. The following equations are computed through the application of the [chain rule](https://en.wikipedia.org/wiki/Chain_rule):\n",
        "\n",
        "<div id=\"partialderivatives\">\n",
        "$$\n",
        "\\begin{align}\n",
        "\\frac {\\partial MSE}{\\partial a} & = \n",
        "\\frac {\\partial MSE}{\\partial \\hat y_i} \\cdot\n",
        "\\frac {\\partial \\hat y_i}{\\partial a} \\\\\\\\ & =\n",
        "\\frac {1}{N} \n",
        "\\sum_ {i=1}^N 2(y_i - a - b x_i) \\cdot (-1) \\\\\\\\ & =\n",
        "-2 \\frac {1}{N} \\sum_ {i=1}^N (y_i - \\hat y_i) \\\\\\\\\n",
        "\\frac {\\partial MSE}{\\partial b} & = \n",
        "\\frac {\\partial MSE}{\\partial \\hat y_i} \\cdot\n",
        "\\frac {\\partial \\hat y_i}{\\partial b} \\\\\\\\ & =\n",
        "\\frac {1}{N} \n",
        "\\sum_ {i=1}^N 2(y_i - a - b x_i) \\cdot (-x_i) \\\\\\\\ & =\n",
        "-2 \\frac {1}{N} \\sum_ {i=1}^N x_i (y_i - \\hat y_i)\n",
        "\\end{align}\n",
        "$$\n",
        "</div>"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Using sympy to perform the same computation symbolically\n",
        "\n",
        "We can also use `sympy` to compute the equations that we will be using for our gradients."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's begin by importing the `sympy` library and declaring that `a`, `b` and `N` will be scalar symbols:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from sympy import Sum, IndexedBase, diff, Symbol\n",
        "from sympy.abc import x, y, i, a, b\n",
        "N = Symbol('N')"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'sympy'",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-82-ff223d176d7a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0msympy\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mSum\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mIndexedBase\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdiff\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mSymbol\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msympy\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mabc\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mb\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mN\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mSymbol\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'N'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sympy'"
          ]
        }
      ],
      "execution_count": 82,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.537Z",
          "iopub.status.busy": "2020-04-14T03:54:26.287Z",
          "iopub.execute_input": "2020-04-14T03:54:26.327Z",
          "iopub.status.idle": "2020-04-14T03:54:26.402Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will declare $x_i$ and $y_i$ as __indexed__ symbols:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "y = IndexedBase('y')\n",
        "x = IndexedBase('x')"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'IndexedBase' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-83-19e9c9d867ef>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mIndexedBase\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'y'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mIndexedBase\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'x'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mNameError\u001b[0m: name 'IndexedBase' is not defined"
          ]
        }
      ],
      "execution_count": 83,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.573Z",
          "iopub.status.busy": "2020-04-14T03:54:26.471Z",
          "iopub.execute_input": "2020-04-14T03:54:26.528Z",
          "iopub.status.idle": "2020-04-14T03:54:26.597Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we will define the __mean square error__ equation using a combination of Python arithmetic expressions and the sympy `Sum` method to define the summation expression:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "mse = (1/N) * Sum((y[i] - a - b * x[i]) ** 2, [i, 1, N])\n",
        "mse"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'N' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-84-30037c1ceca6>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m/\u001b[0m\u001b[0mN\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m*\u001b[0m \u001b[0mSum\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m-\u001b[0m \u001b[0ma\u001b[0m \u001b[1;33m-\u001b[0m \u001b[0mb\u001b[0m \u001b[1;33m*\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m**\u001b[0m \u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mN\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mmse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mNameError\u001b[0m: name 'N' is not defined"
          ]
        }
      ],
      "execution_count": 84,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.614Z",
          "iopub.status.busy": "2020-04-14T03:54:26.663Z",
          "iopub.execute_input": "2020-04-14T03:54:26.706Z",
          "iopub.status.idle": "2020-04-14T03:54:26.788Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we can compute the partial derivative of `mse` with respect to `a`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "diff(mse, a)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'diff' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-85-4c7c4ff19ce0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mdiff\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'diff' is not defined"
          ]
        }
      ],
      "execution_count": 85,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.656Z",
          "iopub.status.busy": "2020-04-14T03:54:26.857Z",
          "iopub.execute_input": "2020-04-14T03:54:26.898Z",
          "iopub.status.idle": "2020-04-14T03:54:26.963Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "and the partial derivative of `mse` with respect to `b`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "diff(mse, b)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'diff' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-86-089687db86ed>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mdiff\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'diff' is not defined"
          ]
        }
      ],
      "execution_count": 86,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.700Z",
          "iopub.status.busy": "2020-04-14T03:54:27.027Z",
          "iopub.execute_input": "2020-04-14T03:54:27.073Z",
          "iopub.status.idle": "2020-04-14T03:54:27.145Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is the same result as we got by manually applying the chain rule in the previous section. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Using learning rate to update parameters\n",
        "\n",
        "The $a$ and $b$ parameters need to be updated by $\\eta$ which is the __learning rate__. It is a multiplicative factor that we need to apply to the gradient to update the values of the parameters.\n",
        "\n",
        "Here are the equations that we will need to use to update the parameters:\n",
        "\n",
        "$$\n",
        "\\begin{align}\n",
        "a = a - \\eta \\frac{\\partial MSE}{\\partial a} \\\\\\\\\n",
        "b = b - \\eta \\frac{\\partial MSE}{\\partial b}\n",
        "\\end{align}\n",
        "$$"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training the model\n",
        "\n",
        "While training the model, we complete __epochs__. An [epoch](https://stackoverflow.com/questions/31155388/meaning-of-an-epoch-in-neural-networks-training) is complete whenever every point has already been used for computing the loss. \n",
        "\n",
        "__Batch__ gradient descent uses all points for computing the loss."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training the model using plain old Python\n",
        "\n",
        "TODO write a section that will train the model using nothing more than plain vanilla Python, i.e., no `numpy`."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "random.seed(42)\n",
        "pa = random.random()\n",
        "pb = random.random()\n",
        "print(f\"pa = {pa}, pb = {pb}\")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pa = 0.6394267984578837, pb = 0.025010755222666936\n"
          ]
        }
      ],
      "execution_count": 87,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.742Z",
          "iopub.status.busy": "2020-04-14T03:54:27.213Z",
          "iopub.execute_input": "2020-04-14T03:54:27.253Z",
          "iopub.status.idle": "2020-04-14T03:54:27.319Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lr = 1e-1"
      ],
      "outputs": [],
      "execution_count": 88,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.788Z",
          "iopub.status.busy": "2020-04-14T03:54:27.386Z",
          "iopub.execute_input": "2020-04-14T03:54:27.427Z",
          "iopub.status.idle": "2020-04-14T03:54:27.470Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dy_train[1][0]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 89,
          "data": {
            "text/plain": [
              "2.2379436272641637"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 89,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.827Z",
          "iopub.status.busy": "2020-04-14T03:54:27.542Z",
          "iopub.execute_input": "2020-04-14T03:54:27.581Z",
          "iopub.status.idle": "2020-04-14T03:54:27.644Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "yhat = [pa + pb * x_i[0] for x_i in dx_train]\n",
        "yhat[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 90,
          "data": {
            "text/plain": [
              "[0.6401247933339054,\n",
              " 0.654585396626457,\n",
              " 0.6586219440949098,\n",
              " 0.6434464458304892,\n",
              " 0.653743067684144]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 90,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.866Z",
          "iopub.status.busy": "2020-04-14T03:54:27.708Z",
          "iopub.execute_input": "2020-04-14T03:54:27.753Z",
          "iopub.status.idle": "2020-04-14T03:54:27.819Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "error = [y_i[0] - yhat_i for y_i, yhat_i in zip(dy_train, yhat)]\n",
        "error[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 91,
          "data": {
            "text/plain": [
              "[0.4508243381819097,\n",
              " 1.5833582306377068,\n",
              " 2.1804979777832822,\n",
              " 0.5520647088597275,\n",
              " 1.5033421262507254]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 91,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.904Z",
          "iopub.status.busy": "2020-04-14T03:54:27.881Z",
          "iopub.execute_input": "2020-04-14T03:54:27.921Z",
          "iopub.status.idle": "2020-04-14T03:54:27.988Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "a_grad = -2 * (sum(error) / len(error))\n",
        "a_grad"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 92,
          "data": {
            "text/plain": [
              "-2.7602118605457115"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 92,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.944Z",
          "iopub.status.busy": "2020-04-14T03:54:28.050Z",
          "iopub.execute_input": "2020-04-14T03:54:28.091Z",
          "iopub.status.idle": "2020-04-14T03:54:28.165Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "inter = [-2 * x_i[0] * error_i for x_i, error_i in zip(dx_train, error)]\n",
        "b_grad = sum(inter) / len(inter)\n",
        "b_grad"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 93,
          "data": {
            "text/plain": [
              "-1.8313019273748363"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 93,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:22.985Z",
          "iopub.status.busy": "2020-04-14T03:54:28.237Z",
          "iopub.execute_input": "2020-04-14T03:54:28.277Z",
          "iopub.status.idle": "2020-04-14T03:54:28.344Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pa = pa - lr * a_grad\n",
        "pa"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 94,
          "data": {
            "text/plain": [
              "0.9154479845124549"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 94,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.025Z",
          "iopub.status.busy": "2020-04-14T03:54:28.406Z",
          "iopub.execute_input": "2020-04-14T03:54:28.444Z",
          "iopub.status.idle": "2020-04-14T03:54:28.509Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pb = pb - lr * b_grad\n",
        "pb"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 95,
          "data": {
            "text/plain": [
              "0.2081409479601506"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 95,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.067Z",
          "iopub.status.busy": "2020-04-14T03:54:28.569Z",
          "iopub.execute_input": "2020-04-14T03:54:28.613Z",
          "iopub.status.idle": "2020-04-14T03:54:28.695Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pa = random.random()\n",
        "pb = random.random()\n",
        "lr = 1e-1\n",
        "n_epochs = 1000\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    yhat = [pa + pb * x_i[0] for x_i in dx_train]\n",
        "    error = [y_i[0] - yhat_i for y_i, yhat_i in zip(dy_train, yhat)]\n",
        "    a_grad = -2 * (sum(error) / len(error))\n",
        "    inter = [-2 * x_i[0] * error_i for x_i, error_i in zip(dx_train, error)]\n",
        "    b_grad = sum(inter) / len(inter)\n",
        "    pa = pa - lr * a_grad\n",
        "    pb = pb - lr * b_grad\n",
        "    \n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.9598033235275223 2.044986367514909\n"
          ]
        }
      ],
      "execution_count": 96,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.112Z",
          "iopub.status.busy": "2020-04-14T03:54:28.763Z",
          "iopub.execute_input": "2020-04-14T03:54:28.803Z",
          "iopub.status.idle": "2020-04-14T03:54:28.870Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training the model using numpy\n",
        "\n",
        "Next, let's train the model using the `numpy` library which makes your code much more succinct as it lets you define __vectorized__ operations easily."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's begin by randomly initializing our $a$ and $b$ parameters. Note that I'm using a naming convention in this notebook where I prefix __parameters__ with `p`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "np.random.seed(42)\n",
        "pa = np.random.randn(1)\n",
        "pb = np.random.randn(1)\n",
        "print(f\"pa = {pa}, pb = {pb}\")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pa = [0.49671415], pb = [-0.1382643]\n"
          ]
        }
      ],
      "execution_count": 97,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.155Z",
          "iopub.status.busy": "2020-04-14T03:54:28.934Z",
          "iopub.execute_input": "2020-04-14T03:54:28.973Z",
          "iopub.status.idle": "2020-04-14T03:54:29.038Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that type types of $a$ and $b$ are `np.ndarray`."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "type(pa)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 98,
          "data": {
            "text/plain": [
              "numpy.ndarray"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 98,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.195Z",
          "iopub.status.busy": "2020-04-14T03:54:29.102Z",
          "iopub.execute_input": "2020-04-14T03:54:29.143Z",
          "iopub.status.idle": "2020-04-14T03:54:29.217Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's define the learning rate. We won't say why we chose this value - choosing a learning rate turns out to be a very important training parameter to tune."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "lr = 1e-1"
      ],
      "outputs": [],
      "execution_count": 99,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.239Z",
          "iopub.status.busy": "2020-04-14T03:54:29.281Z",
          "iopub.execute_input": "2020-04-14T03:54:29.323Z",
          "iopub.status.idle": "2020-04-14T03:54:29.366Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that type type of `x_train` is `numpy.ndarray`. This is important as this will force all of the expressions below to be __vectorized__. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "type(dx_train)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 100,
          "data": {
            "text/plain": [
              "numpy.ndarray"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 100,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.283Z",
          "iopub.status.busy": "2020-04-14T03:54:29.427Z",
          "iopub.execute_input": "2020-04-14T03:54:29.467Z",
          "iopub.status.idle": "2020-04-14T03:54:29.539Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's compute `yhat` using the simple vectorized expression below. Note that `pb` is a scalar and `dx_train` is a vector, which results in `yhat` becoming a vector:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "yhat = pa + pb * dx_train\n",
        "yhat[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 101,
          "data": {
            "text/plain": [
              "array([[0.4928555 ],\n",
              "       [0.41291449],\n",
              "       [0.39059967],\n",
              "       [0.47449276],\n",
              "       [0.41757104]])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 101,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.324Z",
          "iopub.status.busy": "2020-04-14T03:54:29.601Z",
          "iopub.execute_input": "2020-04-14T03:54:29.643Z",
          "iopub.status.idle": "2020-04-14T03:54:29.708Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can compute `error` by computing the vector difference between `dy_train` and `yhat`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "error = dy_train - yhat\n",
        "error[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 102,
          "data": {
            "text/plain": [
              "array([[0.59809363],\n",
              "       [1.82502914],\n",
              "       [2.44852025],\n",
              "       [0.72101839],\n",
              "       [1.73951415]])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 102,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.364Z",
          "iopub.status.busy": "2020-04-14T03:54:29.772Z",
          "iopub.execute_input": "2020-04-14T03:54:29.810Z",
          "iopub.status.idle": "2020-04-14T03:54:29.876Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's compute the __gradients__ for a and b, based on the [partial derivatives](#partialderivatives) that we computed earlier. First we have $\\frac {\\partial MSE}{\\partial a}$:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "a_grad = -2 * error.mean()\n",
        "a_grad"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 103,
          "data": {
            "text/plain": [
              "-3.2169534586935553"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 103,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.404Z",
          "iopub.status.busy": "2020-04-14T03:54:29.939Z",
          "iopub.execute_input": "2020-04-14T03:54:29.980Z",
          "iopub.status.idle": "2020-04-14T03:54:30.043Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similarly, we can do the same thing for $\\frac {\\partial MSE}{\\partial b}$. But here, note that the \"inner loop\" in this calculation is succinctly represented by the vectorized operation `dx_train * error`. The result, of course, is a scalar because we are computing the `mean` of the vector generated previously:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "b_grad = -2 * (dx_train * error).mean()\n",
        "b_grad"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 104,
          "data": {
            "text/plain": [
              "-2.101896169929899"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 104,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.444Z",
          "iopub.status.busy": "2020-04-14T03:54:30.103Z",
          "iopub.execute_input": "2020-04-14T03:54:30.141Z",
          "iopub.status.idle": "2020-04-14T03:54:30.218Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we can compute the new values of the $a$ and $b$ __parameters__ by multiplying the gradients by the __learning rate__:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "pa = pa - lr * a_grad\n",
        "pa"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 105,
          "data": {
            "text/plain": [
              "array([0.8184095])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 105,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.488Z",
          "iopub.status.busy": "2020-04-14T03:54:30.279Z",
          "iopub.execute_input": "2020-04-14T03:54:30.317Z",
          "iopub.status.idle": "2020-04-14T03:54:30.383Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pb = pb - lr * b_grad\n",
        "pb"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 106,
          "data": {
            "text/plain": [
              "array([0.07192532])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 106,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.533Z",
          "iopub.status.busy": "2020-04-14T03:54:30.445Z",
          "iopub.execute_input": "2020-04-14T03:54:30.493Z",
          "iopub.status.idle": "2020-04-14T03:54:30.556Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Putting all of this together, we can write the loop that iterates over __1000 epochs__, taking care to initialize all of our parameters at the start:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "pa = np.random.randn(1)\n",
        "pb = np.random.randn(1)\n",
        "lr = 1e-1\n",
        "n_epochs = 1000\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    yhat = pa + pb * dx_train\n",
        "    error = (dy_train - yhat)\n",
        "    a_grad = -2 * error.mean()\n",
        "    b_grad = -2 * (dx_train * error).mean()\n",
        "    pa = pa - lr * a_grad\n",
        "    pb = pb - lr * b_grad\n",
        "    \n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0.95980314] [2.0449867]\n"
          ]
        }
      ],
      "execution_count": 107,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.571Z",
          "iopub.status.busy": "2020-04-14T03:54:30.618Z",
          "iopub.execute_input": "2020-04-14T03:54:30.665Z",
          "iopub.status.idle": "2020-04-14T03:54:30.736Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To make sure we didn't make a mistake, let's check our result against what scikit-learn's [linear regression model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) computes:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.linear_model import LinearRegression\n",
        "linr = LinearRegression()\n",
        "linr.fit(dx_train, dy_train)\n",
        "print(linr.intercept_, linr.coef_)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'sklearn'",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-108-3908a64b7e36>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlinear_model\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mLinearRegression\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mlinr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mLinearRegression\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mlinr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdx_train\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdy_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlinr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mintercept_\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlinr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcoef_\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'"
          ]
        }
      ],
      "execution_count": 108,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.611Z",
          "iopub.status.busy": "2020-04-14T03:54:30.803Z",
          "iopub.execute_input": "2020-04-14T03:54:30.844Z",
          "iopub.status.idle": "2020-04-14T03:54:30.908Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Pytorch"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, let's determine whether a GPU, i.e., [CUDA](https://en.wikipedia.org/wiki/CUDA) is available. If not, perform our computations on the CPU instead."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "print(torch.__version__)\n",
        "\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.1.0\n"
          ]
        }
      ],
      "execution_count": 109,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.660Z",
          "iopub.status.busy": "2020-04-14T03:54:30.973Z",
          "iopub.execute_input": "2020-04-14T03:54:31.017Z",
          "iopub.status.idle": "2020-04-14T03:54:31.124Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we need to import our `numpy` arrays into `pytorch` tensors. It's worth noting that pytorch refers to all non-scalar data as `tensors`, so a vector is a [tensor](https://en.wikipedia.org/wiki/Tensor), as is a matrix. Let's use the [torch.fromnumpy()](https://pytorch.org/docs/stable/torch.html) method to import our data from the numpy `dx_train`, `dy_train` vectors (really matrices). \n",
        "\n",
        "It's worth noting that this method appears to round to 5 significant digits for some reason as well, regardless of whether you explicitly convert via `float()` or not."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dx_train_tensor = torch.from_numpy(dx_train).float().to(device)\n",
        "dy_train_tensor = torch.from_numpy(dy_train).float().to(device)\n",
        "dx_train_tensor[0]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 110,
          "data": {
            "text/plain": [
              "tensor([0.0279], device='cuda:0')"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 110,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.701Z",
          "iopub.status.busy": "2020-04-14T03:54:31.221Z",
          "iopub.execute_input": "2020-04-14T03:54:31.275Z",
          "iopub.status.idle": "2020-04-14T03:54:31.359Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can use the `type()` API to get additional information about where the tensor is located (i.e., what device it resides on). I believe that the absence of a device in the output implies that it is located on the CPU, e.g., a tensor that resides on your GPU will return its type as `torch.cuda.FloatTensor`."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dx_train_tensor.type()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 111,
          "data": {
            "text/plain": [
              "'torch.cuda.FloatTensor'"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 111,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.745Z",
          "iopub.status.busy": "2020-04-14T03:54:31.437Z",
          "iopub.execute_input": "2020-04-14T03:54:31.487Z",
          "iopub.status.idle": "2020-04-14T03:54:31.569Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's initialize our $a$ and $b$ parameters. As before, we need to initialize a seed for torch's random number generator so we our results are reproducible. Then we need to create our parameters and assign them to the device that we picked earlier. A key thing that we're doing is setting a flag, `requires_grad=True` which tells pytorch to automatically compute gradients for us vs. the manual computation that we did before."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)\n",
        "pa = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "pb = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.6226], device='cuda:0', requires_grad=True) tensor([1.4505], device='cuda:0', requires_grad=True)\n"
          ]
        }
      ],
      "execution_count": 112,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.784Z",
          "iopub.status.busy": "2020-04-14T03:54:31.650Z",
          "iopub.execute_input": "2020-04-14T03:54:31.704Z",
          "iopub.status.idle": "2020-04-14T03:54:31.795Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that these parameters are also tensors, much like in the `numpy` example earlier where `pa` and `pb` were of type `numpy.ndarray`. This is required to take advantage of the autograd functionality in pytorch."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "How are the gradients computed? Pytorch contains an [autograd](https://pytorch.org/docs/stable/autograd.html) package that is automatically invoked when you enable `requires_grad=True` on your tensors. So rather than have to define (and compute) the partial derivatives $\\frac {\\partial MSE}{\\partial a}$ and $\\frac {\\partial MSE}{\\partial b}$, you just have to define the __loss function, MSE__ instead.\n",
        "\n",
        "Let's walk through the code that we wrote earlier for `numpy` line by line:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "yhat = pa + pb * dx_train_tensor\n",
        "yhat[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 113,
          "data": {
            "text/plain": [
              "tensor([[0.6631],\n",
              "        [1.5017],\n",
              "        [1.7358],\n",
              "        [0.8557],\n",
              "        [1.4529]], device='cuda:0', grad_fn=<SliceBackward>)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 113,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.828Z",
          "iopub.status.busy": "2020-04-14T03:54:31.875Z",
          "iopub.execute_input": "2020-04-14T03:54:31.949Z",
          "iopub.status.idle": "2020-04-14T03:54:32.057Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "error = dy_train_tensor - yhat\n",
        "error[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 114,
          "data": {
            "text/plain": [
              "tensor([[0.4279],\n",
              "        [0.7362],\n",
              "        [1.1033],\n",
              "        [0.3398],\n",
              "        [0.7042]], device='cuda:0', grad_fn=<SliceBackward>)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 114,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.910Z",
          "iopub.status.busy": "2020-04-14T03:54:32.135Z",
          "iopub.execute_input": "2020-04-14T03:54:32.187Z",
          "iopub.status.idle": "2020-04-14T03:54:32.272Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we compute the loss function, MSE, as opposed to manually computing the gradient functions."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "loss = (error ** 2).mean()\n",
        "loss"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 115,
          "data": {
            "text/plain": [
              "tensor(0.4668, device='cuda:0', grad_fn=<MeanBackward0>)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 115,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:23.970Z",
          "iopub.status.busy": "2020-04-14T03:54:32.354Z",
          "iopub.execute_input": "2020-04-14T03:54:32.402Z",
          "iopub.status.idle": "2020-04-14T03:54:32.482Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "But we need to explicitly tell pytorch to compute the gradient by calling `backward` on the loss function."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "loss.backward()"
      ],
      "outputs": [],
      "execution_count": 116,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.033Z",
          "iopub.status.busy": "2020-04-14T03:54:32.551Z",
          "iopub.execute_input": "2020-04-14T03:54:32.598Z",
          "iopub.status.idle": "2020-04-14T03:54:32.648Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "What are the computed gradients? The autograd function knows how to extract that presumably from the computational graph that was generated in the equations that we defined earlier. This is cool!"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "print(pa.grad, pb.grad)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([-1.2982], device='cuda:0') tensor([-0.7939], device='cuda:0')\n"
          ]
        }
      ],
      "execution_count": 117,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.092Z",
          "iopub.status.busy": "2020-04-14T03:54:32.732Z",
          "iopub.execute_input": "2020-04-14T03:54:32.785Z",
          "iopub.status.idle": "2020-04-14T03:54:32.859Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, let's update our parameters $a$ and $b$ by multiplying our automatically computed gradients by our fixed learning rate."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "with torch.no_grad():\n",
        "    pa -= lr * pa.grad\n",
        "    pb -= lr * pb.grad\n",
        "    \n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.7524], device='cuda:0', requires_grad=True) tensor([1.5298], device='cuda:0', requires_grad=True)\n"
          ]
        }
      ],
      "execution_count": 118,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.157Z",
          "iopub.status.busy": "2020-04-14T03:54:32.946Z",
          "iopub.execute_input": "2020-04-14T03:54:33.000Z",
          "iopub.status.idle": "2020-04-14T03:54:33.083Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "By default, gradients are __accumulated__, so we need to explicitly zero out the gradients on each iteration.\n",
        "\n",
        "Note that any pytorch methods that end in `_` are __in-place__ operations. Make sure you use this variant over the ones that do not end in `_`, as those methods will return a __new object__ which is not what we want here."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "pa.grad.zero_()\n",
        "pb.grad.zero_()\n",
        "\n",
        "print(pa.grad, pb.grad)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.], device='cuda:0') tensor([0.], device='cuda:0')\n"
          ]
        }
      ],
      "execution_count": 119,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.219Z",
          "iopub.status.busy": "2020-04-14T03:54:33.160Z",
          "iopub.execute_input": "2020-04-14T03:54:33.223Z",
          "iopub.status.idle": "2020-04-14T03:54:33.314Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's take everything we've learned here and write a loop that will do the same training using pytorch over 1000 epochs:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)\n",
        "pa = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "pb = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "\n",
        "lr = 1e-1\n",
        "n_epochs = 1000\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    yhat = pa + pb * dx_train_tensor\n",
        "    error = dy_train_tensor - yhat\n",
        "    loss = (error ** 2).mean()\n",
        "    loss.backward()\n",
        "    with torch.no_grad():\n",
        "        pa -= lr * pa.grad\n",
        "        pb -= lr * pb.grad\n",
        "    pa.grad.zero_()\n",
        "    pb.grad.zero_()\n",
        "    \n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.9598], device='cuda:0', requires_grad=True) tensor([2.0450], device='cuda:0', requires_grad=True)\n"
          ]
        }
      ],
      "execution_count": 120,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.280Z",
          "iopub.status.busy": "2020-04-14T03:54:33.383Z",
          "iopub.execute_input": "2020-04-14T03:54:33.445Z",
          "iopub.status.idle": "2020-04-14T03:54:33.522Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's examine the pytorch dyanmic computation graph using the [pytorchviz](https://github.com/szagoruyko/pytorchviz) package and its `make_dot()` method."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can invoke `make_dot` method across arbitrary expressions that define pytorch's dynamic compute graph. For the yhat expression, recall that the equation is:\n",
        "\n",
        "$$\n",
        "\\hat y_i = a + b x_i\n",
        "$$"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torchviz import make_dot\n",
        "make_dot(yhat)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'torchviz'",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-121-1bf41d4934cd>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mtorchviz\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmake_dot\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mmake_dot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0myhat\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'torchviz'"
          ]
        }
      ],
      "execution_count": 121,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.336Z",
          "iopub.status.busy": "2020-04-14T03:54:33.597Z",
          "iopub.execute_input": "2020-04-14T03:54:33.647Z",
          "iopub.status.idle": "2020-04-14T03:54:33.743Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The graph depicts the flow of the gradient computations. It is intended to be read from the bottom up.\n",
        "\n",
        "The green box represents the starting point for the computation of gradients. The blue boxes are the parameters that we are training __and have autograd enabled__. We are missing the labels for the boxes (for now - we will see how to do this later when we define an actual model), but the top blue box represents $b$, and the blue box to the left represents $a$. \n",
        "\n",
        "The green box adds $a$ to the result of multiplying $b$ against the training tensor `dx_train_tensor`."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "make_dot(error)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'make_dot' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-122-4c942495153e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmake_dot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merror\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'make_dot' is not defined"
          ]
        }
      ],
      "execution_count": 122,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.372Z",
          "iopub.status.busy": "2020-04-14T03:54:33.820Z",
          "iopub.execute_input": "2020-04-14T03:54:33.870Z",
          "iopub.status.idle": "2020-04-14T03:54:33.958Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "make_dot(loss)"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'make_dot' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-123-b845bae12187>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmake_dot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'make_dot' is not defined"
          ]
        }
      ],
      "execution_count": 123,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.414Z",
          "iopub.status.busy": "2020-04-14T03:54:34.039Z",
          "iopub.execute_input": "2020-04-14T03:54:34.090Z",
          "iopub.status.idle": "2020-04-14T03:54:34.165Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are currently manually updating our training parameters $a$ and $b$, much like what we did for the `numpy` implementation. Instead of doing it by hand, we can use a pytorch __optimizer__. Since we are performing a batch gradient descent we can use pytorch's [Stochastic Gradient Descent optimizer](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) in place of our code.\n",
        "\n",
        "We need to initialize the optimizer with the names of our training parameters (which will be inserted into the pytorch dynamic computation graph) as well as our choice of the __learning rate__. Let's rewrite our previous program to explicitly use the SGD optimizer:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.optim import SGD\n",
        "\n",
        "torch.manual_seed(42)\n",
        "pa = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "pb = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
        "\n",
        "lr = 1e-1\n",
        "n_epochs = 1000\n",
        "\n",
        "optimizer = SGD([pa, pb], lr=lr)\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    yhat = pa + pb * dx_train_tensor\n",
        "    error = dy_train_tensor - yhat\n",
        "    loss = (error ** 2).mean()\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    optimizer.zero_grad()\n",
        "    \n",
        "print(pa, pb)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.9598], device='cuda:0', requires_grad=True) tensor([2.0450], device='cuda:0', requires_grad=True)\n"
          ]
        }
      ],
      "execution_count": 124,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.453Z",
          "iopub.status.busy": "2020-04-14T03:54:34.251Z",
          "iopub.execute_input": "2020-04-14T03:54:34.294Z",
          "iopub.status.idle": "2020-04-14T03:54:34.357Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that in place of the manual calculation of updated values for `pa` and `pb`, we call `optimizer.step()` instead, which does that operation for us. Zeroing of the gradients is still an explicit step, but we no longer need to specify `pa` and `pb`; the optimizer already knows that those are the parameters that it is optimizing for as they were initialized during construction of the `optimizer` object."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## pytorch models\n",
        "\n",
        "Pytorch lets you organize your code into a Python class vs. defining everything at global scope. In return for this ceremony, it provides these benefits:\n",
        "\n",
        "- a dictionary of all of the parameters used by the model (we'll see how this is useful in our graphs)\n",
        "- "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import Module, Parameter\n",
        "\n",
        "class ManualLinearRegression(Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.pa = Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))\n",
        "        self.pb = Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))\n",
        "        \n",
        "    def forward(self, dx):\n",
        "        return self.pa + self.pb * dx"
      ],
      "outputs": [],
      "execution_count": 125,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.492Z",
          "iopub.status.busy": "2020-04-14T03:54:34.416Z",
          "iopub.execute_input": "2020-04-14T03:54:34.459Z",
          "iopub.status.idle": "2020-04-14T03:54:34.507Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can use the `model` as well as a set of helper functions like the `MSELoss` function to avoid implementing your own loss function."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import MSELoss\n",
        "\n",
        "model = ManualLinearRegression().to(device)\n",
        "\n",
        "lr = 1e-1\n",
        "n_epochs = 1000\n",
        "\n",
        "loss_fn = MSELoss(reduction='mean')\n",
        "optimizer = SGD(model.parameters(), lr=lr)\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    model.train()\n",
        "    yhat = model(dx_train_tensor)\n",
        "    loss = loss_fn(dy_train_tensor, yhat)\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    optimizer.zero_grad()\n",
        "    \n",
        "print(model.state_dict()['pa'], model.state_dict()['pb'])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0.9598], device='cuda:0') tensor([2.0450], device='cuda:0')\n"
          ]
        }
      ],
      "execution_count": 126,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.564Z",
          "iopub.status.busy": "2020-04-14T03:54:34.575Z",
          "iopub.execute_input": "2020-04-14T03:54:34.617Z",
          "iopub.status.idle": "2020-04-14T03:54:34.683Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For the sake of the next few examples that follow, let's refactor all of the code that __uses the model__ into a function that we can reuse in our subsequent examples:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import MSELoss\n",
        "from torch.optim import SGD\n",
        "\n",
        "def train_linear_model(model, dx, dy):\n",
        "    lr = 1e-1\n",
        "    EPOCHS = 1000\n",
        "    loss_fn = MSELoss(reduction='mean')\n",
        "    optimizer = SGD(model.parameters(), lr=lr)\n",
        "    \n",
        "    for epoch in range(EPOCHS):\n",
        "        model.train()\n",
        "        yhat = model(dx)\n",
        "        loss = loss_fn(dy, yhat)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        optimizer.zero_grad()\n",
        "    \n",
        "    print(model.state_dict())"
      ],
      "outputs": [],
      "execution_count": 127,
      "metadata": {
        "execution": {
          "shell.execute_reply": "2020-04-14T03:54:24.602Z",
          "iopub.status.busy": "2020-04-14T03:54:34.755Z",
          "iopub.execute_input": "2020-04-14T03:54:34.797Z",
          "iopub.status.idle": "2020-04-14T03:54:34.844Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's use `train_model` with our `ManualLinearRegression` object:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "model = ManualLinearRegression().to(device)\n",
        "train_linear_model(model, dx_train_tensor, dy_train_tensor)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('pa', tensor([0.9598], device='cuda:0')), ('pb', tensor([2.0450], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 128,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:34.908Z",
          "iopub.execute_input": "2020-04-14T03:54:34.949Z",
          "iopub.status.idle": "2020-04-14T03:54:35.046Z",
          "shell.execute_reply": "2020-04-14T03:54:37.891Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "make_dot(model.forward(dx_train_tensor))"
      ],
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'make_dot' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-129-3bceb85da34e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmake_dot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdx_train_tensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'make_dot' is not defined"
          ]
        }
      ],
      "execution_count": 129,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:35.112Z",
          "iopub.execute_input": "2020-04-14T03:54:35.157Z",
          "iopub.status.idle": "2020-04-14T03:54:35.260Z",
          "shell.execute_reply": "2020-04-14T03:54:37.920Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Of course, you don't even need to create a linear model, pytorch has a [Linear Model](https://pytorch.org/docs/stable/nn.html#torch.nn.Linear) already.\n",
        "\n",
        "Let's use it by constructing a new class `LayeredLinearRegression` that simply delegates to the underlying `Linear` object:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import Linear\n",
        "\n",
        "class LayeredLinearRegression(Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.linear = Linear(1,1)\n",
        "    \n",
        "    def forward(self, dx):\n",
        "        return self.linear(dx)"
      ],
      "outputs": [],
      "execution_count": 130,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:35.329Z",
          "iopub.execute_input": "2020-04-14T03:54:35.377Z",
          "iopub.status.idle": "2020-04-14T03:54:35.448Z",
          "shell.execute_reply": "2020-04-14T03:54:37.946Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can see that it automatically constructs parameters for the intercept and slope (weight and bias) with `requires_grad=True`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "model = LayeredLinearRegression()\n",
        "[*model.parameters()]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 131,
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[0.8815]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([-0.7336], requires_grad=True)]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 131,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:35.517Z",
          "iopub.execute_input": "2020-04-14T03:54:35.558Z",
          "iopub.status.idle": "2020-04-14T03:54:35.655Z",
          "shell.execute_reply": "2020-04-14T03:54:37.977Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's replace our code from before that used `ManualLinearRegression` with our new `LayeredLinearRegression` type:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "model = LayeredLinearRegression().to(device)\n",
        "train_linear_model(model, dx_train_tensor, dy_train_tensor)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('linear.weight', tensor([[2.0450]], device='cuda:0')), ('linear.bias', tensor([0.9598], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 132,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:35.719Z",
          "iopub.execute_input": "2020-04-14T03:54:35.765Z",
          "iopub.status.idle": "2020-04-14T03:54:35.859Z",
          "shell.execute_reply": "2020-04-14T03:54:38.016Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can see that the weight ($b$) and bias ($a$) parameters are the same as we got from our `ManualLinearRegression` type."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "But why define a separate `LayeredLinearRegression` class to simply forward computations to a `Linear` class? In neural networks, there are models that sequentially feed the output of one layer as the input to the next. pytorch provides a model type called [Sequential](https://pytorch.org/docs/stable/nn.html#torch.nn.Sequential) model that does exactly this. \n",
        "\n",
        "Here is some code that constructs a new `Sequential` model and uses it to make the same set of predictions:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import Sequential\n",
        "\n",
        "model = Sequential(Linear(1, 1)).to(device)\n",
        "train_linear_model(model, dx_train_tensor, dy_train_tensor)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('0.weight', tensor([[2.0450]], device='cuda:0')), ('0.bias', tensor([0.9598], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 133,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:35.931Z",
          "iopub.execute_input": "2020-04-14T03:54:35.972Z",
          "iopub.status.idle": "2020-04-14T03:54:36.066Z",
          "shell.execute_reply": "2020-04-14T03:54:38.042Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that the names of the parameters are a more generic `0.weight` and `0.bias` representing the underlying layer and the parameter name."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Taking a step back to generalize\n",
        "\n",
        "There is a set of operations that need to be completed during each training step. We can write a function `train_step` that is a further generalization of the `train_model` function that abstracts away the training loop in a way that lets us parameterize the __model__, the __loss function__, and the __optimizer__. Let's see what we can do here with the function `make_train_step_function` which accepts a model, a loss function and an optimizer as the parameters.\n",
        "\n",
        "What is unclear to me right now is whether the `loss` object returned by the `loss_fn` is something that is generalizable. But I get the impression that this is a key abstraction within the pytorch framework to represent this concept."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "def make_train_step_function(model, loss_fn, optimizer):\n",
        "    def train_step(dx, dy):\n",
        "        model.train()\n",
        "        yhat = model(dx)\n",
        "        loss = loss_fn(dy, yhat)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        optimizer.zero_grad()\n",
        "        return loss.item()\n",
        "    return train_step"
      ],
      "outputs": [],
      "execution_count": 134,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:36.141Z",
          "iopub.execute_input": "2020-04-14T03:54:36.185Z",
          "iopub.status.idle": "2020-04-14T03:54:36.255Z",
          "shell.execute_reply": "2020-04-14T03:54:38.070Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's rewrite our `train_linear_model` function to take advantage of the `make_train_step_function` abstraction:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "def train_linear_model(model, dx, dy):\n",
        "    lr = 1e-1\n",
        "    EPOCHS = 1000\n",
        "    loss_fn = MSELoss(reduction='mean')\n",
        "    optimizer = SGD(model.parameters(), lr=lr)\n",
        "    train_step = make_train_step_function(model, loss_fn, optimizer)\n",
        "    losses = []\n",
        "    \n",
        "    for epoch in range(EPOCHS):\n",
        "        loss = train_step(dx, dy)\n",
        "        losses.append(loss)\n",
        "    \n",
        "    print(model.state_dict())"
      ],
      "outputs": [],
      "execution_count": 135,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:36.326Z",
          "iopub.execute_input": "2020-04-14T03:54:36.372Z",
          "iopub.status.idle": "2020-04-14T03:54:36.445Z",
          "shell.execute_reply": "2020-04-14T03:54:38.094Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential(Linear(1, 1)).to(device)\n",
        "train_linear_model(model, dx_train_tensor, dy_train_tensor)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('0.weight', tensor([[2.0450]], device='cuda:0')), ('0.bias', tensor([0.9598], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 136,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:36.509Z",
          "iopub.execute_input": "2020-04-14T03:54:36.554Z",
          "iopub.status.idle": "2020-04-14T03:54:36.640Z",
          "shell.execute_reply": "2020-04-14T03:54:38.119Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Managing the data for our models\n",
        "\n",
        "Pytorch provides a number of helper classes and methods for dealing with data. Data isn't as simple as just a couple of arrays as we initially created at the start of this tutorial. Rather, there are operations like splitting data into training and validation datasets and splitting each one of those into a set of mini-batches that can be downloaded into your GPU's RAM (because the entire dataset won't fit at once for anything significant).\n",
        "\n",
        "A [Dataset](https://pytorch.org/docs/stable/data.html) is a base type that you can use for definining your own custom dataset. Below is a simple implementation of a `CustomDataset` that stores all data in memory:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import Dataset\n",
        "\n",
        "class CustomDataset(Dataset):\n",
        "    def __init__(self, x_tensor, y_tensor):\n",
        "        self.dx = x_tensor\n",
        "        self.dy = y_tensor\n",
        "        \n",
        "    def __getitem__(self, index):\n",
        "        return (self.dx[index], self.dy[index])\n",
        "    \n",
        "    def __len__(self):\n",
        "        return len(self.dx)"
      ],
      "outputs": [],
      "execution_count": 137,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:36.712Z",
          "iopub.execute_input": "2020-04-14T03:54:36.782Z",
          "iopub.status.idle": "2020-04-14T03:54:36.860Z",
          "shell.execute_reply": "2020-04-14T03:54:38.146Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's use `CustomDataset` to load our training tensors and display some data:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "train_data = CustomDataset(dx_train_tensor, dy_train_tensor)\n",
        "train_data[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 138,
          "data": {
            "text/plain": [
              "(tensor([[0.0279],\n",
              "         [0.6061],\n",
              "         [0.7675],\n",
              "         [0.1607],\n",
              "         [0.5724]], device='cuda:0'),\n",
              " tensor([[1.0909],\n",
              "         [2.2379],\n",
              "         [2.8391],\n",
              "         [1.1955],\n",
              "         [2.1571]], device='cuda:0'))"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 138,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:36.932Z",
          "iopub.execute_input": "2020-04-14T03:54:36.982Z",
          "iopub.status.idle": "2020-04-14T03:54:37.077Z",
          "shell.execute_reply": "2020-04-14T03:54:38.172Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can get the same functionality if you use the built-in `TensorDataset` class:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import TensorDataset\n",
        "\n",
        "train_data = TensorDataset(dx_train_tensor, dy_train_tensor)\n",
        "train_data[:5]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 139,
          "data": {
            "text/plain": [
              "(tensor([[0.0279],\n",
              "         [0.6061],\n",
              "         [0.7675],\n",
              "         [0.1607],\n",
              "         [0.5724]], device='cuda:0'),\n",
              " tensor([[1.0909],\n",
              "         [2.2379],\n",
              "         [2.8391],\n",
              "         [1.1955],\n",
              "         [2.1571]], device='cuda:0'))"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 139,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:37.142Z",
          "iopub.execute_input": "2020-04-14T03:54:37.187Z",
          "iopub.status.idle": "2020-04-14T03:54:37.284Z",
          "shell.execute_reply": "2020-04-14T03:54:38.195Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The real value provided by representing your data as a `Dataset` comes when you combine it with a `DataLoader`. This type lets you easily implement __mini-batch gradient descent__ where only a subset of the dataset can fit in your GPU RAM at a time.\n",
        "\n",
        "The `train_loader` object created below behaves like a Python iterator, so we can use the `next` function to retrieve the next mini-batch containing 16 values:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "train_loader = DataLoader(dataset=train_data, batch_size=16, shuffle=True)\n",
        "next(iter(train_loader))"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 140,
          "data": {
            "text/plain": [
              "[tensor([[0.9187],\n",
              "         [0.6362],\n",
              "         [0.9503],\n",
              "         [0.7026],\n",
              "         [0.9446],\n",
              "         [0.1352],\n",
              "         [0.3195],\n",
              "         [0.6194],\n",
              "         [0.4631],\n",
              "         [0.9466],\n",
              "         [0.5952],\n",
              "         [0.2915],\n",
              "         [0.9626],\n",
              "         [0.4193],\n",
              "         [0.6889],\n",
              "         [0.0023]], device='cuda:0'),\n",
              " tensor([[2.8174],\n",
              "         [2.2060],\n",
              "         [2.8224],\n",
              "         [2.4068],\n",
              "         [3.0300],\n",
              "         [1.2977],\n",
              "         [1.6704],\n",
              "         [2.1268],\n",
              "         [1.7979],\n",
              "         [2.7793],\n",
              "         [2.0967],\n",
              "         [1.3341],\n",
              "         [2.8611],\n",
              "         [1.7007],\n",
              "         [2.4431],\n",
              "         [0.7953]], device='cuda:0')]"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 140,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:37.356Z",
          "iopub.execute_input": "2020-04-14T03:54:37.403Z",
          "iopub.status.idle": "2020-04-14T03:54:37.492Z",
          "shell.execute_reply": "2020-04-14T03:54:38.218Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally we can implement mini-batch stochastic gradient descent."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "losses = []\n",
        "train_step = make_train_step_function(model, loss_fn, optimizer)\n",
        "\n",
        "for epoch in range(1000):\n",
        "    for dx_batch, dy_batch in train_loader:\n",
        "        # In the case of GPU machines, we need to move the mini-batch to\n",
        "        # the device RAM first before we can train\n",
        "        dx_batch = dx_batch.to(device)\n",
        "        dy_batch = dy_batch.to(device)\n",
        "        \n",
        "        loss = train_step(dx_batch, dy_batch)\n",
        "        losses.append(loss)\n",
        "        \n",
        "print(model.state_dict())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('0.weight', tensor([[2.0450]], device='cuda:0')), ('0.bias', tensor([0.9598], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 141,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:37.564Z",
          "iopub.execute_input": "2020-04-14T03:54:37.608Z",
          "iopub.status.idle": "2020-04-14T03:54:44.928Z",
          "shell.execute_reply": "2020-04-14T03:54:45.152Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Validating our model\n",
        "\n",
        "Up until now, we've been looking at only training our model, not validating the results of the model against the validation dataset (perhaps I should go back and do this manually as well both from scratch and using numpy)."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data.dataset import random_split\n",
        "\n",
        "dx_tensor = torch.from_numpy(dx).float()\n",
        "dy_tensor = torch.from_numpy(dy).float()\n",
        "\n",
        "dataset = TensorDataset(dx_tensor, dy_tensor)\n",
        "\n",
        "train_dataset, val_dataset = random_split(dataset, [80, 20])\n",
        "\n",
        "train_loader = DataLoader(dataset=train_dataset, batch_size=16)\n",
        "val_loader = DataLoader(dataset=val_dataset, batch_size=20)"
      ],
      "outputs": [],
      "execution_count": 142,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:44.962Z",
          "iopub.execute_input": "2020-04-14T03:54:44.984Z",
          "iopub.status.idle": "2020-04-14T03:54:45.035Z",
          "shell.execute_reply": "2020-04-14T03:54:45.171Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "With our data prepared, next let's write both a training and a validation loop for our mini-batches:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "train_losses = []\n",
        "val_losses = []\n",
        "train_step = make_train_step_function(model, loss_fn, optimizer)\n",
        "\n",
        "for epoch in range(1000):\n",
        "    for dx_batch, dy_batch in train_loader:\n",
        "        dx_batch = dx_batch.to(device)\n",
        "        dy_batch = dy_batch.to(device)\n",
        "        loss = train_step(dx_batch, dy_batch)\n",
        "        train_losses.append(loss)\n",
        "        \n",
        "    with torch.no_grad():\n",
        "        for dx_val, dy_val in val_loader:\n",
        "            dx_val = dx_val.to(device)\n",
        "            dy_val = dy_val.to(device)\n",
        "            model.eval()\n",
        "            yhat = model(dx_val)\n",
        "            val_loss = loss_fn(dy_val, yhat)\n",
        "            val_losses.append(val_loss.item())\n",
        "            \n",
        "print(model.state_dict())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "OrderedDict([('0.weight', tensor([[2.0450]], device='cuda:0')), ('0.bias', tensor([0.9598], device='cuda:0'))])\n"
          ]
        }
      ],
      "execution_count": 143,
      "metadata": {
        "execution": {
          "iopub.status.busy": "2020-04-14T03:54:45.078Z",
          "iopub.execute_input": "2020-04-14T03:54:45.106Z",
          "iopub.status.idle": "2020-04-14T03:54:56.384Z",
          "shell.execute_reply": "2020-04-14T03:54:56.417Z"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "__BUT__ I don't know how to visualize how well the validation losses?"
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.8",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "nteract": {
      "version": "0.22.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}