lab_linreg.ipynb

{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3", 
      "name": "python3", 
      "language": "python"
    }, 
    "language_info": {
      "mimetype": "text/x-python", 
      "nbconvert_exporter": "python", 
      "name": "python", 
      "file_extension": ".py", 
      "version": "3.6.8", 
      "pygments_lexer": "ipython3", 
      "codemirror_mode": {
        "version": 3, 
        "name": "ipython"
      }
    }, 
    "anaconda-cloud": {}, 
    "nteract": {
      "version": "0.7.1"
    }
  }, 
  "nbformat": 4, 
  "nbformat_minor": 2, 
  "cells": [
    {
      "source": [
        "# Linear Regression Multiple Ways"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "source": [
        "## Making the data\n", 
        "\n", 
        "We'll first construct a synthetic data set..using a function from the `scikit-learn` library. Synthetic data is nice in the sense that we can constrain how the noise behaves, and thus isolate effects."
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 1, 
      "cell_type": "code", 
      "source": [
        "%matplotlib inline\n", 
        "from sklearn.datasets import make_regression\n", 
        "import numpy as np\n", 
        "import matplotlib.pyplot as plt"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 2, 
      "cell_type": "code", 
      "source": [
        "#code adapted from http://tillbergmann.com/blog/python-gradient-descent.html\n", 
        "X, y, coef = make_regression(n_samples = 100, \n", 
        "                       n_features=1, \n", 
        "                       noise=20,\n", 
        "                       random_state=2017,\n", 
        "                       coef=True)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "Notice that the X is in the canonical array-of-arrays format.\n", 
        "**Try and print its shape**"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "We are fitting a model with an intercept. Lets see what it is."
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 3, 
      "cell_type": "code", 
      "source": [
        "coef"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "We can plot the data."
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 4, 
      "cell_type": "code", 
      "source": [
        "plt.plot(X,y, 'o');"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "For the purposes of drawing the regression line, lets create a uniform grid of points, and then reshape it into the canonical format"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 5, 
      "cell_type": "code", 
      "source": [
        "xgrid = np.linspace(-2.5,2.5,1000)\n", 
        "Xgrid = xgrid.reshape(-1,1)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "## Fit using sklearn"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 6, 
      "cell_type": "code", 
      "source": [
        "from sklearn.linear_model import LinearRegression"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 7, 
      "cell_type": "code", 
      "source": [
        "lr = LinearRegression()\n", 
        "lr.fit(X,y)\n", 
        "ypgrid = lr.predict(Xgrid)\n", 
        "lr.coef_, lr.intercept_"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 8, 
      "cell_type": "code", 
      "source": [
        "plt.plot(Xgrid, ypgrid)\n", 
        "plt.plot(X, y, '.')"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 9, 
      "cell_type": "code", 
      "source": [
        "from sklearn.metrics import r2_score"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 10, 
      "cell_type": "code", 
      "source": [
        "r2_score(y, lr.predict(X))"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "## The impact of sample size\n", 
        "\n", 
        "We'll sample 20 points from the data set. We do this by sampling 20 indices, index into X and y, and then fit on the sample"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 11, 
      "cell_type": "code", 
      "source": [
        "sample_indices = np.random.choice(range(100), size=20)\n", 
        "sample_indices"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 12, 
      "cell_type": "code", 
      "source": [
        "Xsample = X[sample_indices]\n", 
        "ysample = y[sample_indices]"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 13, 
      "cell_type": "code", 
      "source": [
        "lr_s = LinearRegression().fit(Xsample, ysample)\n", 
        "r2_score(ysample, lr_s.predict(Xsample)), lr_s.score(Xsample, ysample)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "Lets check the sensitivity of our prediction to our sample. We'll do this 1000 times"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 14, 
      "cell_type": "code", 
      "source": [
        "scores = []\n", 
        "for i in range(1000):\n", 
        "    sample_indices = np.random.choice(range(100), size=20)\n", 
        "    Xsample = X[sample_indices]\n", 
        "    ysample = y[sample_indices]\n", 
        "    scores.append(LinearRegression().fit(Xsample, ysample).score(Xsample, ysample))\n", 
        "plt.hist(scores,  bins=np.linspace(0.7, 1, 30))\n", 
        "plt.xlim(0.7,1)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "Redo this with a higher amount of noise (about 400). For this you will need to create a new dataset. Plot the data. Plot the histogram of the R^2 as well as that of the coefficients.Try a smaller dataset as well. What conclusions can you draw?"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 15, 
      "cell_type": "code", 
      "source": [
        "# your code here\n"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 16, 
      "cell_type": "code", 
      "source": [
        "# your code here\n"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 17, 
      "cell_type": "code", 
      "source": [
        "# your code here\n"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 23, 
      "cell_type": "code", 
      "source": [
        "# your code here\n"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 25, 
      "cell_type": "code", 
      "source": [
        "# your code here\n"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "## Testing and training\n", 
        "\n", 
        "A grid like the one we created might contain some of the points we fit this model on. This is called **Data Contamination** and is a big no-no. If we want an independent estimate of the error, we should hold out some points in a test set."
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 29, 
      "cell_type": "code", 
      "source": [
        "from sklearn.model_selection import train_test_split"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 30, 
      "cell_type": "code", 
      "source": [
        "Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2, random_state=2017)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "Now lets fit the model on the training set and evaluate it both on the training set and the test set. We print the R^2"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 31, 
      "cell_type": "code", 
      "source": [
        "lr2 = LinearRegression().fit(Xtrain, ytrain)\n", 
        "r2_test = r2_score(ytest, lr.predict(Xtest))\n", 
        "r2_train = r2_score(ytrain, lr.predict(Xtrain))"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 32, 
      "cell_type": "code", 
      "source": [
        "\"Train R2 is {}, while test R^2 is {}\".format(r2_train, r2_test)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "## Using Keras to fit the model\n", 
        "\n", 
        "We'll use plain and simple gradient descent (why?) and Keras's Sequential API"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 67, 
      "cell_type": "code", 
      "source": [
        "from keras.models import Sequential\n", 
        "from keras.layers import Dense\n", 
        "lr3 = Sequential()\n", 
        "lr3.add(Dense(1, input_shape=(1,)))\n", 
        "lr3.compile(optimizer='sgd', loss='mean_squared_error',  metrics=['mae','accuracy'])\n", 
        "lr3.summary()"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 68, 
      "cell_type": "code", 
      "source": [
        "history = lr3.fit(Xtrain, ytrain, epochs=400, batch_size=80)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 69, 
      "cell_type": "code", 
      "source": [
        "plt.plot(history.history['loss'])"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 70, 
      "cell_type": "code", 
      "source": [
        "lr3.get_weights()"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "source": [
        "### Using the Keras Functional API\n", 
        "\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": 71, 
      "cell_type": "code", 
      "source": [
        "from keras.layers import Input, Dense\n", 
        "from keras.models import Model\n", 
        "\n", 
        "inputs_placeholder = Input(shape=(1,))\n", 
        "outputs_placeholder = Dense(1, activation='linear')(inputs_placeholder)\n", 
        "\n", 
        "m = Model(inputs=inputs_placeholder, outputs=outputs_placeholder)\n", 
        "m.compile(optimizer='sgd', loss='mean_squared_error',  metrics=['mae','accuracy'])\n", 
        "m.summary()"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 72, 
      "cell_type": "code", 
      "source": [
        "history2 = m.fit(Xtrain, ytrain, epochs=250, batch_size=80)"
      ], 
      "outputs": [], 
      "metadata": {}
    }, 
    {
      "execution_count": 73, 
      "cell_type": "code", 
      "source": [
        "m.get_weights()"
      ], 
      "outputs": [], 
      "metadata": {}
    }
  ]
}