ml/train_test_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Overview\n",
    "\n",
    "We use data to train and evaluate how good a machine learning (ML) model is. A common practice is to split an available dataset into training and test datasets.\n",
    "\n",
    "* Use training data to train the model.\n",
    "* Use test data to tune the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_wine\n",
    "from sklearn.model_selection import train_test_split"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Sample Dataset\n",
    "\n",
    "We use the Scikit-learn (aka `sklearn`) library for simple machine learning in Python. It comes a few [sample (toy) datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html) that can be used for experimentation or exploration of the various algorithms in the library. Here are some datasets:\n",
    "\n",
    "* [The iris dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html) - dataset for classification.\n",
    "* [The diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) - dataset for regression.\n",
    "* [The digits dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) - dataset for classification.\n",
    "* [the wine dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html).\n",
    "* [the breast cancer](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) - dataset for classification.\n",
    "\n",
    "This notebook uses the wine dataset. Let's load the sample dataset wine into a Dataframe object."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
      "0    14.23        1.71  2.43               15.6      127.0           2.80   \n",
      "1    13.20        1.78  2.14               11.2      100.0           2.65   \n",
      "2    13.16        2.36  2.67               18.6      101.0           2.80   \n",
      "3    14.37        1.95  2.50               16.8      113.0           3.85   \n",
      "4    13.24        2.59  2.87               21.0      118.0           2.80   \n",
      "\n",
      "   flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity   hue  \\\n",
      "0        3.06                  0.28             2.29             5.64  1.04   \n",
      "1        2.76                  0.26             1.28             4.38  1.05   \n",
      "2        3.24                  0.30             2.81             5.68  1.03   \n",
      "3        3.49                  0.24             2.18             7.80  0.86   \n",
      "4        2.69                  0.39             1.82             4.32  1.04   \n",
      "\n",
      "   od280/od315_of_diluted_wines  proline  \n",
      "0                          3.92   1065.0  \n",
      "1                          3.40   1050.0  \n",
      "2                          3.17   1185.0  \n",
      "3                          3.45   1480.0  \n",
      "4                          2.93    735.0  \n"
     ]
    }
   ],
   "source": [
    "wine = load_wine()\n",
    "df = pd.DataFrame(data=wine['data'], columns=wine['feature_names'])\n",
    "print(df.head())"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### EDA\n",
    "\n",
    "Let's perform EDA before diving into data analysis and manipulation of the data."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns:\n",
      "Index(['alcohol', 'malic_acid', 'ash', 'alcalinity_of_ash', 'magnesium',\n",
      "       'total_phenols', 'flavanoids', 'nonflavanoid_phenols',\n",
      "       'proanthocyanins', 'color_intensity', 'hue',\n",
      "       'od280/od315_of_diluted_wines', 'proline'],\n",
      "      dtype='object')\n",
      "Dtypes:\n",
      "alcohol                         float64\n",
      "malic_acid                      float64\n",
      "ash                             float64\n",
      "alcalinity_of_ash               float64\n",
      "magnesium                       float64\n",
      "total_phenols                   float64\n",
      "flavanoids                      float64\n",
      "nonflavanoid_phenols            float64\n",
      "proanthocyanins                 float64\n",
      "color_intensity                 float64\n",
      "hue                             float64\n",
      "od280/od315_of_diluted_wines    float64\n",
      "proline                         float64\n",
      "dtype: object\n",
      "Dimension:\n",
      "(178, 13)\n"
     ]
    }
   ],
   "source": [
    "print('Columns:')\n",
    "print(df.columns)\n",
    "print('Dtypes:')\n",
    "print(df.dtypes)\n",
    "print('Dimension:')\n",
    "print(df.shape)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Split Data into Training and Test Data\n",
    "\n",
    "Now we can split the data into data and training datasets. We use the [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function in `sklearn`.\n",
    "\n",
    "| Argument     | Description |\n",
    "|--------------|-------------|\n",
    "| train_size   | Proportion of the dataset to be split into the training dataset. Value should be between 0.0 and 1.0 |\n",
    "| random_state | Controls the shuffling applied to the data before applying split. See [here](https://scikit-learn.org/stable/glossary.html#term-random_state) for details. |\n",
    "| suffle       | If shuffle=True, shuffles the data before splitting. |"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X train dimension:  (117, 13)\n",
      "X test  dimension:  (61, 13)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df, df, train_size=0.66, random_state=42)\n",
    "print('X train dimension: ', X_train.shape)\n",
    "print('X test  dimension: ', X_test.shape)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}