part_3.pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Pandas\n",
    "\n",
    "## A short version of a notebook by Chris Fonnesbeck\n",
    "\n",
    "**pandas** is a Python package providing fast, flexible, and expressive data structures designed to work with *relational* or *labeled* data both. It is a fundamental high-level building block for doing practical, real world data analysis in Python. \n",
    "\n",
    "pandas is well suited for:\n",
    "\n",
    "- Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet\n",
    "- Ordered and unordered (not necessarily fixed-frequency) time series data.\n",
    "- Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels\n",
    "- Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure\n",
    "\n",
    "\n",
    "Key features:\n",
    "    \n",
    "- Easy handling of **missing data**\n",
    "- **Size mutability**: columns can be inserted and deleted from DataFrame and higher dimensional objects\n",
    "- Automatic and explicit **data alignment**: objects can be explicitly aligned to a set of labels, or the data can be aligned automatically\n",
    "- Powerful, flexible **group by functionality** to perform split-apply-combine operations on data sets\n",
    "- Intelligent label-based **slicing, fancy indexing, and subsetting** of large data sets\n",
    "- Intuitive **merging and joining** data sets\n",
    "- Flexible **reshaping and pivoting** of data sets\n",
    "- **Hierarchical labeling** of axes\n",
    "- Robust **IO tools** for loading data from flat files, Excel files, databases, and HDF5\n",
    "- **Time series functionality**: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series\n",
    "\n",
    "A **Series** is a single vector of data (like a NumPy array) with an *index* that labels each element in the vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = pd.Series([632, 1638, 569, 115])\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an index is not specified, a default sequence of integers is assigned as the index. A NumPy array comprises the values of the `Series`, while the index is a pandas `Index` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 632, 1638,  569,  115])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=4, step=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign meaningful labels to the index, if they are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Firmicutes         632\n",
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "Bacteroidetes      115\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria = pd.Series([632, 1638, 569, 115], \n",
    "    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])\n",
    "\n",
    "bacteria"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These labels can be used to refer to the values in the `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "569"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria['Actinobacteria']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[[name.endswith('bacteria') for name in bacteria.index]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False, True, True, False]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name.endswith('bacteria') for name in bacteria.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the indexing operation preserved the association between the values and the corresponding indices.\n",
    "\n",
    "We can still use positional indexing if we wish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "632"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can give both the array of values and the index meaningful labels themselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes         632\n",
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "Bacteroidetes      115\n",
       "Name: counts, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria.name = 'counts'\n",
    "bacteria.index.name = 'phylum'\n",
    "bacteria"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy's math functions and other operations can be applied to Series without losing the data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes        6.448889\n",
       "Proteobacteria    7.401231\n",
       "Actinobacteria    6.343880\n",
       "Bacteroidetes     4.744932\n",
       "Name: counts, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(bacteria)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the series itslef hasn't changed. What we got back is a new series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also filter according to the values in the `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria    1638\n",
       "Name: counts, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[bacteria>1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Series` can be thought of as an ordered key-value store. In fact, we can create one from a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actinobacteria     569\n",
       "Bacteroidetes      115\n",
       "Firmicutes         632\n",
       "Proteobacteria    1638\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria_dict = {'Firmicutes': 632, 'Proteobacteria': 1638, 'Actinobacteria': 569,\n",
    "                 'Bacteroidetes': 115}\n",
    "pd.Series(bacteria_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we pass a custom index to `Series`, it will select the corresponding values from the dict, and treat indices without corrsponding values as missing. Pandas uses the `NaN` (not a number) type for missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cyanobacteria        NaN\n",
       "Firmicutes         632.0\n",
       "Proteobacteria    1638.0\n",
       "Actinobacteria     569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2 = pd.Series(bacteria_dict, \n",
    "                      index=['Cyanobacteria','Firmicutes',\n",
    "                             'Proteobacteria','Actinobacteria'])\n",
    "bacteria2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cyanobacteria      True\n",
       "Firmicutes        False\n",
       "Proteobacteria    False\n",
       "Actinobacteria    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Critically, the labels are used to **align data** when used in operations with other Series objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actinobacteria    1138.0\n",
       "Bacteroidetes        NaN\n",
       "Cyanobacteria        NaN\n",
       "Firmicutes        1264.0\n",
       "Proteobacteria    3276.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria + bacteria2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrast this with NumPy arrays, where arrays of the same length will combine values element-wise; adding Series combined values with the same label in the resulting series. Notice also that the missing values were propogated by addition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame\n",
    "\n",
    "Inevitably, we want to be able to store, view and manipulate data that is *multivariate*, where for every index there are multiple fields or columns of data, often of varying data type.\n",
    "\n",
    "A `DataFrame` is a tabular data structure, encapsulating multiple series like columns in a spreadsheet. Data are stored internally as a 2-dimensional object, but the `DataFrame` allows us to represent and manipulate higher-dimensional data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria   1130\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({'value':[632, 1638, 569, 115, 433, 1130, 754, 555],\n",
    "                     'patient':[1, 1, 1, 1, 2, 2, 2, 2],\n",
    "                     'phylum':['Firmicutes', 'Proteobacteria', 'Actinobacteria', \n",
    "    'Bacteroidetes', 'Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes']})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the `DataFrame` is sorted by column name. We can change the order by indexing them in the order we desire:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>patient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1130</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           phylum  value  patient\n",
       "0      Firmicutes    632        1\n",
       "1  Proteobacteria   1638        1\n",
       "2  Actinobacteria    569        1\n",
       "3   Bacteroidetes    115        1\n",
       "4      Firmicutes    433        2\n",
       "5  Proteobacteria   1130        2\n",
       "6  Actinobacteria    754        2\n",
       "7   Bacteroidetes    555        2"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[['phylum','value','patient']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `DataFrame` has a second index, representing the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['patient', 'phylum', 'value'], dtype='object')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `dtypes` attribute reveals the data type for each column in our DataFrame. \n",
    "\n",
    "- `int64` is numeric integer values \n",
    "- `object` strings (letters and numbers)\n",
    "- `float64` floating-point values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "patient     int64\n",
       "phylum     object\n",
       "value       int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wish to access columns, we can do so either by dict-like indexing or by attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    2\n",
       "5    2\n",
       "6    2\n",
       "7    2\n",
       "Name: patient, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['patient']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    2\n",
       "5    2\n",
       "6    2\n",
       "7    2\n",
       "Name: patient, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.patient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   value\n",
       "0    632\n",
       "1   1638\n",
       "2    569\n",
       "3    115\n",
       "4    433\n",
       "5   1130\n",
       "6    754\n",
       "7    555"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[['value']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice this is different than with `Series`, where dict-like indexing retrieved a particular element (row). \n",
    "\n",
    "If we want access to a row in a `DataFrame`, we index its `loc` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "patient                1\n",
       "phylum     Bacteroidetes\n",
       "value                115\n",
       "Name: 3, dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way of initializing a `DataFrame` is with a list of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame([{'patient': 1, 'phylum': 'Firmicutes', 'value': 632},\n",
    "                    {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},\n",
    "                    {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},\n",
    "                    {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},\n",
    "                    {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},\n",
    "                    {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},\n",
    "                    {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},\n",
    "                    {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria   1130\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its important to note that the Series returned when a DataFrame is indexted is merely a **view** on the DataFrame, and not a copy of the data itself. So you must be cautious when manipulating this data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "4     433\n",
       "5    1130\n",
       "6     754\n",
       "7     555\n",
       "Name: value, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals = data.value\n",
    "vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "4     433\n",
       "5       0\n",
       "6     754\n",
       "7     555\n",
       "Name: value, dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals[5] = 0\n",
    "vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we plan on modifying an extracted Series, its a good idea to make a copy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals = data.value.copy()\n",
    "vals[5] = 1000\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create or modify columns by assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes     14\n",
       "4        2      Firmicutes     21\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria      5\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.value[[3,4,6]] = [14, 21, 5]\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year\n",
       "0        1      Firmicutes    632  2013\n",
       "1        1  Proteobacteria   1638  2013\n",
       "2        1  Actinobacteria    569  2013\n",
       "3        1   Bacteroidetes     14  2013\n",
       "4        2      Firmicutes     21  2013\n",
       "5        2  Proteobacteria      0  2013\n",
       "6        2  Actinobacteria      5  2013\n",
       "7        2   Bacteroidetes    555  2013"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['year'] = 2013\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But note, we cannot use the attribute indexing method to add a new column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year\n",
       "0        1      Firmicutes    632  2013\n",
       "1        1  Proteobacteria   1638  2013\n",
       "2        1  Actinobacteria    569  2013\n",
       "3        1   Bacteroidetes     14  2013\n",
       "4        2      Firmicutes     21  2013\n",
       "5        2  Proteobacteria      0  2013\n",
       "6        2  Actinobacteria      5  2013\n",
       "7        2   Bacteroidetes    555  2013"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.treatment = 1\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.treatment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specifying a `Series` as a new columns cause its values to be added according to the `DataFrame`'s index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    1\n",
       "5    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treatment = pd.Series([0]*4 + [1]*2)\n",
    "treatment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0\n",
       "6        2  Actinobacteria      5  2013        NaN\n",
       "7        2   Bacteroidetes    555  2013        NaN"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['treatment'] = treatment\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other Python data structures (ones without an index) need to be the same length as the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Length of values does not match length of index",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m-------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-40-360d03fdde9a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmonth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Jan'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Feb'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Mar'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Apr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'month'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmonth\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m__setitem__\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   2355\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2356\u001b[0m             \u001b[0;31m# set column\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2357\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_item\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2358\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2359\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_setitem_slice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_set_item\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   2421\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2422\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ensure_valid_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2423\u001b[0;31m         \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sanitize_column\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2424\u001b[0m         \u001b[0mNDFrame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_item\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2425\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_sanitize_column\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   2576\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2577\u001b[0m             \u001b[0;31m# turn me into an ndarray\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2578\u001b[0;31m             \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_sanitize_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2579\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2580\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m_sanitize_index\u001b[0;34m(data, index, copy)\u001b[0m\n\u001b[1;32m   2768\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2769\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2770\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Length of values does not match length of '\u001b[0m \u001b[0;34m'index'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2771\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2772\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mPeriodIndex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Length of values does not match length of index"
     ]
    }
   ],
   "source": [
    "month = ['Jan', 'Feb', 'Mar', 'Apr']\n",
    "data['month'] = month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment month\n",
       "0        1      Firmicutes    632  2013        0.0   Jan\n",
       "1        1  Proteobacteria   1638  2013        0.0   Jan\n",
       "2        1  Actinobacteria    569  2013        0.0   Jan\n",
       "3        1   Bacteroidetes     14  2013        0.0   Jan\n",
       "4        2      Firmicutes     21  2013        1.0   Jan\n",
       "5        2  Proteobacteria      0  2013        1.0   Jan\n",
       "6        2  Actinobacteria      5  2013        NaN   Jan\n",
       "7        2   Bacteroidetes    555  2013        NaN   Jan"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['month'] = ['Jan']*len(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `drop` method to remove rows or columns, which by default drops rows. We can be explicit by using the `axis` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0\n",
       "6        2  Actinobacteria      5  2013        NaN\n",
       "7        2   Bacteroidetes    555  2013        NaN"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.drop('month', axis=1)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Either specify 'inplace=True' or remember to asign the new datafrafe after the drop!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the underlying data as a simple `ndarray` by accessing the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 'Firmicutes', 632, 2013, 0.0],\n",
       "       [1, 'Proteobacteria', 1638, 2013, 0.0],\n",
       "       [1, 'Actinobacteria', 569, 2013, 0.0],\n",
       "       [1, 'Bacteroidetes', 14, 2013, 0.0],\n",
       "       [2, 'Firmicutes', 21, 2013, 1.0],\n",
       "       [2, 'Proteobacteria', 0, 2013, 1.0],\n",
       "       [2, 'Actinobacteria', 5, 2013, nan],\n",
       "       [2, 'Bacteroidetes', 555, 2013, nan]], dtype=object)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that because of the mix of string and integer (and `NaN`) values, the dtype of the array is `object`. The dtype will automatically be chosen to be as general as needed to accomodate all the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.4,  1. ],\n",
       "       [-1. ,  2. ],\n",
       "       [ 4.5,  3. ]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'foo': [1,2,3], 'bar':[0.4, -1.0, 4.5]})\n",
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A key, but often under-appreciated, step in data analysis is importing the data that we wish to analyze. Though it is easy to load basic data structures into Python using built-in tools or those provided by packages like NumPy, it is non-trivial to import structured data well, and to easily convert this input into a robust data structure:\n",
    "\n",
    "    genes = np.loadtxt(\"genes.csv\", delimiter=\",\", dtype=[('gene', '|S10'), ('value', '<f4')])\n",
    "\n",
    "Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a `DataFrame` object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with some more bacteria data, stored in csv format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taxonn,Patient,Group,Tissue,Stool\r\n",
      "Firmicutes,1,0,136,4182\r\n",
      "Firmicutes,2,1,1174,703\r\n",
      "Firmicutes,3,0,408,3946\r\n",
      "Firmicutes,4,1,831,8605\r\n",
      "Firmicutes,5,0,693,50\r\n",
      "Firmicutes,6,1,718,717\r\n",
      "Firmicutes,7,0,173,33\r\n",
      "Firmicutes,8,1,228,80\r\n",
      "Firmicutes,9,0,162,3196\r\n",
      "Firmicutes,10,1,372,32\r\n",
      "Firmicutes,11,0,4255,4361\r\n",
      "Firmicutes,12,1,107,1667\r\n",
      "Firmicutes,13,0,96,223\r\n",
      "Firmicutes,14,1,281,2377\r\n",
      "Proteobacteria,1,0,2469,1821\r\n",
      "Proteobacteria,2,1,839,661\r\n",
      "Proteobacteria,3,0,4414,18\r\n",
      "Proteobacteria,4,1,12044,83\r\n",
      "Proteobacteria,5,0,2310,12\r\n",
      "Proteobacteria,6,1,3053,547\r\n",
      "Proteobacteria,7,0,395,2174\r\n",
      "Proteobacteria,8,1,2651,767\r\n",
      "Proteobacteria,9,0,1195,76\r\n",
      "Proteobacteria,10,1,6857,795\r\n",
      "Proteobacteria,11,0,483,666\r\n",
      "Proteobacteria,12,1,2950,3994\r\n",
      "Proteobacteria,13,0,1541,816\r\n",
      "Proteobacteria,14,1,1307,53\r\n",
      "Actinobacteria,1,0,1590,4\r\n",
      "Actinobacteria,2,1,25,2\r\n",
      "Actinobacteria,3,0,259,300\r\n",
      "Actinobacteria,4,1,568,7\r\n",
      "Actinobacteria,5,0,1102,9\r\n",
      "Actinobacteria,6,1,678,377\r\n",
      "Actinobacteria,7,0,260,58\r\n",
      "Actinobacteria,8,1,424,233\r\n",
      "Actinobacteria,9,0,548,21\r\n",
      "Actinobacteria,10,1,201,83\r\n",
      "Actinobacteria,11,0,42,75\r\n",
      "Actinobacteria,12,1,109,59\r\n",
      "Actinobacteria,13,0,51,183\r\n",
      "Actinobacteria,14,1,310,204\r\n",
      "Bacteroidetes,1,0,67,0\r\n",
      "Bacteroidetes,2,1,0,0\r\n",
      "Bacteroidetes,3,0,85,5\r\n",
      "Bacteroidetes,4,1,143,7\r\n",
      "Bacteroidetes,5,0,678,2\r\n",
      "Bacteroidetes,6,1,4829,209\r\n",
      "Bacteroidetes,7,0,74,651\r\n",
      "Bacteroidetes,8,1,169,254\r\n",
      "Bacteroidetes,9,0,106,10\r\n",
      "Bacteroidetes,10,1,73,381\r\n",
      "Bacteroidetes,11,0,30,359\r\n",
      "Bacteroidetes,12,1,51,51\r\n",
      "Bacteroidetes,13,0,2473,2314\r\n",
      "Bacteroidetes,14,1,102,33\r\n",
      "Other,1,0,195,18\r\n",
      "Other,2,1,42,2\r\n",
      "Other,3,0,316,43\r\n",
      "Other,4,1,202,40\r\n",
      "Other,5,0,116,0\r\n",
      "Other,6,1,527,12\r\n",
      "Other,7,0,357,11\r\n",
      "Other,8,1,106,11\r\n",
      "Other,9,0,67,14\r\n",
      "Other,10,1,203,6\r\n",
      "Other,11,0,392,6\r\n",
      "Other,12,1,28,25\r\n",
      "Other,13,0,12,22\r\n",
      "Other,14,1,305,32\r\n"
     ]
    }
   ],
   "source": [
    "!cat microbiome.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table can be read into a DataFrame using `read_csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Taxon  Patient  Group  Tissue  Stool\n",
       "0  Firmicutes        1      0     136   4182\n",
       "1  Firmicutes        2      1    1174    703\n",
       "2  Firmicutes        3      0     408   3946\n",
       "3  Firmicutes        4      1     831   8605\n",
       "4  Firmicutes        5      0     693     50"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb = pd.read_csv(\"microbiome.csv\")\n",
    "mb.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `read_csv` automatically considered the first row in the file to be a header row.\n",
    "\n",
    "We can override default behavior by customizing some the arguments, like `header`, `names` or `index_col`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Taxon</td>\n",
       "      <td>Patient</td>\n",
       "      <td>Group</td>\n",
       "      <td>Tissue</td>\n",
       "      <td>Stool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0        1      2       3      4\n",
       "0       Taxon  Patient  Group  Tissue  Stool\n",
       "1  Firmicutes        1      0     136   4182\n",
       "2  Firmicutes        2      1    1174    703\n",
       "3  Firmicutes        3      0     408   3946\n",
       "4  Firmicutes        4      1     831   8605"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome.csv\", header=None).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`read_csv` is just a convenience function for `read_table`, since csv is such a common format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb = pd.read_table(\"microbiome.csv\", sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sep` argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats: \n",
    "    \n",
    "    sep='\\s+'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more useful index, we can specify the first two columns, which together provide a unique index to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Patient</th>\n",
       "      <th>Taxon</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Group  Tissue  Stool\n",
       "Patient Taxon                           \n",
       "1       Firmicutes      0     136   4182\n",
       "2       Firmicutes      1    1174    703\n",
       "3       Firmicutes      0     408   3946\n",
       "4       Firmicutes      1     831   8605\n",
       "5       Firmicutes      0     693     50"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb = pd.read_csv(\"microbiome.csv\", index_col=['Patient','Taxon'])\n",
    "mb.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is called a *hierarchical* index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have sections of data that we do not wish to import (for example, known bad data), we can populate the `skiprows` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>173</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>228</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Taxon  Patient  Group  Tissue  Stool\n",
       "0  Firmicutes        1      0     136   4182\n",
       "1  Firmicutes        2      1    1174    703\n",
       "2  Firmicutes        5      0     693     50\n",
       "3  Firmicutes        7      0     173     33\n",
       "4  Firmicutes        8      1     228     80"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome.csv\", skiprows=[3,4,6]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we only want to import a small number of rows from, say, a very large data file we can use `nrows`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Taxon  Patient  Group  Tissue  Stool\n",
       "0  Firmicutes        1      0     136   4182\n",
       "1  Firmicutes        2      1    1174    703\n",
       "2  Firmicutes        3      0     408   3946\n",
       "3  Firmicutes        4      1     831   8605"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome.csv\", nrows=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, if we want to process our data in reasonable chunks, the `chunksize` argument will return an iterable object that can be employed in a data processing loop. For example, our microbiome data are organized by bacterial phylum, with 15 patients represented in each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.io.parsers.TextFileReader at 0x10d05bd68>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome.csv\", chunksize=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actinobacteria     313.600000\n",
       "Bacteroidetes      632.200000\n",
       "Firmicutes         806.866667\n",
       "Other              211.300000\n",
       "Proteobacteria    2776.933333\n",
       "dtype: float64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_chunks = pd.read_csv(\"microbiome.csv\", chunksize=15)\n",
    "\n",
    "mean_tissue = pd.Series({chunk.Taxon[0]: chunk.Tissue.mean() for chunk in data_chunks})\n",
    "    \n",
    "mean_tissue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most real-world data is incomplete, with values missing due to incomplete observation, data entry or transcription error, or other reasons. Pandas will automatically recognize and parse common missing data indicators, including `NA` and `NULL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taxon,Patient,Tissue,Stool\r\n",
      "Firmicutes,1,632,305\r\n",
      "Firmicutes,2,136,4182\r\n",
      "Firmicutes,3,,703\r\n",
      "Firmicutes,4,408,3946\r\n",
      "Firmicutes,5,831,8605\r\n",
      "Firmicutes,6,693,50\r\n",
      "Firmicutes,7,718,717\r\n",
      "Firmicutes,8,173,33\r\n",
      "Firmicutes,9,228,NA\r\n",
      "Firmicutes,10,162,3196\r\n",
      "Firmicutes,11,372,-99999\r\n",
      "Firmicutes,12,4255,4361\r\n",
      "Firmicutes,13,107,1667\r\n",
      "Firmicutes,14,?,223\r\n",
      "Firmicutes,15,281,2377\r\n",
      "Proteobacteria,1,1638,3886\r\n",
      "Proteobacteria,2,2469,1821\r\n",
      "Proteobacteria,3,839,661\r\n",
      "Proteobacteria,4,4414,18\r\n",
      "Proteobacteria,5,12044,83\r\n",
      "Proteobacteria,6,2310,12\r\n",
      "Proteobacteria,7,3053,547\r\n",
      "Proteobacteria,8,395,2174\r\n",
      "Proteobacteria,9,2651,767\r\n",
      "Proteobacteria,10,1195,76\r\n",
      "Proteobacteria,11,6857,795\r\n",
      "Proteobacteria,12,483,666\r\n",
      "Proteobacteria,13,2950,3994\r\n",
      "Proteobacteria,14,1541,816\r\n",
      "Proteobacteria,15,1307,53\r\n",
      "Actinobacteria,1,569,648\r\n",
      "Actinobacteria,2,1590,4\r\n",
      "Actinobacteria,3,25,2\r\n",
      "Actinobacteria,4,259,300\r\n",
      "Actinobacteria,5,568,7\r\n",
      "Actinobacteria,6,1102,9\r\n",
      "Actinobacteria,7,678,377\r\n",
      "Actinobacteria,8,260,58\r\n",
      "Actinobacteria,9,424,233\r\n",
      "Actinobacteria,10,548,21\r\n",
      "Actinobacteria,11,201,83\r\n",
      "Actinobacteria,12,42,75\r\n",
      "Actinobacteria,13,109,59\r\n",
      "Actinobacteria,14,51,183\r\n",
      "Actinobacteria,15,310,204\r\n",
      "Bacteroidetes,1,115,380\r\n",
      "Bacteroidetes,2,67,0\r\n",
      "Bacteroidetes,3,0,0\r\n",
      "Bacteroidetes,4,85,5\r\n",
      "Bacteroidetes,5,143,7\r\n",
      "Bacteroidetes,6,678,2\r\n",
      "Bacteroidetes,7,4829,209\r\n",
      "Bacteroidetes,8,74,651\r\n",
      "Bacteroidetes,9,169,254\r\n",
      "Bacteroidetes,10,106,10\r\n",
      "Bacteroidetes,11,73,381\r\n",
      "Bacteroidetes,12,30,359\r\n",
      "Bacteroidetes,13,51,51\r\n",
      "Bacteroidetes,14,2473,2314\r\n",
      "Bacteroidetes,15,102,33\r\n",
      "Other,1,114,277\r\n",
      "Other,2,195,18\r\n",
      "Other,3,42,2\r\n",
      "Other,4,316,43\r\n",
      "Other,5,202,40\r\n",
      "Other,6,116,0\r\n",
      "Other,7,527,12\r\n",
      "Other,8,357,11\r\n",
      "Other,9,106,11\r\n",
      "Other,10,67,14\r\n",
      "Other,11,203,6\r\n",
      "Other,12,392,6\r\n",
      "Other,13,28,25\r\n",
      "Other,14,12,22\r\n",
      "Other,15,305,32\r\n"
     ]
    }
   ],
   "source": [
    "!cat microbiome_missing.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>632</td>\n",
       "      <td>305.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>136</td>\n",
       "      <td>4182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>703.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>408</td>\n",
       "      <td>3946.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>831</td>\n",
       "      <td>8605.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>6</td>\n",
       "      <td>693</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>718</td>\n",
       "      <td>717.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>173</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>9</td>\n",
       "      <td>228</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>10</td>\n",
       "      <td>162</td>\n",
       "      <td>3196.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>11</td>\n",
       "      <td>372</td>\n",
       "      <td>-99999.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>12</td>\n",
       "      <td>4255</td>\n",
       "      <td>4361.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>13</td>\n",
       "      <td>107</td>\n",
       "      <td>1667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>14</td>\n",
       "      <td>?</td>\n",
       "      <td>223.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>15</td>\n",
       "      <td>281</td>\n",
       "      <td>2377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>1638</td>\n",
       "      <td>3886.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>2469</td>\n",
       "      <td>1821.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>3</td>\n",
       "      <td>839</td>\n",
       "      <td>661.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>4</td>\n",
       "      <td>4414</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>12044</td>\n",
       "      <td>83.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Taxon  Patient Tissue    Stool\n",
       "0       Firmicutes        1    632    305.0\n",
       "1       Firmicutes        2    136   4182.0\n",
       "2       Firmicutes        3    NaN    703.0\n",
       "3       Firmicutes        4    408   3946.0\n",
       "4       Firmicutes        5    831   8605.0\n",
       "5       Firmicutes        6    693     50.0\n",
       "6       Firmicutes        7    718    717.0\n",
       "7       Firmicutes        8    173     33.0\n",
       "8       Firmicutes        9    228      NaN\n",
       "9       Firmicutes       10    162   3196.0\n",
       "10      Firmicutes       11    372 -99999.0\n",
       "11      Firmicutes       12   4255   4361.0\n",
       "12      Firmicutes       13    107   1667.0\n",
       "13      Firmicutes       14      ?    223.0\n",
       "14      Firmicutes       15    281   2377.0\n",
       "15  Proteobacteria        1   1638   3886.0\n",
       "16  Proteobacteria        2   2469   1821.0\n",
       "17  Proteobacteria        3    839    661.0\n",
       "18  Proteobacteria        4   4414     18.0\n",
       "19  Proteobacteria        5  12044     83.0"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome_missing.csv\").head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, Pandas recognized `NA` and an empty field as missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Taxon Patient Tissue  Stool\n",
       "0   False   False  False  False\n",
       "1   False   False  False  False\n",
       "2   False   False   True  False\n",
       "3   False   False  False  False\n",
       "4   False   False  False  False\n",
       "5   False   False  False  False\n",
       "6   False   False  False  False\n",
       "7   False   False  False  False\n",
       "8   False   False  False   True\n",
       "9   False   False  False  False\n",
       "10  False   False  False  False\n",
       "11  False   False  False  False\n",
       "12  False   False  False  False\n",
       "13  False   False  False  False\n",
       "14  False   False  False  False\n",
       "15  False   False  False  False\n",
       "16  False   False  False  False\n",
       "17  False   False  False  False\n",
       "18  False   False  False  False\n",
       "19  False   False  False  False"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.isnull(pd.read_csv(\"microbiome_missing.csv\")).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, there will sometimes be inconsistency with the conventions for missing data. In this example, there is a question mark \"?\" and a large negative number where there should have been a positive integer. We can specify additional symbols with the `na_values` argument:\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>632.0</td>\n",
       "      <td>305.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>136.0</td>\n",
       "      <td>4182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>703.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>408.0</td>\n",
       "      <td>3946.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>831.0</td>\n",
       "      <td>8605.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>6</td>\n",
       "      <td>693.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>718.0</td>\n",
       "      <td>717.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>173.0</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>9</td>\n",
       "      <td>228.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>10</td>\n",
       "      <td>162.0</td>\n",
       "      <td>3196.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>11</td>\n",
       "      <td>372.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>12</td>\n",
       "      <td>4255.0</td>\n",
       "      <td>4361.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>13</td>\n",
       "      <td>107.0</td>\n",
       "      <td>1667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>223.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>15</td>\n",
       "      <td>281.0</td>\n",
       "      <td>2377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>1638.0</td>\n",
       "      <td>3886.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>2469.0</td>\n",
       "      <td>1821.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>3</td>\n",
       "      <td>839.0</td>\n",
       "      <td>661.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>4</td>\n",
       "      <td>4414.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>12044.0</td>\n",
       "      <td>83.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Taxon  Patient   Tissue   Stool\n",
       "0       Firmicutes        1    632.0   305.0\n",
       "1       Firmicutes        2    136.0  4182.0\n",
       "2       Firmicutes        3      NaN   703.0\n",
       "3       Firmicutes        4    408.0  3946.0\n",
       "4       Firmicutes        5    831.0  8605.0\n",
       "5       Firmicutes        6    693.0    50.0\n",
       "6       Firmicutes        7    718.0   717.0\n",
       "7       Firmicutes        8    173.0    33.0\n",
       "8       Firmicutes        9    228.0     NaN\n",
       "9       Firmicutes       10    162.0  3196.0\n",
       "10      Firmicutes       11    372.0     NaN\n",
       "11      Firmicutes       12   4255.0  4361.0\n",
       "12      Firmicutes       13    107.0  1667.0\n",
       "13      Firmicutes       14      NaN   223.0\n",
       "14      Firmicutes       15    281.0  2377.0\n",
       "15  Proteobacteria        1   1638.0  3886.0\n",
       "16  Proteobacteria        2   2469.0  1821.0\n",
       "17  Proteobacteria        3    839.0   661.0\n",
       "18  Proteobacteria        4   4414.0    18.0\n",
       "19  Proteobacteria        5  12044.0    83.0"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"microbiome_missing.csv\", na_values=['?', -99999]).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These can be specified on a column-wise basis using an appropriate dict as the argument for `na_values`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing data\n",
    "\n",
    "The occurence of missing data is so prevalent that it pays to use tools like Pandas, which seamlessly integrates missing data handling so that it can be dealt with easily, and in the manner required by the analysis at hand.\n",
    "\n",
    "Missing data are represented in `Series` and `DataFrame` objects by the `NaN` floating point value. However, `None` is also treated as missing, since it is commonly used as such in other contexts (*e.g.* NumPy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       NaN\n",
       "1        -3\n",
       "2      None\n",
       "3    foobar\n",
       "dtype: object"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foo = pd.Series([np.nan, -3, None, 'foobar'])\n",
    "foo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1    False\n",
       "2     True\n",
       "3    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foo.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing values may be dropped or indexed out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes           NaN\n",
       "Proteobacteria     632.0\n",
       "Actinobacteria    1638.0\n",
       "Bacteroidetes      569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "bacteria2.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria    False\n",
       "Actinobacteria    False\n",
       "Bacteroidetes     False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria     632.0\n",
       "Actinobacteria    1638.0\n",
       "Bacteroidetes      569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2[bacteria2.notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `dropna` drops entire rows in which one or more values are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be overridden by passing the `how='all'` argument, which only drops a row when every field is a missing value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0\n",
       "6        2  Actinobacteria      5  2013        NaN\n",
       "7        2   Bacteroidetes    555  2013        NaN"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dropna(how='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be customized further by specifying how many values need to be present before a row is dropped via the `thresh` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value    year  treatment\n",
       "0        1      Firmicutes    632  2013.0        0.0\n",
       "1        1  Proteobacteria   1638  2013.0        0.0\n",
       "2        1  Actinobacteria    569  2013.0        0.0\n",
       "3        1   Bacteroidetes     14  2013.0        0.0\n",
       "4        2      Firmicutes     21  2013.0        1.0\n",
       "5        2  Proteobacteria      0  2013.0        1.0\n",
       "6        2  Actinobacteria      5  2013.0        NaN\n",
       "7        2   Bacteroidetes    555     NaN        NaN"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[7, 'year'] = np.nan\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value    year  treatment\n",
       "0        1      Firmicutes    632  2013.0        0.0\n",
       "1        1  Proteobacteria   1638  2013.0        0.0\n",
       "2        1  Actinobacteria    569  2013.0        0.0\n",
       "3        1   Bacteroidetes     14  2013.0        0.0\n",
       "4        2      Firmicutes     21  2013.0        1.0\n",
       "5        2  Proteobacteria      0  2013.0        1.0\n",
       "6        2  Actinobacteria      5  2013.0        NaN"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dropna(thresh=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is typically used in time series applications, where there are repeated measurements that are incomplete for some subjects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than omitting missing data from an analysis, in some cases it may be suitable to fill the missing value in, either with a default value (such as zero) or a value that is either imputed or carried forward/backward from similar data points. We can do this programmatically in Pandas with the `fillna` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria     632.0\n",
       "Actinobacteria    1638.0\n",
       "Bacteroidetes      569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value    year  treatment\n",
       "0        1      Firmicutes    632  2013.0        0.0\n",
       "1        1  Proteobacteria   1638  2013.0        0.0\n",
       "2        1  Actinobacteria    569  2013.0        0.0\n",
       "3        1   Bacteroidetes     14  2013.0        0.0\n",
       "4        2      Firmicutes     21  2013.0        1.0\n",
       "5        2  Proteobacteria      0  2013.0        1.0\n",
       "6        2  Actinobacteria      5  2013.0        2.0\n",
       "7        2   Bacteroidetes    555  2013.0        2.0"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.fillna({'year': 2013, 'treatment':2})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `fillna` by default returns a new object with the desired filling behavior, rather than changing the `Series` or  `DataFrame` in place (**in general, we like to do this, by the way!**)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can alter values in-place using `inplace=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value    year  treatment\n",
       "0        1      Firmicutes    632  2013.0        0.0\n",
       "1        1  Proteobacteria   1638  2013.0        0.0\n",
       "2        1  Actinobacteria    569  2013.0        0.0\n",
       "3        1   Bacteroidetes     14  2013.0        0.0\n",
       "4        2      Firmicutes     21  2013.0        1.0\n",
       "5        2  Proteobacteria      0  2013.0        1.0\n",
       "6        2  Actinobacteria      5  2013.0        NaN\n",
       "7        2   Bacteroidetes    555  2013.0        NaN"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.year.fillna(2013, inplace=True)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing values can also be interpolated, using any one of a variety of methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria     632.0\n",
       "Actinobacteria    1638.0\n",
       "Bacteroidetes      569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.fillna(method='bfill')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data summarization\n",
    "\n",
    "We often wish to summarize data in `Series` or `DataFrame` objects, so that they can more easily be understood or compared with similar data. The NumPy package contains several functions that are useful here, but several summarization or reduction methods are built into Pandas data structures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "patient                                                     12\n",
       "phylum       FirmicutesProteobacteriaActinobacteriaBacteroi...\n",
       "value                                                     3434\n",
       "year                                                     16104\n",
       "treatment                                                    2\n",
       "dtype: object"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, `sum` is more meaningful for some columns than others. For methods like `mean` for which application to string variables is not just meaningless, but impossible, these columns are automatically exculded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "patient         1.500000\n",
       "value         429.250000\n",
       "year         2013.000000\n",
       "treatment       0.333333\n",
       "dtype: float64"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The important difference between NumPy's functions and Pandas' methods is that the latter have built-in support for handling missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cyanobacteria        NaN\n",
       "Firmicutes         632.0\n",
       "Proteobacteria    1638.0\n",
       "Actinobacteria     569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "946.33333333333337"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we may not want to ignore missing values, and allow the `nan` to propagate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.mean(skipna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Passing `axis=1` will summarize over rows instead of columns, which only makes sense in certain situations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A useful summarization that gives a quick snapshot of multiple statistics for a `Series` or `DataFrame` is `describe`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/miniconda3/envs/detection/lib/python3.5/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile\n",
      "  RuntimeWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count       3.000000\n",
       "mean      946.333333\n",
       "std       599.828587\n",
       "min       569.000000\n",
       "25%              NaN\n",
       "50%              NaN\n",
       "75%              NaN\n",
       "max      1638.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`describe` can detect non-numeric data and sometimes yield useful information about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing Data to Files\n",
    "\n",
    "As well as being able to read several data input formats, Pandas can also export data to a variety of storage formats. We will bring your attention to just a couple of these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb.to_csv(\"mb.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `to_csv` method writes a `DataFrame` to a comma-separated values (csv) file. You can specify custom delimiters (via `sep` argument), how missing values are written (via `na_rep` argument), whether the index is writen (via `index` argument), whether the header is included (via `header` argument), among other options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An efficient way of storing data to disk is in binary format. Pandas supports this using Python’s built-in pickle serialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb.to_pickle(\"mb_pickle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complement to `to_pickle` is the `read_pickle` function, which restores the pickle to a `DataFrame` or `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Patient</th>\n",
       "      <th>Taxon</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>718</td>\n",
       "      <td>717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>173</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>228</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>162</td>\n",
       "      <td>3196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>372</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>4255</td>\n",
       "      <td>4361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>107</td>\n",
       "      <td>1667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>96</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>281</td>\n",
       "      <td>2377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>2469</td>\n",
       "      <td>1821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>839</td>\n",
       "      <td>661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>4414</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>12044</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>2310</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>3053</td>\n",
       "      <td>547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>395</td>\n",
       "      <td>2174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>2651</td>\n",
       "      <td>767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>1195</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>6857</td>\n",
       "      <td>795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>483</td>\n",
       "      <td>666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>3994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>1541</td>\n",
       "      <td>816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>Proteobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>1307</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Actinobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>1590</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Actinobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <th>Actinobacteria</th>\n",
       "      <td>0</td>\n",
       "      <td>51</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>Actinobacteria</th>\n",
       "      <td>1</td>\n",
       "      <td>310</td>\n",
       "      <td>204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>85</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>143</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>678</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>4829</td>\n",
       "      <td>209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>74</td>\n",
       "      <td>651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>169</td>\n",
       "      <td>254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>106</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>73</td>\n",
       "      <td>381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>0</td>\n",
       "      <td>2473</td>\n",
       "      <td>2314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>Bacteroidetes</th>\n",
       "      <td>1</td>\n",
       "      <td>102</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>316</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>202</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>116</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>527</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>357</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>106</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>67</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>203</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>392</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <th>Other</th>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>Other</th>\n",
       "      <td>1</td>\n",
       "      <td>305</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>70 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Group  Tissue  Stool\n",
       "Patient Taxon                               \n",
       "1       Firmicutes          0     136   4182\n",
       "2       Firmicutes          1    1174    703\n",
       "3       Firmicutes          0     408   3946\n",
       "4       Firmicutes          1     831   8605\n",
       "5       Firmicutes          0     693     50\n",
       "6       Firmicutes          1     718    717\n",
       "7       Firmicutes          0     173     33\n",
       "8       Firmicutes          1     228     80\n",
       "9       Firmicutes          0     162   3196\n",
       "10      Firmicutes          1     372     32\n",
       "11      Firmicutes          0    4255   4361\n",
       "12      Firmicutes          1     107   1667\n",
       "13      Firmicutes          0      96    223\n",
       "14      Firmicutes          1     281   2377\n",
       "1       Proteobacteria      0    2469   1821\n",
       "2       Proteobacteria      1     839    661\n",
       "3       Proteobacteria      0    4414     18\n",
       "4       Proteobacteria      1   12044     83\n",
       "5       Proteobacteria      0    2310     12\n",
       "6       Proteobacteria      1    3053    547\n",
       "7       Proteobacteria      0     395   2174\n",
       "8       Proteobacteria      1    2651    767\n",
       "9       Proteobacteria      0    1195     76\n",
       "10      Proteobacteria      1    6857    795\n",
       "11      Proteobacteria      0     483    666\n",
       "12      Proteobacteria      1    2950   3994\n",
       "13      Proteobacteria      0    1541    816\n",
       "14      Proteobacteria      1    1307     53\n",
       "1       Actinobacteria      0    1590      4\n",
       "2       Actinobacteria      1      25      2\n",
       "...                       ...     ...    ...\n",
       "13      Actinobacteria      0      51    183\n",
       "14      Actinobacteria      1     310    204\n",
       "1       Bacteroidetes       0      67      0\n",
       "2       Bacteroidetes       1       0      0\n",
       "3       Bacteroidetes       0      85      5\n",
       "4       Bacteroidetes       1     143      7\n",
       "5       Bacteroidetes       0     678      2\n",
       "6       Bacteroidetes       1    4829    209\n",
       "7       Bacteroidetes       0      74    651\n",
       "8       Bacteroidetes       1     169    254\n",
       "9       Bacteroidetes       0     106     10\n",
       "10      Bacteroidetes       1      73    381\n",
       "11      Bacteroidetes       0      30    359\n",
       "12      Bacteroidetes       1      51     51\n",
       "13      Bacteroidetes       0    2473   2314\n",
       "14      Bacteroidetes       1     102     33\n",
       "1       Other               0     195     18\n",
       "2       Other               1      42      2\n",
       "3       Other               0     316     43\n",
       "4       Other               1     202     40\n",
       "5       Other               0     116      0\n",
       "6       Other               1     527     12\n",
       "7       Other               0     357     11\n",
       "8       Other               1     106     11\n",
       "9       Other               0      67     14\n",
       "10      Other               1     203      6\n",
       "11      Other               0     392      6\n",
       "12      Other               1      28     25\n",
       "13      Other               0      12     22\n",
       "14      Other               1     305     32\n",
       "\n",
       "[70 rows x 3 columns]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_pickle(\"mb_pickle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As Wes warns in his book, it is recommended that binary storage of data via pickle only be used as a temporary storage format, in situations where speed is relevant. This is because there is no guarantee that the pickle format will not change with future versions of Python."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": false,
   "threshold": "3",
   "toc_cell": true,
   "toc_section_display": "none",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}