64_imputation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Imputation of Missing Values](https://scikit-learn.org/stable/modules/impute.html)\n",
    "\n",
    "- Many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. They are incompatible with scikit-learn estimators (which assume all values in an array are numerical & have a meaning.)\n",
    "\n",
    "- You could discard rows or columns with missing values. This comes at the price of losing possibly valuable data. A better strategy is to infer missing data from known data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Univariate feature imputation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer)\n",
    "\n",
    "- `SimpleImputer` does *univariate* imputation (imputes values in a given dimension using only non-missing data in that dimension.)\n",
    "\n",
    "- Below: replace np.nan with the mean of the columns (axis 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4.         2.        ]\n",
      " [6.         3.66666667]\n",
      " [7.         6.        ]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.impute import SimpleImputer as SI\n",
    "\n",
    "imp = SI(missing_values=np.nan, strategy='mean')\n",
    "\n",
    "imp.fit([[1, 2], [np.nan, 3], [7, 6]])\n",
    "\n",
    "X = [[np.nan, 2], [6, np.nan], [7, 6]]\n",
    "\n",
    "print(imp.transform(X))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Sparse matrices are supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3. 2.]\n",
      " [6. 3.]\n",
      " [7. 6.]]\n"
     ]
    }
   ],
   "source": [
    "import scipy.sparse as sp\n",
    "X   = sp.csc_matrix([[1, 2], [0, -1], [8, 4]])\n",
    "imp = SI(missing_values=-1, \n",
    "         strategy='mean').fit(X)\n",
    "\n",
    "X_test = sp.csc_matrix([[-1,  2], \n",
    "                         [6, -1], \n",
    "                         [7, 6]])\n",
    "print(imp.transform(X_test).toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Category data (strings or pandas categoricals) is supported with `most_frequent` or `constant` options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['a' 'x']\n",
      " ['a' 'y']\n",
      " ['a' 'y']\n",
      " ['b' 'y']]\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame([[\"a\",    \"x\"],\n",
    "                   [np.nan, \"y\"],\n",
    "                   [\"a\",    np.nan],\n",
    "                   [\"b\",    \"y\"]], dtype=\"category\")\n",
    "\n",
    "imp = SI(strategy=\"most_frequent\")\n",
    "print(imp.fit_transform(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Multivariate Feature Imputation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html#sklearn.impute.IterativeImputer)\n",
    "\n",
    "- `IterativeImputer` models each feature's missing values as a function of other features. It uses a round-robin algorithm for `max_iter` iterations.\n",
    "\n",
    "- This estimator is considered experimental."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 6. 12.]\n",
      " [ 3.  6.]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.experimental import enable_iterative_imputer\n",
    "from sklearn.impute import IterativeImputer as II\n",
    "\n",
    "imp = II(max_iter=10, \n",
    "         random_state=0).fit([[1,      2], [3, 6], [4, 8], \n",
    "                              [np.nan, 3], [7, np.nan]])\n",
    "\n",
    "X_test = [[np.nan, 2], \n",
    "          [6, np.nan], \n",
    "          [np.nan, 6]]\n",
    "\n",
    "# the model learns that the second feature is double the first\n",
    "print(np.round(imp.transform(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: Iterative Imputing Variations](https://scikit-learn.org/stable/auto_examples/impute/plot_iterative_imputer_variants_comparison.html#sphx-glr-auto-examples-impute-plot-iterative-imputer-variants-comparison-py)\n",
    "\n",
    "- Goal: compare estimators to see which is best when evaluating Cal Housing dataset - with a single value randomly removed.\n",
    "\n",
    "- Estimator options:\n",
    "    - Bayes Ridge (regularized linear regression\n",
    "    - Decision Tree (non-linear regression)\n",
    "    - Extra Trees (similar to missForest in R)\n",
    "    - K Neighbors (compare to KNN approaches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "# To use this experimental feature, we need to explicitly ask for it:\n",
    "from sklearn.experimental import enable_iterative_imputer  # noqa\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.impute import IterativeImputer\n",
    "from sklearn.linear_model import BayesianRidge\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/impute/_iterative.py:685: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  warnings.warn(\"[IterativeImputer] Early stopping criterion not\"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/impute/_iterative.py:685: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  warnings.warn(\"[IterativeImputer] Early stopping criterion not\"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/impute/_iterative.py:685: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  warnings.warn(\"[IterativeImputer] Early stopping criterion not\"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/impute/_iterative.py:685: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  warnings.warn(\"[IterativeImputer] Early stopping criterion not\"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/impute/_iterative.py:685: ConvergenceWarning: [IterativeImputer] Early stopping criterion not reached.\n",
      "  warnings.warn(\"[IterativeImputer] Early stopping criterion not\"\n"
     ]
    }
   ],
   "source": [
    "N_SPLITS              = 5\n",
    "rng                   = np.random.RandomState(0)\n",
    "X_full, y_full        = fetch_california_housing(return_X_y=True)\n",
    "# ~2k samples is enough for the purpose of the example.\n",
    "# Remove the following two lines for a slower run with different error bars.\n",
    "X_full                = X_full[::10]\n",
    "y_full                = y_full[::10]\n",
    "n_samples, n_features = X_full.shape\n",
    "\n",
    "# Estimate score on entire dataset (no missing values)\n",
    "br_estimator          = BayesianRidge()\n",
    "score_full_data       = pd.DataFrame(\n",
    "    cross_val_score(\n",
    "        br_estimator, \n",
    "        X_full, y_full, \n",
    "        scoring='neg_mean_squared_error',\n",
    "        cv=N_SPLITS\n",
    "    ),\n",
    "    columns=['Full Data']\n",
    ")\n",
    "\n",
    "# Add a single missing value to each row\n",
    "X_missing        = X_full.copy()\n",
    "y_missing        = y_full\n",
    "missing_samples  = np.arange(n_samples)\n",
    "missing_features = rng.choice(n_features, \n",
    "                              n_samples, \n",
    "                              replace=True)\n",
    "X_missing[missing_samples, \n",
    "          missing_features] = np.nan\n",
    "\n",
    "# Estimate score after imputation (mean and median strategies)\n",
    "score_simple_imputer = pd.DataFrame()\n",
    "for strategy in ('mean', 'median'):\n",
    "    estimator = make_pipeline(\n",
    "        SimpleImputer(missing_values=np.nan, \n",
    "                      strategy=strategy),\n",
    "        br_estimator\n",
    "    )\n",
    "    score_simple_imputer[strategy] = cross_val_score(\n",
    "        estimator, \n",
    "        X_missing, \n",
    "        y_missing, \n",
    "        scoring='neg_mean_squared_error',\n",
    "        cv=N_SPLITS\n",
    "    )\n",
    "\n",
    "# Estimate the score after iterative imputation of the missing values\n",
    "# with different estimators\n",
    "estimators = [\n",
    "    BayesianRidge(),\n",
    "    DecisionTreeRegressor(max_features='sqrt', \n",
    "                          random_state=0),\n",
    "    ExtraTreesRegressor(n_estimators=10, \n",
    "                        random_state=0),\n",
    "    KNeighborsRegressor(n_neighbors=15)]\n",
    "\n",
    "score_iterative_imputer = pd.DataFrame()\n",
    "for impute_estimator in estimators:\n",
    "    estimator = make_pipeline(\n",
    "        IterativeImputer(random_state=0, \n",
    "                         estimator=impute_estimator,\n",
    "                         max_iter=1000,),\n",
    "        br_estimator)\n",
    "    \n",
    "    score_iterative_imputer[impute_estimator.__class__.__name__] = \\\n",
    "        cross_val_score(\n",
    "            estimator, X_missing, y_missing, scoring='neg_mean_squared_error',\n",
    "            cv=N_SPLITS)\n",
    "\n",
    "scores = pd.concat(\n",
    "    [score_full_data, \n",
    "     score_simple_imputer, \n",
    "     score_iterative_imputer],\n",
    "    keys=['Original', \n",
    "          'SimpleImputer', \n",
    "          'IterativeImputer'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(13, 6))\n",
    "means = -scores.mean()\n",
    "errors = scores.std()\n",
    "means.plot.barh(xerr=errors, ax=ax)\n",
    "ax.set_title('California Housing Regression with Different Imputation Methods')\n",
    "ax.set_xlabel('MSE (smaller is better)')\n",
    "ax.set_yticks(np.arange(means.shape[0]))\n",
    "ax.set_yticklabels([\" w/ \".join(label) for label in means.index.tolist()])\n",
    "plt.tight_layout(pad=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple vs Single Imputation\n",
    "\n",
    "- Using multiple imputations to generate $m$ imputations for a single feature matrix is a best practice. Each imputation is put through the  analysis pipeline. The $m$ analysis results help you understand how results can vary due to the uncertainty caused by the missing values. This is called multiple imputation.\n",
    "\n",
    "- `IterativeImputer` is based on the R MICE package (Multivariate Imputation by Chained Equations), but returns a single imputation instead of multiple imputations. `IterativeImputer` can also be used for multiple imputations by applying it repeatedly to the same dataset with different random seeds when `sample_posterior=True`.\n",
    "\n",
    "- Note: calling `IterativeImputer`s `transform` method is not allowed to change the number of samples. Therefore multiple imputations cannot be achieved by a single call to transform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Nearest Neighbors Imputation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer)\n",
    "\n",
    "- `nan_euclidean_distances` (a euclidean distance metric that supports missing values) is used to find the NNs. Each missing feature is imputed using values from `n_neighbors` NN that have a value for the feature. \n",
    "\n",
    "- The feature of the neighbors are averaged uniformly or weighted by distance to each neighbor. \n",
    "\n",
    "- If a sample has more than one missing feature, the neighbors for that sample can be different depending on the feature being imputed. When the number of available neighbors is less than n_neighbors and there are no defined distances to the training set, the training set average for that feature is used during imputation. \n",
    "\n",
    "- If there is at least one neighbor with a defined distance, the weighted or unweighted average of the remaining neighbors will be used during imputation. If a feature is always missing in training, it is removed during transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1. , 2. , 4. ],\n",
       "       [3. , 4. , 3. ],\n",
       "       [5.5, 6. , 5. ],\n",
       "       [8. , 8. , 7. ]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.impute import KNNImputer\n",
    "nan = np.nan\n",
    "X = [[1, 2, nan], [3, 4, 3], [nan, 6, 5], [8, 8, 7]]\n",
    "imputer = KNNImputer(n_neighbors=2, weights=\"uniform\")\n",
    "imputer.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Marking Imputed Values](https://scikit-learn.org/stable/modules/generated/sklearn.impute.MissingIndicator.html#sklearn.impute.MissingIndicator)\n",
    "\n",
    "- `MissingIndicator` transforms a dataset into a binary matrix which indicates the presence of missing values.\n",
    "\n",
    "- `SimpleImputer` and `IterativeImputer` have an `add_indicator` option, False by default, which can stack the missing data matrix with imputer's output.\n",
    "\n",
    "- Nan is the usual placeholder value. `missing_values` accepts other values, such as integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ True,  True, False],\n",
       "       [False,  True,  True],\n",
       "       [False,  True, False]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import MissingIndicator as MI\n",
    "X = np.array([[-1, -1, 1, 3],\n",
    "              [4, -1, 0, -1],\n",
    "              [8, -1, 1, 0]])\n",
    "\n",
    "indicator = MI(missing_values=-1)\n",
    "mask_missing_values_only = indicator.fit_transform(X)\n",
    "mask_missing_values_only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `features` chooses the features for building the mask. `missing-only` is the default setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 3])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indicator.features_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ True  True False False]\n",
      " [False  True False  True]\n",
      " [False  True False False]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indicator = MI(missing_values=-1, features=\"all\")\n",
    "mask_all = indicator.fit_transform(X)\n",
    "print(mask_all)\n",
    "indicator.features_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When using `MissingIndicator` in a pipeline, be sure to use `FeatureUnion` or `ColumnTransformer` to add the indicators to the regular features.\n",
    "\n",
    "- Below: first load \"iris\" dataset & add missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.impute import SimpleImputer, MissingIndicator\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import FeatureUnion, make_pipeline\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "X, y    = load_iris(return_X_y=True)\n",
    "mask    = np.random.randint(0, 2, size=X.shape).astype(bool)\n",
    "X[mask] = np.nan\n",
    "\n",
    "X_train, X_test, y_train, _ = train_test_split(X, y, test_size=100,\n",
    "                                               random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Create a `FeatureUnion` & add indicators from `MissingIndicator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 8)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer = FeatureUnion(\n",
    "    transformer_list=[\n",
    "        ('features',   SimpleImputer(strategy='mean')),\n",
    "        ('indicators', MissingIndicator())])\n",
    "\n",
    "transformer = transformer.fit(X_train, y_train)\n",
    "results     = transformer.transform(X_test)\n",
    "results.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Wrap into a pipeline with a classifier to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100,)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = make_pipeline(transformer, \n",
    "                    DecisionTreeClassifier()).fit(X_train, \n",
    "                                                  y_train)\n",
    "results = clf.predict(X_test)\n",
    "results.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:working] *",
   "language": "python",
   "name": "conda-env-working-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}