33d_dummy_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Dummy Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier) and [Dummy Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html#sklearn.dummy.DummyRegressor) Metrics\n",
    "\n",
    "- When doing supervised learning, compare your estimator against a simple example as a sanity test. *DummyClassifier* provides several strategies for this.\n",
    "\n",
    "    - `stratified`: generates random predictions by respecting the training set class distribution.\n",
    "\n",
    "    - `most_frequent`: always predicts the most frequent label in the training set.\n",
    "\n",
    "    - `prior`: always predicts the class that maximizes the class prior (like most_frequent) and predict_proba returns the class prior.\n",
    "\n",
    "    - `uniform`: generates predictions uniformly at random.\n",
    "\n",
    "    - `constant` always predicts a constant user-specified label.\n",
    "\n",
    "- Note: the `predict` method completely ignores the input data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test unbalanced dataset\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split as TTS\n",
    "X, y = load_iris(return_X_y=True)\n",
    "y[y != 1] = -1\n",
    "X_train, X_test, y_train, y_test = TTS(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.631578947368421\n",
      "0.5789473684210527\n"
     ]
    }
   ],
   "source": [
    "# compare SVC & most_frequent accuracy\n",
    "from sklearn.dummy import DummyClassifier as DC\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "clf1 = SVC(kernel='linear', \n",
    "           C=1).fit(X_train, y_train)\n",
    "clf2 = DC(strategy='most_frequent', \n",
    "          random_state=0).fit(X_train, y_train)\n",
    "\n",
    "print(clf1.score(X_test, y_test))\n",
    "print(clf2.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SVC doesn’t do much better than a dummy classifier. Change the kernel and re-run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9473684210526315\n"
     ]
    }
   ],
   "source": [
    "clf3 = SVC(kernel='rbf', C=1).fit(X_train, y_train)\n",
    "print(clf3.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **DummyRegressor** also implements four rules of thumb for regression:\n",
    "\n",
    "    - `mean`: predicts the mean of the training targets.\n",
    "\n",
    "    - `median`: predicts the median of the training targets.\n",
    "\n",
    "    - `quantile`: predicts a user provided quantile of the training targets.\n",
    "\n",
    "    - `constant`: predicts a constant user-specified value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5. 5. 5. 5.] 0.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.dummy import DummyRegressor as DR\n",
    "\n",
    "X = np.array([1.0, 2.0, 3.0, 4.0])\n",
    "y = np.array([2.0, 3.0, 5.0, 10.0])\n",
    "\n",
    "dummy_regr = DR(strategy=\"mean\").fit(X, y)\n",
    "\n",
    "print(dummy_regr.predict(X),\n",
    "      dummy_regr.score(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:working] *",
   "language": "python",
   "name": "conda-env-working-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}