32_hyperparameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Hypermarameter Tuning](https://scikit-learn.org/stable/modules/grid_search.html)\n",
    "\n",
    "- Hyper-parameters are parameters that are not directly learnt within estimators. In scikit-learn they are passed as arguments to the constructor of the estimator class. Examples include `C`, `kernel` and `gamma` for *Support Vector Classifier*, `alpha` for *Lasso*, etc.\n",
    "\n",
    "- Use `estimator.get_params()` to get the names & current values of all parameters in an estimator.\n",
    "\n",
    "- Scikit-Learn has two approaches to parameter search: **GridSearchCV** exhaustively considers all combinations; **RandomizedSearchCV** samples  candidates from a parameter space with a specified distribution. \n",
    "\n",
    "- Both have *successive halving counterparts* **HalvingGridSearchCV** and **HalvingRandomSearchCV**, which can be much faster.\n",
    "\n",
    "- A small subset of those parameters can have a large impact while others can be left to their default values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV)\n",
    "\n",
    "- Exhaustively generates candidates from a grid of values specified by `param_grid`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Randomized Param Optimization](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html#sklearn.model_selection.RandomizedSearchCV)\n",
    "\n",
    "- While *grid search* is the most popular method for parameter optimization, **RandomizedSearchCV** searches each setting by sampling a distribution over its possible values. This has two benefits:\n",
    "\n",
    "    - A budget can be chosen independent of the #parameters and possible values.\n",
    "    \n",
    "    - Adding parameters that do not influence the performance does not decrease efficiency.\n",
    "\n",
    "- Parameters are chosen with a *dictionary* plus a *computation budget*, (a #sampled candidates or sampling iterations) with `n_iter`. \n",
    "\n",
    "- Supply either a distribution or list of choices (which will be sampled uniformly): `{'C': scipy.stats.expon(scale=100), 'gamma': scipy.stats.expon(scale=.1),\n",
    "  'kernel': ['rbf'], 'class_weight':['balanced', None]}`\n",
    "  \n",
    "- Specify a continuous distribution to use randomization with continuous variables such as `C`.\n",
    "\n",
    "- A continuous log-uniform random variable is available through `loguniform`. For example, *loguniform(1, 100)* can be used instead of [1, 10, 100] or np.logspace(0, 2, num=1000). This is an alias to *SciPy’s stats.reciprocal*.\n",
    "\n",
    "- Mirroring the example above in grid search, we can specify a continuous random variable that is log-uniformly distributed between 1e0 and 1e3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': <scipy.stats._distn_infrastructure.rv_frozen at 0x7fd48b999160>,\n",
       " 'gamma': <scipy.stats._distn_infrastructure.rv_frozen at 0x7fd48b95eb20>,\n",
       " 'kernel': ['rbf'],\n",
       " 'class_weight': ['balanced', None]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils.fixes import loguniform\n",
    "{'C': loguniform(1e0, 1e3),\n",
    " 'gamma': loguniform(1e-4, 1e-3),\n",
    " 'kernel': ['rbf'],\n",
    " 'class_weight':['balanced', None]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: Grid Search with Cross Validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html#sphx-glr-auto-examples-model-selection-plot-grid-search-digits-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split as TTS\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report as CR\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To apply an classifier on this data, we need to flatten the image, to\n",
    "# turn the data in a (samples, feature) matrix\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "n_samples = len(digits.images)\n",
    "X,y = digits.images.reshape((n_samples, -1)), digits.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = TTS(\n",
    "    X, y, test_size=0.5, random_state=0)\n",
    "\n",
    "tuned_parameters = [{'kernel': ['rbf'], \n",
    "                     'gamma' : [1e-3, 1e-4],\n",
    "                     'C'     : [1, 10, 100, 1000]},\n",
    "                    {'kernel': ['linear'], \n",
    "                     'C'     : [1, 10, 100, 1000]}]\n",
    "\n",
    "scores = ['precision', 'recall']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best params:\t {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "Grid scores:\t\n",
      "0.986 (+/-0.016) for {'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.959 (+/-0.028) for {'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.988 (+/-0.017) for {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.982 (+/-0.026) for {'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.988 (+/-0.017) for {'C': 100, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.983 (+/-0.026) for {'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.988 (+/-0.017) for {'C': 1000, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.983 (+/-0.026) for {'C': 1000, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.974 (+/-0.012) for {'C': 1, 'kernel': 'linear'}\n",
      "0.974 (+/-0.012) for {'C': 10, 'kernel': 'linear'}\n",
      "0.974 (+/-0.012) for {'C': 100, 'kernel': 'linear'}\n",
      "0.974 (+/-0.012) for {'C': 1000, 'kernel': 'linear'}\n",
      "Detailed classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        89\n",
      "           1       0.97      1.00      0.98        90\n",
      "           2       0.99      0.98      0.98        92\n",
      "           3       1.00      0.99      0.99        93\n",
      "           4       1.00      1.00      1.00        76\n",
      "           5       0.99      0.98      0.99       108\n",
      "           6       0.99      1.00      0.99        89\n",
      "           7       0.99      1.00      0.99        78\n",
      "           8       1.00      0.98      0.99        92\n",
      "           9       0.99      0.99      0.99        92\n",
      "\n",
      "    accuracy                           0.99       899\n",
      "   macro avg       0.99      0.99      0.99       899\n",
      "weighted avg       0.99      0.99      0.99       899\n",
      "\n",
      "best params:\t {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "Grid scores:\t\n",
      "0.986 (+/-0.019) for {'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.957 (+/-0.028) for {'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.987 (+/-0.019) for {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.981 (+/-0.028) for {'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.987 (+/-0.019) for {'C': 100, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.982 (+/-0.026) for {'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.987 (+/-0.019) for {'C': 1000, 'gamma': 0.001, 'kernel': 'rbf'}\n",
      "0.982 (+/-0.026) for {'C': 1000, 'gamma': 0.0001, 'kernel': 'rbf'}\n",
      "0.971 (+/-0.010) for {'C': 1, 'kernel': 'linear'}\n",
      "0.971 (+/-0.010) for {'C': 10, 'kernel': 'linear'}\n",
      "0.971 (+/-0.010) for {'C': 100, 'kernel': 'linear'}\n",
      "0.971 (+/-0.010) for {'C': 1000, 'kernel': 'linear'}\n",
      "Detailed classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        89\n",
      "           1       0.97      1.00      0.98        90\n",
      "           2       0.99      0.98      0.98        92\n",
      "           3       1.00      0.99      0.99        93\n",
      "           4       1.00      1.00      1.00        76\n",
      "           5       0.99      0.98      0.99       108\n",
      "           6       0.99      1.00      0.99        89\n",
      "           7       0.99      1.00      0.99        78\n",
      "           8       1.00      0.98      0.99        92\n",
      "           9       0.99      0.99      0.99        92\n",
      "\n",
      "    accuracy                           0.99       899\n",
      "   macro avg       0.99      0.99      0.99       899\n",
      "weighted avg       0.99      0.99      0.99       899\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for score in scores:\n",
    "    clf = GridSearchCV(\n",
    "        SVC(), tuned_parameters, scoring='%s_macro' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"best params:\\t\",clf.best_params_)\n",
    "\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    print(\"Grid scores:\\t\")\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    y_true, y_pred = y_test, clf.predict(X_test)\n",
    "    print(CR(y_true, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: PCA Complexity vs Cross-Validated Accuracy](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_refit_callable.html#sphx-glr-auto-examples-model-selection-plot-grid-search-refit-callable-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.svm import LinearSVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lower_bound(cv_results):\n",
    "    \"\"\"\n",
    "    Calculate the lower bound within 1 standard deviation\n",
    "    of the best `mean_test_scores`.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    cv_results : dict of numpy(masked) ndarrays\n",
    "        See attribute cv_results_ of `GridSearchCV`\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    float\n",
    "        Lower bound within 1 standard deviation of the\n",
    "        best `mean_test_score`.\n",
    "    \"\"\"\n",
    "    best_score_idx = np.argmax(cv_results['mean_test_score'])\n",
    "\n",
    "    return (cv_results['mean_test_score'][best_score_idx]\n",
    "            - cv_results['std_test_score'][best_score_idx])\n",
    "\n",
    "# params: cv_results (dict of numpy maskset naarrays)\n",
    "# returns: model index with fewest PCA components with test score <1 stdev \n",
    "#          of best `mean_test_score\n",
    "\n",
    "def best_low_complexity(cv_results):\n",
    "\n",
    "    threshold     = lower_bound(cv_results)\n",
    "    candidate_idx = np.flatnonzero(cv_results['mean_test_score'] >= threshold)\n",
    "    best_idx      = candidate_idx[cv_results['param_reduce_dim__n_components']\n",
    "                                  [candidate_idx].argmin()]\n",
    "    return best_idx\n",
    "\n",
    "\n",
    "pipe = Pipeline([\n",
    "        ('reduce_dim', PCA(random_state=42)),\n",
    "        ('classify',   LinearSVC(random_state=42, C=0.01)),\n",
    "])\n",
    "\n",
    "param_grid = {\n",
    "    'reduce_dim__n_components': [6, 8, 10, 12, 14]\n",
    "}\n",
    "\n",
    "X, y = load_digits(return_X_y=True)\n",
    "\n",
    "grid = GridSearchCV(pipe, cv=10, n_jobs=1, param_grid=param_grid,\n",
    "                    scoring='accuracy', refit=best_low_complexity).fit(X, y)\n",
    "\n",
    "n_components = grid.cv_results_['param_reduce_dim__n_components']\n",
    "test_scores = grid.cv_results_['mean_test_score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best_index_ is 2\n",
      "The n_components selected is 10\n",
      "The corresponding accuracy score is 0.88\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyWklEQVR4nO3deXwV5dn/8c+XEAgBBAQsIhSw7lJBRCtaFR+toiKotVrXYl1aW0Wfij+XVqt207r3casr1rXUgqIoahXccGERF0SEAkpYBKIsIazh+v0xd/Dk5ORkSHLOMTnX+/U6r8zMPWfONfdM5pr1HpkZzjnn8lezXAfgnHMutzwROOdcnvNE4Jxzec4TgXPO5TlPBM45l+c8ETjnXJ7zRJCCpJ6STFLzXMeSDZJGSvpjzHHnSzo80zHVlaRrJD3aANOZIWlg/SNqWJKGSXoz13FkkqSBkkoS+mtcFsnjZiE2k7RTtn4vW5psIggbrLWSyiR9LWmcpO65jss1Dma2p5lNhIZLLq5uEpdFfeRDEq2rJpsIgmPNrA2wPfAl8H85jse5bwVJBbmOwW2dTC6zpp4IADCzdcBTwB6VwyQdI+l9SaskLZB0TU3fl3SWpJmSVkuaK+kXCWUDJZVIukTSUkmLJZ2VUN5K0s2SPpe0UtKbklqFsv0lTZK0QtIH6U5FhCOcSyV9KGmNpAckfUfSCyGu/0jqkDD+kHBIvULSREm7J5TtLWla+N4/gaKk3xosaXr47iRJe8Wp51rmNV08sect4bTdeZIWhfoekSamlHUs6QBJyyuPEiX1CUeOuyXEdLikQcCVwMnh6PIDST+RNDXpd34j6ZkaYqjP+tNR0tiwnr4HfK+WZfDDhPldIGlYGD5S0t2Snpe0BjhU0u5hWawIy2ZIwnSOlvRJiHlhZR1L6iTpufCdryS9IanadkTSZZKeShp2u6S/1VYnKaa15XRkWMdGhmX1CbBv0riXS/pvmO4nko4Pw3cH7gEGhOW4IgxvKekmSV9I+lLSPZXrbCi/NCyTRZJ+XkvdDwvzslrSPEmnJZSdmzC/n0jqVxlXmmWQapl1lfRvScvCbwxPF1NsZtYkP8B84PDQXQw8DPwjoXwg8H2iZLgX0RHDcaGsJ2BA89B/DNE/oIBDgHKgX8J0NgHXAYXA0aG8Qyi/E5gI7AAUAAcALUN/aRi/GfCj0N85zfy8A3wnfHcpMA3Ym2hD/irw+zDuLsCaMM1C4P8Bc4AW4fM58L+h7ERgI/DH8N29w7R/EOL9Wfjtlsn1miLGmua1xnjqMG+Vy+YJoHVYhssSlvU1wKOhO20dA38K024FfARcUMP6s2Waob8l8BWwe8Kw94Ef11Av9Vl/ngRGhXntDSwE3qzhd3oAq4FTwrQ6An1D2UhgJXBgqIu2YRlcGdaJ/wnf3TWMvxg4KHR3SIj3L0Qb1MLwOQhQDbGUA21Df0GY5v4x66SkhmVxPfAGsC3QHfg4adyfAF3DPJ5MtN5tH8qGJdcdcCswNkyvLfAs8JdQNohou9A71P/jROveTinmtzWwKqH+tgf2TIhpIVHSErBTqJ/CWpZB8jIrBqYCV4fxdwTmAkfWe3uZ6w12pj5h5SkDVhBt6BYB308z/m3ArUkbm+Y1jPs0cFHCSrs2cVyiDdn+YeGtBfqkmMZlwCNJw14EfpZmfk5L6P83cHdC/4XA06H7KmBUQlmzsCIOBA4OdaGE8kl8kwjuBv6Q9NuzgEMS4qiWCGqZ1xrjqcO8VS6b3RLK/wo8ELqv4ZtEkLaOif4RpxIlgfFJdbJlPklKBAn19KfQvSfwNSFZxlg3464/BUTrbuK8/pmaE8EVwJgaykZSdUfoIGAJ0Cxh2BPANaH7C+AXwDZJ07kOeIYUG8MUv/kmcGbo/hHw362ok5oSwVxgUELZeYnjppjudGBo6B6WWHdEG+U1wPcShg0A5oXuB4HrE8p2IX0iWAH8GGiVYp27KMV3alsGycvsB8AXKZb5Q3HWu3Sfpn5q6Dgza0+0V3kB8JqkLgCSfiBpQjjEWgn8EuiUaiKSjpL0TjgUXkG015Y4bqmZbUroLwfahHGKgP+mmGwP4CfhkHBFmO4PifYkavJlQvfaFP1tQndXor1+AMxsM7CAaA+5K7DQwloUfJ7Q3QO4JCmu7uF76aSb13TxbO28VVqQFH+q+NLWsZltJPpn6w3cnFQntXkYOFWSgDOIEt36VCPWY/3pDDRPMa816U7q+q+UOJ2uwIKwLBKnXblMfhzi/FzSa5IGhOE3Eu3FvhROg1we5vG0cMqlTNILYdzHiY5OAE4N/YTxa6uTmnQlTX1IOlPfnNZcQbRsa5puZ8JedsL448PwWn8rkZmtIToC+SWwWNHNKbuF4pqWS23LgKTf7wF0TVqfryQ6kq6Xpp4IADCzCjMbDVQQbQggWinHAt3NrB3R4a6SvyupJdEe6k3Ad0JieT7VuCksB9aR+rzuAqK91fYJn9Zmdv3WzV1Ki4hWGgDCxqo70V74YmCHMKzSd5Pi+lNSXMVm9kQtv5luXtPFU1eJd4B9N/xGsrR1LGkH4PfAQ8DNYVmnUi1BmNk7wAaivbpTgUdSfbGe688yotNGyfNakwWkv4aQOB+LgO5J5/e/S1gmZjbZzIYC2xHtrY8Kw1eb2SVmtiMwBPiNpMPM7DEzaxM+R4Xp/QsYKKkbcDwhEdSzThZTQ31I6gHcR7TT1zFM9+OE6SYvx+VEOxl7Jqwf7Sy6wSTtb6ViZi+a2Y+IdjQ+DbFAzcsl7TJIEfMCoqOVxPW5rZkdnS6uOPIiESgylOhc58wwuC3wlZmtk7Qf0T9zKi2IzgkvAzZJOgo4Is7vhkz/IHBLuMhTIGlA+Ed4FDhW0pFheJGiC4fd6j6nW4wCjpF0mKRC4BJgPdEpoLeJNi7DJRVKOgHYL+G79wG/DEdMktRa0YX1tvWY13Tx1NVVkool7QmcBfwzxTg11nFIRiOBB4Czif7p/1DDb30J9FT1i6L/AO4ANppZTbcl1mf9qQBGA9eEed2D6JpNTR4DDpd0kqTmii40961h3HeJjjz+X1gPBgLHAk9KahH28NuFo6ZVwGbYciPBTqH+VhLtXG1O9QNmtozomtFDRBuwyv+9OtcJ0bp0haQO4X/lwoSy1kQbzmUh1rOIjggqfQl0k9QixLeZaH2/VdJ24Ts7SDoy4beGSdpDUjHRTkNKim5uGCqpNdG6XcY39XI/MELSPuF/aqeQtGpcBjX8zHvAakUX4luFdbq3pH1rGD+2pp4InpVURrQi/4no3PCMUPYr4DpJq4kuvoxKNQEzWw0MD+VfEyWMsVsRwwiic9CTiS4w3kB0TnABMJTo0G4ZUba/lAZYJmY2Czid6HbZ5UQr17FmtsHMNgAnEJ0v/YrocHZ0wnenAOcSbeC+JjoNMCzmT9c0rzXGU4/ZfC3E9gpwk5m9lDxCLXU8nGhv96pwSugs4CxJB6X4rX+Fv6WSpiUMf4RoQ1PjMwYNsP5cQHSaaAlR4noozW99QXSK5RKi+p8O9Klh3A1Ey+EoomVyF9H5/E/DKGcA8yWtIjrdUXkHzM7Af4g2dG8Dd5nZhDTxPw4cTsJpoXrWybVEp0/mAS+RcCRmZp8AN4e4viS6keCthO++CswAlkhaHoZdRrQevRPm9T/ArmF6LxBdO3w1jPNqmriaAb8h2sv/iugC+PlhOv8i2v48TnQx+Glg2xjLoIqwYzAY6BvmfzlRkmmXJq5YtHWnRZ3LLUk9if4JCpPOq+cillZEF3b7mdnsXMbiXH009SMC5zLpfGCyJwHX2GUsEUh6UNEDMh/XUC5Jf5M0R9GDRP0yFYtzDU3SfOAiotMwzjVqmTwiGEn0QEZNjiI637gz0b3Ad2cwFtdEmNl8M1OuTwuZWU8z62Fm7+cyDucaQsYSgZm9TnTRpCZDiR6WsHArXntJ6e6hd845lwG5bGZ5B6o+LFEShi1OHlHSeURHDbRu3Xqf3XbbLXkU55xzaUydOnW5mXVOVdYo2ts3s3uBewH69+9vU6ZMyXFEzjnXuEiq8cnoXN41tJCqT+11o35PmjrnnKuDXCaCscCZ4e6h/YGVZlbttJBzzrnMytipIUlPELUi2EnRq+R+T9TaI2Z2D1HbIkcTPbFXTvRkp3POuSzLWCIws1NqKTfg1w3xWxs3bqSkpIR169Y1xOTct1RRURHdunWjsLAw16E416Q0iovFtSkpKaFt27b07NkTKU4Dhq6xMTNKS0spKSmhV69euQ7HuSalSTQxsW7dOjp27OhJoAmTRMeOHf2oz7kMaBKJAPAkkAd8GTuXGU0mETjnnKsbTwQNpKCggL59+9KnTx/69evHpEl1e+fKbbfdRnl5eQNH55xzNfNE0EBatWrF9OnT+eCDD/jLX/7CFVdcUafpZCsRmBmbN6d8sZRzLs94IsiAVatW0aFDhy39N954I/vuuy977bUXv/999La7NWvWcMwxx9CnTx969+7NP//5T/72t7+xaNEiDj30UA499NBq07388svZY4892GuvvRgxYgQAX375Jccffzx9+vShT58+W45EbrnlFnr37k3v3r257bbbAJg/fz677rorZ555Jr1792bBggUpY3PO5ZcmcftosvffH1ht2HbbncQOO/yKiopyPvyw+rueu3QZxvbbD2PDhuXMmHFilbK9955Y62+uXbuWvn37sm7dOhYvXsyrr0ZvtXvppZeYPXs27733HmbGkCFDeP3111m2bBldu3Zl3LhxAKxcuZJ27dpxyy23MGHCBDp16lRl+qWlpYwZM4ZPP/0USaxYsQKA4cOHc8ghhzBmzBgqKiooKytj6tSpPPTQQ7z77ruYGT/4wQ845JBD6NChA7Nnz+bhhx9m//33rzG2gw8+OEYtO+eaCj8iaCCVp4Y+/fRTxo8fz5lnnomZ8dJLL/HSSy+x9957069fPz799FNmz57N97//fV5++WUuu+wy3njjDdq1S//a0Xbt2lFUVMTZZ5/N6NGjKS4uBuDVV1/l/PPPB6LrFO3atePNN9/k+OOPp3Xr1rRp04YTTjiBN954A4AePXqw//77A9QYm3MuvzTJI4J0e/AFBcVpy1u06BTrCCCdAQMGsHz5cpYtW4aZccUVV/CLX/yi2njTpk3j+eef53e/+x2HHXYYV199dY3TbN68Oe+99x6vvPIKTz31FHfccceWo46t0bp16y3d6WJzzuUPPyLIgE8//ZSKigo6duzIkUceyYMPPkhZWRkACxcuZOnSpSxatIji4mJOP/10Lr30UqZNmwZA27ZtWb16dbVplpWVsXLlSo4++mhuvfVWPvjgAwAOO+ww7r47erlbRUUFK1eu5KCDDuLpp5+mvLycNWvWMGbMGA466KBq06wpNudcfmmSRwS5UHmNAKI97YcffpiCggKOOOIIZs6cyYABAwBo06YNjz76KHPmzOHSSy+lWbNmFBYWbtmYn3feeQwaNIiuXbsyYcKELdNfvXo1Q4cOZd26dZgZt9xyCwC333475513Hg888AAFBQXcfffdDBgwgGHDhrHffvsBcM4557D33nszf/78KjHXFNt2222Xyapyzn3LKGr7rfFI9WKamTNnsvvuu+coIpdNvqydqxtJU82sf6qyRndEUF4+q9pdQc2b/xUAswrWrp1T7TvNm3ekRYtObN68kXXr5lYrLyzsTGHhtmzevIF16+alKP8OhYXtqahYx/r11V/y06LF9jRvvg0VFeWsX78gRfkONG/ehk2bytiwofq7d1q27E5BQTGbNq1iw4bqr2Ro2bIHBQVFbNy4go0bv6xWXlTUi2bNWrBx41ds3LgsRfmONGtWyIYNy9m0qbRaeatWOyEVsGHDUjZt+rpaeXHxrgCsX7+EioqVSaWiuHiXUL6Iioqqp7WkAlq12imUl1BRsSapvJBWrXYEYN26L9i8eW2V8mbNWlJU1DOUz2fDhiW8//75W8rbtOnLzjvfBsAnn5zO+vUlVb7frt0AdtzxLwB8/PGP2bix6vx36HAYPXteBcCHHx5FRUXV3+/YcTDf/W50q25D340GsMMO57Pddiezbt0CZs48o1p59+6X0KnTsZSXz2LWrOrXcnr0+B3bbns4q1dPZ86ci6uV77jjn2nX7gBWrpzE3LlXVivfaafbaNu2L1999R8+//yP1cp33fXvFBfvyvLlz7Jgwc3Vynff/RGKirqzdOk/Wbjw7mrle+75FC1adGLx4pEsWTKyWvleez1PQUExCxfexdKlo6qVV16v++KLmygtfa5KWUFBK/ba6wUA5s//A19//UqV8sLCjvTu/W8A5s69gpUr365S3rJlN/bY41EAZs++mLKy6VXKi4t3Yddd7wVg1qzzKC//rEp5Y1/3EjW6ROCccw1Jgl//GnbaqerwkhK4OeS+Sy6Bbt2qls+ZA3feGXVfeSV0Tnob8IwZcP/9Ufe118I221QtnzYNHnkk6r7+emjZsmr522/DqJAbb701+nvIIVs3b3H5qSHXqPiydg2tMbVlWJ/NdZM6NeScaziNZSPYyPZXGx2/fdQ55/KcJwLnnMtznggaSL42Qz1o0CDat2/P4MGDt+p7EydOTFtHbdq0qW9ozrmYPBE0kHxthvrSSy/lkcpbH7ZCbYnAOZc9je5icWlpKSNHjqwyrPKJ3s2bN/PVV19V+05xcTHFxcVUVFTw9dfV75Nv3bo1rVq1YtOmTVta9UzUpk0bioqK2LhxIytXJt9HHzULAWwpLykpoXXr1ixfvhyA++67j9GjR7Nu3ToGDRrEZZddxpo1azjnnHNYvHgxZsbVV1/NwoULWbRoEQcffDDbbrstTz/9NBA1OFdYWMiIESN49tlnad68OQMHDuTaa69l6dKlXHnllcybNw8z4/rrr2e//fbj7rvv5vHHHwfg3HPP5Te/+Q0zZ87k2GOPZZ999uGDDz7giSee4JlnnmHcuHGsX7+eY445hksuuaTa/FW2hFpWVlbtncF9+/blo48+AqKnn9evX1+lvFmzZjz66KPcc889NGvWjJ133pmrrrqKu+66i4KCAh5++GHuuusuunXrxsknn0xZWRmDBg3CzFi+fDnNmzenffv2AKxYsYKysrIqy79Lly4MGjQIgNGjR7Nq1aoqv9+tWzcOP/xwAEaNGrUlyb72WlQ+d24vXn89uifvtNMeo7BwY5Xvf/bZLkyadAAAw4aNJNmMGXsyefK+FBZu5LTTHqtWPn16X6ZP70txcTknnVT9PvnJk/szY0ZvttlmJSecMKZa+aRJA5g1a1eWL1/Oc889V6384IMPZscdd2TJkiWMHz++Wvlhhx1G9+7dWbBgAa+88kq18i5dBrFkSRd23HEuBx/8erXyZ58dTGlpJ3bZZRYHHPB2tfLRo49n1ap27Lnnx+y775Rq5aNGnUR5eTF9+06nb9/p1cofe+w0Nm4sZN99J7PnnjOqlY8cOSzUwyQ++6zqffyFhYWcdtppALz22mvMm1f1GaDi4mJOOukkAP7zn/9QUlL1Pv9tttmGE044AYBBg8bTpcuSKuWlpR159tljATj22Gfp2LHqcwBLlnRh/Pho3TvhhNFss03VdW/Bgm688kq07p100iiKi6vu4NVl3Uvc9O25557su+++bNy4kcceq77u9e3bl759+1JeXs6oUdXXvUSNLhHEMWZM1X+ogoICvv/979OvXz82btxY7R+qoKCAfv360bt3b9auXVvtH+qMM6o/6JNs7dq19O/fn/Lycr788sstMUyYMIE5c+bw3nvvsX79eoYMGcKkSZMoLS2lS5cuPPHEE7Rr147y8nKKioq47bbbGDNmDB07dqwy/dLSUsaOHctbb72FpC0J6corr9zStlBZWRlLly7dspEfP348ZsYxxxzDoYceSlFREXPnzuWOO+6gf//+TJgwgblz5/LOO+8giWOOOYZJkyZxwAEHbHWdp3P99dczb9481q9fz7Jly2jXrh3Dhg2jdevWDB8+nA4dOjBkyBB+/vOfc+KJJ/LAAw806O8759Jrks8RJB8xQP2y57Bhw2qNq02bNlsab3v77bc555xz+Pjjj7n00kt56qmntuzVlpWVccUVV3DQQQdxxBFHcPLJJzN48OAtjcL17NmTKVOmVHsfwaZNm9hnn33YZ599GDx4MIMHD6ZFixZ07tyZkpISWiY8jXL77bdTWlrKddddB8BVV11F586dGTJkCIceeuiWPacRI0akjO3ss8+udX4TTZw4kZtuuinlHitE1xHatGnDcccdx3HHHUebNm245ppraNOmzZYX7HTs2JElS5ZQWFjIqlWr6Nq165b6TNRQzxE0ltsmIbO3TjaWevA6iPhzBFsh3Ya7sLAwbXlxcXGsDX86TakZ6jFjxnDttdcCcP/999O/f8r1KK1x48bx+uuv8+yzz/KnP/1py6mkZGpM/5HONSF+sTgDmlIz1McffzzTp09n+vTpdUoCmzdvZsGCBRx66KHccMMNrFy5krKysmrzeeCBB/Lkk08CpDxic85lTpM8IsiFfG2G+qCDDuLTTz+lrKyMbt268cADD3DkkUduKa+oqOD0009n5cqVmBnDhw+nffv2HHvssZx44ok888wz/N///R+33347p556KjfccANDhw7dusp3ztVLk7xG4Jouv0bQsBpLPXgdRPwagWswU6rf5fetVIczUc65OvBrBM45l+eaTCJobKe43NbzZexcZjSJRFBUVERpaalvKJowM6O0tJSioqJch+Jck9MkrhF069aNkpISli2r/ppGV11o+eJbb+bMqv1FRUV0S35NlHOu3ppEIigsLKRXr165DqPR2GOPXEcQjx/gOZcdTeLUkHPOubqrNRFImirp15I6bO3EJQ2SNEvSHEmXpyj/rqQJkt6X9KGko7f2N5xzztVPnCOCk4GuwGRJT0o6UjEahZFUANwJHAXsAZwiKfmkxO+AUWa2N/BT4K6tit4551y91ZoIzGyOmf0W2AV4HHgQ+FzStZK2TfPV/YA5ZjbXzDYATwLJbQcYsE3obgcs2toZcM45Vz+xrhFI2gu4GbgR+DfwE2AVkK75yx2ABQn9JWFYomuA0yWVAM8DF9bw++dJmiJpit8Z5JxzDSvWNQLgVmAysJeZDTezd83sZmBuPX//FGCkmXUDjgYekVQtJjO718z6m1n/zp071/MnnXPOJYpz++hPzCzlBt/MTkjzvYVA94T+bmFYorOBQWFab0sqAjoBS3HOOZcVcU4NnSOpfWWPpA6S/hjje5OBnSX1ktSC6GLw2KRxvgAOC9PdHSgC/NyPc85lUZxEcJSZrajsMbOviU7jpGVmm4ALgBeBmUR3B82QdJ2kIWG0S4BzJX0APAEMM28nwjnnsirOqaECSS3NbD2ApFZAy1q+A4CZPU90EThx2NUJ3Z8AB8YPt37ypd1x55zbGnESwWPAK5IeCv1nAQ9nLiTnnHPZVGsiMLMbJH1IOJcP/MHMXsxsWM4557IlVqNzZvYC8EKGY3HOOZcDcZ4j2F/SZEllkjZIqpC0KhvBOeecy7w4dw3dQfTg12ygFXAOURtCzjnnmoBYTUyY2RygwMwqzOwhwkNgzjnnGr841wjKwwNh0yX9FViMv8fAOeeajDgb9DPCeBcAa4iajfhxJoNyzjmXPWmPCMI7Bf5sZqcB64BrsxKVc865rEl7RGBmFUCPcGrIOedcExTnGsFc4C1JY4lODQFgZrdkLCrnnHNZEycR/Dd8mgFtMxuOc865bIvTxIRfF3DOuSas1kQgaQLRu4WrMLP/yUhEzjnnsirOqaERCd1FRLeObspMOM4557ItzqmhqUmD3pL0Xobicc45l2VxTg1tm9DbDNgHaJexiJxzzmVVnFNDU4muEYjolNA8opfOO+ecawLinBrqlY1AnHPO5Uac9xH8WlL7hP4Okn6V0aicc85lTZxG5841sxWVPWb2NXBuxiJyzjmXVXESQYEkVfaEhui87SHnnGsi4lwsHg/8U9LfQ/8vwjDnnHNNQJxEcBlwHnB+6H8ZuD9jETnnnMuqOImgFXCfmd0DW04NtQTKMxmYc8657IhzjeAVomRQqRXwn8yE45xzLtviJIIiMyur7AndxZkLyTnnXDbFSQRrJPWr7JG0D7A2cyE555zLpjjXCC4G/iVpEVEzE12AkzMZlHPOueyJ08TEZEm7AbuGQbPMbGNmw3LOOZctcY4IIEoCexC9j6CfJMzsH5kLyznnXLbEaYb698BAokTwPHAU8CbgicA555qAOBeLTwQOA5aY2VlAH/x9BM4512TESQRrzWwzsEnSNsBSoHtmw3LOOZctcRLBlNAM9X1EL6mZBrwdZ+KSBkmaJWmOpMtrGOckSZ9ImiHp8biBO+ecaxhx7hqqfPfAPZLGA9uY2Ye1fS80RXEn8COgBJgsaayZfZIwzs7AFcCBZva1pO3qMhPOOefqLu5dQwCY2fytGH0/YI6ZzQWQ9CQwFPgkYZxzgTvDOw4ws6VbE49zzrn6i3NqqK52ABYk9JeEYYl2AXaR9JakdyQNSjUhSedJmiJpyrJlyzIUrnPO5adMJoI4mgM7E92eegpwX+JrMSuZ2b1m1t/M+nfu3Dm7ETrnXBMX69RQON//ncTxzeyLWr62kKp3F3ULwxKVAO+GJ5XnSfqMKDFMjhOXc865+ovz8voLgS+JXkgzLnyeizHtycDOknpJagH8FBibNM7TREcDSOpEdKpobszYnXPONYA4RwQXAbuaWenWTNjMNkm6AHgRKAAeNLMZkq4DppjZ2FB2hKRPgArg0q39Heecc/UTJxEsAFbWZeJm9jxRsxSJw65O6DbgN+HjnHMuB+IkgrnAREnjgPWVA83sloxF5ZxzLmviJIIvwqdF+DjnnGtC4jxZfC2ApDahvyz9N5xzzjUmce4a6i3pfWAGMEPSVEl7Zj4055xz2RDngbJ7gd+YWQ8z6wFcQtQAnXPOuSYgTiJobWYTKnvMbCLQOmMROeecy6pYdw1Jugp4JPSfjj/05ZxzTUacI4KfA52B0eHTOQxzzjnXBMS5a+hrYHgWYnHOOZcDNSYCSbeZ2cWSngUsudzMhmQ0Muecc1mR7oig8prATdkIxDnnXG7UmAjMbGro7GtmtyeWSboIeC2TgTnnnMuOOBeLf5Zi2LAGjsM551yOpLtGcApwKtBLUuJ7BNoCX2U6MOecc9mR7hrBJGAx0Am4OWH4auDDTAblnHMue9JdI/gc+BwYkL1wnHPOZVucRuf2lzRZUpmkDZIqJK3KRnDOOecyL87F4juAU4DZQCvgHODOTAblnHMue+IkAsxsDlBgZhVm9hAwKLNhOeecy5Y4jc6VS2oBTJf0V6ILyLESiHPOuW+/OBv0M8J4FwBrgO7AjzMZlHPOueyJc0SwHNhgZuuAayUVAC0zG5ZzzrlsiXNE8ApQnNDfCvhPZsJxzjmXbXESQVHiC+tDd3Ga8Z1zzjUicRLBGkn9Knsk7QOszVxIzjnnsinONYKLgX9JWgQI6AKcnMmgnHPOZU+cN5RNlrQbsGsYNMvMNmY2LOecc9mSrvXR/zGzVyWdkFS0iyTMbHSGY3POOZcF6Y4IDgZeBY5NUWZEL7J3zjnXyKVLBF+Hvw+Y2ZvZCMY551z2pbtr6Kzw92/ZCMQ551xupDsimClpNtBVUuKLaASYme2V2dCcc85lQ7oX05wiqQvwIjAkeyE555zLprS3j5rZEqBPlmJxzjmXA+luHx1lZidJ+ojoLqEtRfipIeecazLSHRFcFP4OruvEJQ0CbgcKgPvN7Poaxvsx8BSwr5lNqevvOeec23o13jVkZotD53JgQXiZfUuiU0WLaptwaK76TuAoYA/gFEl7pBivLVHSeXero3fOOVdvcRqdex0okrQD8BLRi2pGxvjefsAcM5trZhuAJ4GhKcb7A3ADsC5WxM455xpUnEQgMysHTgDuMrOfAHvG+N4OwIKE/pIw7JsJR62adjezcWkDkM6TNEXSlGXLlsX4aeecc3HFSgSSBgCnAZUb7IL6/rCkZsAtwCW1jWtm95pZfzPr37lz5/r+tHPOuQRxEsHFwBXAGDObIWlHYEKM7y0ker9xpW5hWKW2QG9goqT5wP7AWEn9Y0zbOedcA4nTDPVrwGuwZS9+uZkNjzHtycDOknoRJYCfAqcmTHcl0KmyX9JEYITfNeScc9lV6xGBpMclbSOpNfAx8ImkS2v7npltAi4gejJ5JjAqHFFcJ8mfVHbOuW+JOG8o28PMVkk6DXgBuByYCtxY2xfN7Hng+aRhV9cw7sAYsTjnnGtgca4RFEoqBI4Dxoa3k1n6rzjnnGss4iSCvwPzgdbA65J6AKsyGZRzzrnsiXOx+G9UfSfB55IOzVxIzjnnsinONQIkHUP0EFlRwuDrMhKRc865rIpz19A9wMnAhUQtj/4E6JHhuJxzzmVJnGsEB5jZmcDXZnYtMADYJbNhOeecy5Y4iWBt+FsuqSuwEdg+cyE555zLpjjXCJ6T1J7ouYFpRLeO3p/JoJxzzmVPnLuG/hA6/y3pOaAoNA/hnHOuCUj3qsoT0pRhZqMzE5JzzrlsSndEcGyaMgM8ETjnXBNQYyIws7OyGYhzzrnciPMcwZ/DxeLK/g6S/pjRqJxzzmVNnNtHjzKzFZU9ZvY1cHTGInLOOZdVcRJBgaSWlT2SWgEt04zvnHOuEYnzHMFjwCuSHgr9ZwEPZy4k55xz2RTnOYIbJH0AHB4G/cHMXsxsWM4557IlVuujZjYeGJ/hWJxzzuVAnGsEzjnnmjBPBM45l+fiPEdwUZxhzjnnGqc4RwQ/SzFsWAPH4ZxzLkfSNTp3CnAq0EvS2ISitsBXmQ7MOedcdqS7a2gSsBjoBNycMHw18GEmg3LOOZc96Rqd+xz4nOjVlM4555qodKeG3jSzH0paTdTs9JYiwMxsm4xH55xzLuPSHRH8MPxtm71wnHPOZVutTxZL2jbF4NVmtjED8TjnnMuyOLePTgOWAZ8Bs0P3fEnTJO2TyeCcc85lXpxE8DJwtJl1MrOOwFHAc8CvgLsyGZxzzrnMi5MI9k9sbdTMXgIGmNk7+HsJnHOu0YvT+uhiSZcBT4b+k4EvJRUAmzMWmXPOuayIc0RwKtANeDp8vhuGFQAnZSow55xz2RHnxTTLgQtrKJ7TsOE455zLtnQPlN1mZhdLepaqD5QBYGZDapu4pEHA7URHD/eb2fVJ5b8BzgE2Ed2N9PPwRLNzzrksSXdE8Ej4e1NdJhyuIdwJ/AgoASZLGmtmnySM9j7Q38zKJZ0P/JXoGoRzzrksSfdk8dTw9zVJnUP3sq2Y9n7AHDObCyDpSWAosCURmNmEhPHfAU7fiuk755xrAGkvFku6RtJyYBbwmaRlkq6OOe0dgAUJ/SVhWE3OBl6oIY7zJE2RNGXZsq3JRc4552pTYyII5+8PBPY1s23NrAPwA+BASf/bkEFIOh3oD9yYqtzM7jWz/mbWv3Pnzg350845l/fSHRGcAZxiZvMqB4TTPKcDZ8aY9kKge0J/tzCsCkmHA78FhpjZ+jhBO+ecazjpEkFhuHW0inCdoDDGtCcDO0vqJakF8FMg8U1nSNob+DtRElgaP2znnHMNJV0i2FDHMgDMbBNwAfAiMBMYZWYzJF0nqfLW0xuBNsC/JE1PeiWmc865LEh3+2gfSatSDBdQFGfiZvY88HzSsKsTug+PMx3nnHOZk+720YJsBuKccy434rQ15JxzrgnzROCcc3nOE4FzzuU5TwTOOZfnPBE451ye80TgnHN5zhOBc87lOU8EzjmX5zwROOdcnvNE4Jxzec4TgXPO5TlPBM45l+c8ETjnXJ7zROCcc3nOE4FzzuU5TwTOOZfnPBE451ye80TgnHN5zhOBc87lOU8EzjmX5zwROOdcnvNE4Jxzec4TgXPO5TlPBM45l+c8ETjnXJ7zROCcc3nOE4FzzuU5TwTOOZfnPBE451ye80TgnHN5zhOBc87lOU8EzjmX5zwROOdcnvNE4JxzeS6jiUDSIEmzJM2RdHmK8paS/hnK35XUM5PxOOecqy5jiUBSAXAncBSwB3CKpD2SRjsb+NrMdgJuBW7IVDzOOedSy+QRwX7AHDOba2YbgCeBoUnjDAUeDt1PAYdJUgZjcs45l6R5Bqe9A7Agob8E+EFN45jZJkkrgY7A8sSRJJ0HnBd6yyTNykjEddeJpJjrqxGmQ6+DDNQBeD2A10GletZDj5oKMpkIGoyZ3Qvcm+s4aiJpipn1z3UcueR14HVQyeuh8dVBJk8NLQS6J/R3C8NSjiOpOdAOKM1gTM4555JkMhFMBnaW1EtSC+CnwNikccYCPwvdJwKvmpllMCbnnHNJMnZqKJzzvwB4ESgAHjSzGZKuA6aY2VjgAeARSXOAr4iSRWP0rT1tlUVeB14HlbweGlkdyHfAnXMuv/mTxc45l+c8ETjnXJ7zRFAPktpLekrSp5JmShqQ65hyQdL/Spoh6WNJT0gqynVMmSbpQUlLJX2cMGxbSS9Lmh3+dshljNlQQz3cGP4nPpQ0RlL7HIaYcanqIKHsEkkmqVMuYovLE0H93A6MN7PdgD7AzBzHk3WSdgCGA/3NrDfRjQGN9aL/1hgJDEoadjnwipntDLwS+pu6kVSvh5eB3ma2F/AZcEW2g8qykVSvAyR1B44Avsh2QFvLE0EdSWoHHEx05xNmtsHMVuQ0qNxpDrQKz4IUA4tyHE/GmdnrRHe6JUpsMuVh4LhsxpQLqerBzF4ys02h9x2iZ4iarBrWBYjaT/t/wLf+jhxPBHXXC1gGPCTpfUn3S2qd66CyzcwWAjcR7fUsBlaa2Uu5jSpnvmNmi0P3EuA7uQzmW+LnwAu5DiLbJA0FFprZB7mOJQ5PBHXXHOgH3G1mewNryI9TAVWE8+BDiRJjV6C1pNNzG1XuhQcjv/V7gpkk6bfAJuCxXMeSTZKKgSuBq3MdS1yeCOquBCgxs3dD/1NEiSHfHA7MM7NlZrYRGA0ckOOYcuVLSdsDhL9LcxxPzkgaBgwGTsvD1gK+R7Rj9IGk+USnxqZJ6pLTqNLwRFBHZrYEWCBp1zDoMOCTHIaUK18A+0sqDk2IH0YeXjQPEptM+RnwTA5jyRlJg4jOjQ8xs/Jcx5NtZvaRmW1nZj3NrCfRTmO/sM34VvJEUD8XAo9J+hDoC/w5t+FkXzgiegqYBnxEtE41qsfr60LSE8DbwK6SSiSdDVwP/EjSbKIjpetzGWM21FAPdwBtgZclTZd0T06DzLAa6qBR8SYmnHMuz/kRgXPO5TlPBM45l+c8ETjnXJ7zROCcc3nOE4FzzuU5TwSO0DrizQn9IyRd00DTHinpxIaYVi2/85PQAuyEpOE9Ja0NtzF+IukeSc1C2S6Sng+thU6TNErSdxK+e5ukhZXjN3WhNd1ffQviuEbSiFzHkU/yYgV3tVoPnPBtayo3NGIX19nAuWZ2aIqy/5pZX2AvYA/guNBU9jiiJkJ2NrN+wF1A5/DbzYDjgQXAIXWfi0alPZDzROCyzxOBg6g9mHuB/00uSN6jl1QW/g6U9JqkZyTNlXS9pNMkvSfpI0nfS5jM4ZKmSPpM0uDw/YLQbv3k0G79LxKm+4aksaR4UlvSKWH6H0u6IQy7Gvgh8ICkG2uaydAi5iRgJ+BU4G0zezahfKKZVbYpPxCYAdwNnJJqemEebgqxfCjpwjD8sNAQ4UehrfqWYfh8SX8JRydTJPWT9KKk/0r6ZcL8vy5pnKRZSUcw1ea9cplI+pOkDyS9U3lUI6mzpH+HOp4s6cAw/JoQ18Sw7IaHSV0PfC/Ed6Ok7UMs08NvHpSiDuZX7kBI6i9pYug+JHxveqiLtmH4pQnL/NqE6fw2rB9vArsm/47LMDPzT55/gDJgG2A+0A4YAVwTykYCJyaOG/4OBFYA2wMtgYXAtaHsIuC2hO+PJ9rp2Jnocfsi4Dzgd2GclsAUovZZBhI14NcrRZxdiZq06EzU6N+rwHGhbCLROxGSv9MT+Dh0FwOTgaOAW4CL0tTJfcAZoV4WAoUpxjmf6Knq5qF/2zBvC4BdwrB/ABeH7vnA+aH7VuBDoidwOwNfJtTrOmBHonc7vAycWMu8G3Bs6P5rQr0+DvwwdH8XmBm6ryFKiC2BTkApUJhYV2G8S4Dfhu4CoG2KOpgPdArd/YGJoftZ4MDQ3SbEfATRDofC+vAcUVPu+xA9lV4c6nsOMCLX/xf59PEjAgeAma0i2mgNr23cBJPNbLGZrQf+C1Q2P/0R0Ual0igz22xms4G5wG5EG4UzJU0H3gU6EiUKgPfMbF6K39uXaEOzzKK9+8eINiS1+V74nbeAcWaWtllkSS2Ao4GnQ728CxyZYtTDgb+HWDCzr4j2ZueZ2WdhnIeTYhwb/n4EvGtmq81sGbBe37zJ6z0zm2tmFcATREc76eZ9A9FGFWAq39T94cAdYd7HAttIahPKxpnZejNbTtQ4XqomsycDZym6XvR9M1udssJSewu4JRxttA8xHxE+7xM1SbIb0TI/CBhjZuWhvsfWME2XIVtzDtY1fbcR/YM+lDBsE+EUYjhF0SKhbH1C9+aE/s1UXbeS2zExor3CC83sxcQCSQOJjggaUuU1gkQzqPnc/5FE58s/kgTRnupavtnY1kdiHSXXX2WdpaqvdDZa2PUGKhKm0wzY38zWJY4c5inxtxO/882Pmr0u6WDgGGCkpFvM7B9Jo21ZP4iOhiq/e72kcUQJ9S1JRxIt87+Y2d+T4rm4lvlzGeZHBG6LsEc7iujCa6X5RIfuAEOITiFsrZ9IahauG+wIzAJeBM6XVAhb7uCp7cU+7wGHSOokqYDo3P1rdYgHotMmB0g6pnKApIMl9Q7TPce+aT2yF1FjcsVJ03gZ+IXCRW1J24Z56ylppzDOGXWIcT9JvULiPRl4k7rN+0tEDSNWzl/fWsZfTXSqqnL8HkSnrO4D7id1M+vz+Wb9+HHCd79nUSucNxAdWexGtMx/XnlUImkHSdsBrxNdwG8VriUcW0ucroF5InDJbiY6b1zpPqIN0AfAAOq2t/4F0YbsBeCXYQ/1fqKLwdMUvfT779RyhGrR278uByYAHwBTzaxOTT2b2Vqi9vIvVHT76CdEd8ysJnr/7LiEcdcQbYyTN1D3h3n7MNTPqWHezgL+Jekjoj39rW19czJRC54zgXlEp03qMu/Dgf7hwuwnwC/TjWxmpUR77x+Hi+4DidrUf58oId2e4mvXArdLmkJ0ZFHp4sqL6MBG4AWL3lz3OPB2qJuniK47TAP+GebrhTD/Lou89VHnvkXCqbERZjY4x6G4POJHBM45l+f8iMA55/KcHxE451ye80TgnHN5zhOBc87lOU8EzjmX5zwROOdcnvv/05Q78pBkRVkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.bar(n_components, test_scores, width=1.3, color='b')\n",
    "\n",
    "lower = lower_bound(grid.cv_results_)\n",
    "\n",
    "plt.axhline(np.max(test_scores), linestyle='--', color='y', label='Best score')\n",
    "plt.axhline(lower, linestyle='--', color='.5', label='Best score - 1 std')\n",
    "\n",
    "plt.title(\"Balance model complexity and cross-validated score\")\n",
    "plt.xlabel('Number of PCA components used')\n",
    "plt.ylabel('Digit classification accuracy')\n",
    "plt.xticks(n_components.tolist())\n",
    "plt.ylim((0, 1.0))\n",
    "plt.legend(loc='upper left')\n",
    "\n",
    "best_index_ = grid.best_index_\n",
    "\n",
    "print(\"The best_index_ is %d\" % best_index_)\n",
    "print(\"The n_components selected is %d\" % n_components[best_index_])\n",
    "print(\"The corresponding accuracy score is %.2f\"\n",
    "      % grid.cv_results_['mean_test_score'][best_index_])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: Random Search vs Grid Search](https://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html#sphx-glr-auto-examples-model-selection-plot-randomized-search-py)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from time import time\n",
    "import scipy.stats as stats\n",
    "from sklearn.utils.fixes import loguniform\n",
    "from sklearn.model_selection import GridSearchCV as GS\n",
    "from sklearn.model_selection import RandomizedSearchCV as RS\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.linear_model import SGDClassifier as SGDC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 24.49 seconds for 20 candidates parameter settings.\n",
      "Model with rank: 1\n",
      "Mean validation score: 0.920 (std: 0.031)\n",
      "Parameters: {'alpha': 0.0006108341940380009, 'average': False, 'l1_ratio': 0.7389206483086175}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.919 (std: 0.035)\n",
      "Parameters: {'alpha': 0.008490848548217928, 'average': False, 'l1_ratio': 0.3743915046152}\n",
      "\n",
      "Model with rank: 3\n",
      "Mean validation score: 0.918 (std: 0.036)\n",
      "Parameters: {'alpha': 0.0003484324373622794, 'average': False, 'l1_ratio': 0.7747484234866344}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n",
      "/home/bjpcjp/.local/lib/python3.8/site-packages/sklearn/linear_model/_stochastic_gradient.py:574: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\"Maximum number of iteration reached before \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV took 91.15 seconds for 100 candidate parameter settings.\n",
      "Model with rank: 1\n",
      "Mean validation score: 0.931 (std: 0.030)\n",
      "Parameters: {'alpha': 1.0, 'average': True, 'l1_ratio': 0.0}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.928 (std: 0.027)\n",
      "Parameters: {'alpha': 1.0, 'average': False, 'l1_ratio': 0.0}\n",
      "\n",
      "Model with rank: 3\n",
      "Mean validation score: 0.928 (std: 0.024)\n",
      "Parameters: {'alpha': 0.0001, 'average': True, 'l1_ratio': 0.2222222222222222}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "X, y = load_digits(return_X_y=True)\n",
    "\n",
    "clf = SGDC(loss='hinge', penalty='elasticnet', fit_intercept=True)\n",
    "\n",
    "# Utility function to report best scores\n",
    "def report(results, n_top=3):\n",
    "    for i in range(1, n_top + 1):\n",
    "        candidates = np.flatnonzero(results['rank_test_score'] == i)\n",
    "        for candidate in candidates:\n",
    "            print(\"Model with rank: {0}\".format(i))\n",
    "            print(\"Mean validation score: {0:.3f} (std: {1:.3f})\"\n",
    "                  .format(results['mean_test_score'][candidate],\n",
    "                          results['std_test_score'][candidate]))\n",
    "            print(\"Parameters: {0}\".format(results['params'][candidate]))\n",
    "            print(\"\")\n",
    "\n",
    "# specify parameters and distributions to sample from\n",
    "param_dist = {'average': [True, False],\n",
    "              'l1_ratio': stats.uniform(0, 1),\n",
    "              'alpha': loguniform(1e-4, 1e0)}\n",
    "\n",
    "# run randomized search\n",
    "n_iter_search = 20\n",
    "random_search = RS(clf, \n",
    "                   param_distributions=param_dist,\n",
    "                   n_iter=n_iter_search)\n",
    "\n",
    "start = time()\n",
    "random_search.fit(X, y)\n",
    "print(\"RandomizedSearchCV took %.2f seconds for %d candidates\"\n",
    "      \" parameter settings.\" % ((time() - start), n_iter_search))\n",
    "report(random_search.cv_results_)\n",
    "\n",
    "# use a full grid over all parameters\n",
    "param_grid = {'average': [True, False],\n",
    "              'l1_ratio': np.linspace(0, 1, num=10),\n",
    "              'alpha': np.power(10, np.arange(-4, 1, dtype=float))}\n",
    "\n",
    "# run grid search\n",
    "grid_search = GS(clf, param_grid=param_grid)\n",
    "start = time()\n",
    "grid_search.fit(X, y)\n",
    "\n",
    "print(\"GridSearchCV took %.2f seconds for %d candidate parameter settings.\"\n",
    "      % (time() - start, len(grid_search.cv_results_['params'])))\n",
    "report(grid_search.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Successive Halving for [Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.HalvingGridSearchCV.html#sklearn.model_selection.HalvingGridSearchCV) and [Random Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.HalvingRandomSearchCV.html#sklearn.model_selection.HalvingRandomSearchCV)\n",
    "\n",
    "- Successive halving (SH) is like a tournament among candidate parameter combinations. It is an iterative selection process where all parameter combinations are evaluated with a small amount of resources (typically training samples) at the first iteration. Only some of these candidates are selected for the next iteration, which will be allocated more resources. The resources can also be an arbitrary numeric parameter such as `n_estimators` in a random forest.\n",
    "\n",
    "- Below: only a subset of candidates ‘survive’ until the last iteration. They have consistently ranked among the top-scoring candidates across all iterations. Each iteration is allocated an increasing amount of resources per candidate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: Successive Halving Iterations](https://scikit-learn.org/stable/auto_examples/model_selection/plot_successive_halving_iterations.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import randint\n",
    "import numpy as np\n",
    "from sklearn.experimental import enable_halving_search_cv  # noqa\n",
    "from sklearn.model_selection import HalvingRandomSearchCV\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.RandomState(0)\n",
    "\n",
    "X, y = datasets.make_classification(n_samples=700, random_state=rng)\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=20, random_state=rng)\n",
    "\n",
    "param_dist = {\"max_depth\": [3, None],\n",
    "              \"max_features\": randint(1, 11),\n",
    "              \"min_samples_split\": randint(2, 11),\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "rsh = HalvingRandomSearchCV(\n",
    "    estimator=clf,\n",
    "    param_distributions=param_dist,\n",
    "    factor=2,\n",
    "    random_state=rng).fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEYCAYAAAAeWvJ8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABMS0lEQVR4nO2dd5wkVdm2r6fjzE7YnPMCC0sOS1Ik5ygGBBRBRcwv+hkwEwyYFVFMgIi+gAkVFQSEF0ElLSoISk4L7MLG2cmdnu+Pc3qntre7p2ame3p6+rnmV7+p1KfOXemuk0VVMQzDMIxaEal1BAzDMIzGxozIMAzDqClmRIZhGEZNMSMyDMMwaooZkWEYhlFTzIgMwzCMmmJGZNQcEXm1iDwhIl0i8toaxkNFZFs//30R+UyYfRsNEVngr1W0hnF4s4jcUqvjG5VFrB1RfSIiBwBfAXYCssB/gQ+q6v01jdgwEJHbgBtU9ZIax0OB7VT1yUrtKyKLgGeAuKpmKhLRMYaI3AH8TFUvr1L4ixjn57DRidU6AsbQEZF24A/Ae4BfAAngNUB/hY8TVdVsJcMswULgkVE4jjFERCRW7Zf/KN5nxlhFVW2qswlYDmwcZJ934lJJncB/gD39+mXAHcBG3Mv/xMBvrgK+B9wIdAOHA3OAXwNrcF+l/xPYfx9gBbAJeBn4xiDxeRJYD9wAzPHrnwJyQC/QBSSL/HY+cL2PwzrgO379NsDtft1a4H+BSYHfPQt8BHgI6AB+DjQFtn8UWAW8BLwdUGDbwLn4fMh9jwP+6c/DSuCCwO+e9/t2+Wl/v/7t/vpsAG4GFvr1AnwTeMWH929g5xLndI4/l+v9uX1nYH0vMCWw7x7+HMXLHd9vU+B9wBPAM0WOu8jvEwO+gEuR93l9+WuzA3Crj9tjwCmD3GdDOofAWcBfA/u8CrjfX+f7gVcFtt0BfA74G+55uAWY5rc1AT/D3UMb/W9n1voZb7Sp5hGwaRgXDdr9g/MT4BhgcsH2NwIvAnv7F9u2uFRH3L+wPolLRR3qH8zt/e+u8g/yq3HlhxOAB4DP+v2XAE8DR/n97wbO8POtwH4l4nuofwnuCSSBS4E7A9ufBQ4v8dso8CDu5dziXxwH+G3bAkf4MKcDdwLfKgj3PtyLeQruxftuv+1onHnu7MO9hhJGFGLfg4Fd/Dnb1e/7Wr9tkd83FojXSf46LMO9zD8N/N1vO8qf80n+2i0DZpc4N3cCl/lzsjvOqA/1227HG5Nf/irw/cGO77crzkSmAM1FjruFJtyL/uzA9hacmbzNh583wR1L3GdNwziHZ+GNyMdzA3CGP95pfnlqIH5PAUuBZr/8Jb/tXcDvcfd6FNgLaK/1M95oU80jYNMwL5x7iVwFvABkcF/GM/22m4Fzi/zmNcBqIBJYdy3+69OHd3Vg277A8wVhfAL4sZ+/E7gQ/3VZJq5XAF8JLLcCaWCRX36W0ka0P+4FGyt3DL/va4F/BpafBd4SWP4KAy/jK/MvI7+8lNJGVHbfIvH4FvBNP1/sJXoT8I7AcgTowX0sHAo8DuwXvE5FjjEflxJpC6y7GLjKz58N3O7nBWcMBw52fL+seEMrcewtNLG1Eb0JuKvgNz8Azi92nw3zHJ7FgBGdAdxX8Pu7gbMC8ft0YNt7gT/5+bcDfwd2rdSzadPQJ6s1V6eo6n9V9SxVnYf7Up+De3jBvaSeKvKzOcBKVc0F1j0HzA0srwzMLwTmiMjG/IRLTc3029+Beyk/KiL3i8jxJaI7xx8nH/cuXIpubon9g8wHntMi5RQiMlNErhORF0VkEy6LZVrBbqsD8z04E8zHKaj1OUpTdl8R2VdE/k9E1ohIB/DuIvEIshC4JHBO1+PMYq6q3g58B/gu8IqI/NCXCRaL03pV7SyIV/6c/hrYX0RmAwfisj/vGuz4gbCCeofKQmDfgvvmzcCsUuEP4xwG2eL+8hTe16Xug5/iPtyuE5GXROQrIhIPeVyjQpgRjQNU9VHcV+bOftVKXPlJIS8B80UkeN0X4LLxNgcXmF+JKyOYFJjaVPVYf9wnVPU0YAbwZeBXItJS4rgL8wt+n6kFxy3FSmCBiBSrWPNFH99dVLUdeAvuhRqGVTiTy7NgBPteg0uRzlfVicD3A/FQtmYl8K6C89qsqn8HUNVvq+pewI44o/9okTBeAqaISFtBvF70YWzAlYW8CTgduE59EmCw45eJdykK910J/KUg/FZVfU+Z3wz1HAbZ4v7yFN7XxSOumlbVC1V1R1w50/HAWwf7nVFZzIjqEBHZQUQ+LCLz/PJ8XL74PX6Xy4GPiMhe4thWRBYC9+K+Bj8mInERORg4AbiuxKHuAzpF5DwRaRaRqIjsLCJ7++O+RUSm+xTWRv+bXJFwrgXeJiK7i0gSZyD3quqzIeTehzOCL4lIi4g0icir/bY2XOF1h4jMpfgLuxS/AM4SkR1FZAJw/gj2bcOlTvpEZB/ciz/PGtw5WRJY933gEyKyE4CITBSRN/r5vX3qII4ryO+jyDlV1ZW4LKWL/TnZFZdC/Vlgt2twL9U3+PlBjz9MXi7Q9wdgqYic4e+zuNe1rEwYQz2HQW70xztdRGIi8iacif9hsIiLyCEisotvE7UJl2Vc7B42qogZUX3SiSu/uVdEunEG9DDwYQBV/SWuNtM1ft/f4mpQpXDGcwyu8Pgy4K0+RbUV6qrUHo8rCH/G/+ZyYKLf5WjgERHpAi4BTlXV3iLh/Bn4DC67aBUutXZqGKE+DifgKiY8jysTe5PffCGuAkQH8EdczbpQqOpNuKzM23EF97ePYN/3AheJSCeuYscvAr/twV2Lv/lsqv1U9Te4FOR1PkvxYdw1AVcR5Ue4wvbncFmYXy0RtdNw5ScvAb/BlcH8ObD9BmA7YLWqPhiIU7njD4dLgDeIyAYR+bbPLjwSd41fwmWLfRlXqaQUQzqHwR+q6jrcffph3Pn6GHC8qq4NEfdZwK9wJvRf4C+47DpjFLEGrYZhGEZNsRSRYRiGUVPMiAzDMIyaYkZkGIZh1BQzIsMwDKOmjItOT6dNm6aLFi2qdTQMwzDqjgceeGCtqk6vZRzGhREtWrSIFStW1DoahmEYdYeIlOtVZFSwrDnDMAyjppgRGYZhGDXFjMgwDMOoKWZEhmEYdUw2myPVW98jqI+LygqGYRjjCc0p6f4s/b0ZUn0ZUr1Z/99PfVn/P0MmnaOlPcE+J5TqE3bsY0ZkGIYxCqgqmXSuqJm4dQNmk+7PFh37IhaLkGiOkWiK0To5SaK5hURTjKbW+h5CyYzIMAxjBGQzuYFUS95Y8iZTkKLJ5ba2l0hESDTFSDRHaWqJ0z6t2ZtN1K932xJNMaKx8VmaYkZkGIZRgOa0iKFktzKWVG+GTGbr4YsEiCejm1MvE9oTJJpjxBMREnElHs0RjynxSI4IWUin0XQ/mupC0ym0J412pN18KkUqnaY/lV9Oo+ngfIroxIlMPfvs0T9RFcKMyDCMhkBVyaRyRbPCNptLT4pUVx+Z3n40m0EyaSSbhmwGyWaISZZ41BlJayRHLJIjJllikRxRyRIlS5QcEc1AZ3qzUWw2j0wGBVJ+CoPE40gi4f4H5iMtrcjkBJJIEJ04cfCAxjChjUhEZuAGnlqOGzb5ZFV9RETOBe5T1burFEfDMIyiqPpC/Y5uUq+sJ7V2PamOLtLdfWR6+sn0OFPJ9PST7UuhmRSSzThj8SYjuexmE5kgSmtUiMYiRKNCJBYhEhWi0QiRmCBSMBK9yIBBFBpGa1sRE4kj8cSW84n877ecjyTiEI9vfcxxSCgj8kP33oobsvcvwMEMjLY4G2dQb6hC/AzDaCDyxpLuz5Luc/9TfRlSHd2k1q0ns24DmQ0byW7cSK5jI9q5CentRNL9W4UV8YYSSyZIJhNEmhLE2puINiWITWgjNqHJTc2JrQ0hbxQBE4kkEs4Y/LzEG8coqk3YFNE3gf8DXodre/S2wLb72HJ8ecMwDKCEsfhaYan+DOm+LKm+LOnObrLrN5Dt6EB6O4n0dCK9m4j0dBLp7UQyKSIRIRJ1UyyZJDpxEtFF04lOWUpi6mTi06aSnD6ZxOR2kq1NRJqSEIuZUdQBYY1oT+AkVc3J1ld1HTCjstEyDGMsojklnRowllRfZsBggsbSl9lsQAqQ7t9sKhFvNLFUl5v6u0jk0ptNJhqNEGlKEps5ifjUhSSmTSExfQqxyZOITZpEdPJkpLnZDGYcEdaIOoBS3YQvAV6uTHQMwxhNShlLqj9LupyxFJLuJ5nuJp7tJp7upiXdTbSvk0hvF5HeTUSyaSIxIRJxZS3R5iSxOZOJTl5EdNKkzdNQjEZV6cn00JnqZFNqE52pTjpTnUxKTmL3GbtX43QZVSKsEd0AXCgidwP5LsNVRKYBHwGur0bkDMMYOppT+nsz9HWltzKWVG/edDKk+rNkShkLEE9EiSejxJMRJjQpsWgvsWgnsVQ3kb5Oor2d0N2JdHVAJrWFcUgi4cxl4Wyik5YN2WhS2RSbUpvoSnUNmEy6k039br4r3UVnqpOsZrf4nSDsMGUHM6I6I6wRnQfcBvwHeMCv+z6wLfAM8NnKR80wjGIEjaavOz3wPz/fk0F1a3uJJ6LEm6IkkjEmTEwyMRkl0RQllowSJ0Ms1Umkx6Vg6Okk17GR7JqNZDdsRPu3rAwgiQTRyZOIzppCdNKS0EaT05wzke6X2NS/ia60M5r8fD5105/duvJBMpqkNd5Ke6KdBW0LaEu00Z5opzXROjAfbyUaiVbuZBujQigjUtUNIrIfcAZwGNANrAcuB65W1a3vGsMwhsVwjCbZHNvcKn9Ga5ymljjJlhjJ5jjxZISYpslt6iC7YQPZja7WWeYF9z9vNOlAeJuNZtIkEgsXEps8eYsstEKjUVX6sn1sSHXSmVpFZ9eW2WX5qSvdhRakwSJEaE04g5nWPI3FExfTlmjbPLUn2mlLtJGMJjHGJ4MakYg04bLmvqiqVwBXVD1WRnVQhdX/hhfuh0gMkq2QaPP/WyDZBolWN8WbwQqDq8KIjGZ6MzNavNEkIUGKuKSgt4dc93py3T3kXu4m19NDrrub3k0ddJVL0YQwmnQuTVeqy5vJBjZtfI7OVwbMJW84Gd26B+jmWPNmM5nZMpO2+NYG0xJvsYoHDc6gRqSqfSKyN2Dp3Xolb0CP3wybXoCmiRCJQ6oLMn3FfyPRAnNq2dq4Eq0D2824NjNso0lCe3Oa6RPSJCJp4qSI5/qJZvugr4/cK13knnUGk+vpIZVKFW2dL7EokZYWIhMmEJ04saTRAHSnuwsK+1+mc9WWJtOb6d3qGDGJbTaTua1ztzKXtkQbrYlW4pH67ozTGB2GUlnhtbhyIqNeKDSglumw+1tg7l4Q8Z0nZjOQ6oRUN/R3OXPqzy93uuVUF2xcN4hxRQLGlDep8WlcoY0mk0ZSvURSfSQkTVMkRbukmUo/8VyKWK6PaDZFJN2L9vSg6fSWx8F1AxM0lkhLK7FpU91yfpowgUhLCzQ3k2mCVCRNf7aXvmwvPZkcvZrzhf7P0dn5MJ3rOjdXBMixZT9pgtAad2Uuk5KTmNc2j/ZE++byl/akM5qmaJOlYoyKEdaIbga+KiKzgRtx1bW3+KRT1RsrHDdjuKjC6oe8Ab1Y3IDyRGPQPNlNYShnXJvnu6BjpZsva1wtkGzf2rg2p75qY1yaU/p7MluaS3ea3g3d9G/oJNXRifT1IqleJNWHpHpJaD9xUrRrP1OyKaLZPtfvWDxCNBZBRFBVcpohG8mRa4rT3xwn2xQl0xohPTtOOhEnnRT6k0o6Dv1JpS+RIyVp+nPrSedWk872k8qlSWdTpDNpUhvSpNemSefSZApqkAEkk7Nobp7v5qPJzSmWRe2LnKn4rLL2ZPtmA4rI+Ozh2Ri7hDWin/n/r/NTIYpl3dWeoRjQcBmWceUNazDj6oIi2UDAgHFtTmG1bmlkWxhXC8QnlDSuvNH0buimd20Hfes20b9+E33rN9LXsY7+ro3QtwnS+Tj2EM30IZKBaI5IJAeRHIiSkxy5qNLTFPEmAqkk9CeU/gT0xnP0JJT+eJaeJGSSQjYm5U0166c+iEdixCRGIpogHokRjySIRxO0xFuJRxMkosnNUzyaJBFpIhlrcvPRZiZOmMO01sW0J9pJRBPhrplhjDJhjWhxVWNhjIxiBrTHGTBnz8oZ0HCJxqB5kpvCUMy4iplYgXGpKt39Gdb39NHRm6KzL0VPWulLZUinMqT702hfBunPEOnPEE1lIZcjh7o/cf9FIBcVMk1R0k0R0m0RMjOipJMRt5wQ9z8ZIZWMIM1xIokkcW8UiWiCWCRBIhInEU3SHk0wNWAUyWgTiWiTn28mGZtAMtZMPNpMU2wCiVgziWgzyVgLiegEIrW+foYxCoStvv3c4HsZo85YNqDh4o0rE2+lTzewbn0XHes20LV+NT0d6+jbtI5UVweZ7k5yvT3kenuRvj4ifWnI5ss7grnGChEhl4iSS0TRpECbIMko0hQjlowQa44QbYoQS8aJNsVIRCIkI3HiEiGpUZISIS5xmiLNJOLOKJqiE4hHJxCJN0E0DrEkRBNbTrFEuHXRhNNtGA3KUIaBiAGvBw4ApuDaEd0FXK9apN6mUT22MqAZY96AMpkMfV0bWLf6OTrWraRz/Wp6N66hr3PDlsbS14v09BPpT0Nu6wHHAIhEyDXF0eYk0txMdMpUYi1tJFon0tQ+leaJU2mfMof2qXOZOmshTa2TicUCt7oq5LKQTUG2H7JpN5/Jzxdb5/fNpPx8amBdqsvvF1ivJeJeColANFlgavGBddFEEbMrsm+h0cWb3T6GMYYJOwzEDOAWYFfgWVxlhf2B9wEPisiRqromZFhHA5fgypQuV9UvFWxfCFyJ69tuPfAWVX0hlJrxzhgyoEwmQ0/nOja8vLKIsWwk091FrrcH7e2F3jDGkkCbE0hTM9HJU4m1tpNoaffGMp32qbNpnzKHqbMWMqF96siyrERcCiQaAyYMP5xSlDK6bMobWRmjy6YDppYeMLrN+w7R6BYfBDsXK9Y1jLFD2BTRN4CpwH6qel9+pW9f9Gu//YzBAhGRKPBd4AjgBeB+EblBVf8T2O1ruN4afiIihwIXhwl7XDMKBuSMZS0bXl7JhjXP073xFWcsm9aT6u4YMJaeXujrJ9KfglyJXsoCxhJpbiYyZRqx1rYBY5k0nfYps5k4dQ5TZi6kZeK08VUVuNpGB64sLVuYOitidG2zqnN8w6ggUqxPqq12ElkPvF9Vrymy7c3Apao6JUQ4+wMXqOpRfvkTAKp6cWCfR4CjVXWlH3KiQ1Xby4W7fPlyXbFixaA6Crnwu9eRe3Il0SjEY1GSsQjRsVZ1NedfKLmsy76JJV1j1GGgKKlshkw6SyKdojXdQySbJZLJEsnmnOEBhZagIuRiUTTmXq4ad1lCEk8iiSYk0YIkW4k2tRFNthNNVOnlO0IWL5jIfgfPr3U0DGNMISIPqOryWsYhbIooCXSW2NYJhK0XOhdYGVh+Adi3YJ8HcVXELwFOBtpEZKqqrgvuJCLnAOcALFiwIOTht6Rv7SpmrV+31XoRV3sqIiAi/v+wDjF8VIGcK3cXgMiwI5FTRdUFmcAFmYsKOVHS0SjZpiSZaNxPCdLRJjLRJtKxCWRiE5BYE1GJEI04o45JhKhEiUaiRCQyYFxpP9EzUvVVYfL05lpHwTCMIoQ1onuA80TkdlXtzq8UkRZcz9z3VDBOHwG+IyJnAXcCL+JaVWyBqv4Q+CG4FNFwDnTx+R+iL5Xmjuef5+7nX+DxNetY1bGJju5++vqzBBOL8bjQ1pxgensLi6dOZs+5szliyWJmt7UN59DFUYVVD7osuM6XXBbc0qNhzh5DyoJLZzLc+tSj3PLUizy8qo9MH2yzMcECjTJ/fivHHrsjkyY3synVSUd/N53pHjale9mU7qMz009XJk1nOk13NkNXNktXtot0iZRzDKElKrREI7TForTE4rTF4rTGEkyMN9Eab2ZiYgLtiRbaE200R5NWJdkwjC0ImzW3O26ocMVVWngZNyrrUbjv9YNV9cEQ4QyaNVewfyvwqKrOKxfucLPmylHUoHr66eurgkFtNqA/QeeqYRlQofn0piEmwu7JOLv0tLGwfTI77DWbedtPRiJDT1n1ZfrLGpebMnRlc3RnlVQI42qNRWktYVxt8QlMTLabcRlGlRkLWXOhjAggMAje3sBsYBVwL/ANVV0bMowY8DhuKIkXgfuB01X1kYLjrPfDkn8ByKpq2fGOqmFEpaioQY3QgIqaT0RYOiPGQXNnsP3GqXS90s/Eac3ssP9sJrSPXsv6zcaV6qEz1T0i45oQFVqLGFd7vIk2My7DGBF1ZUQVO6DIscC3cNW3r1TVL4jIRcAKVb1BRN6AqymnuKy59w023tFoGlEphmxQzcJi6WDP2AaOmNHE7J2OCmVA5cznkCUzOW67ZfSuSvPkilfIZXMs3n36sFNBo0kp4+rKpOjMpEZkXK3ROG3xBLtNWciSiVZZwTCC1I0RichuwNxiHZt6Y3lBVR+qQvxCMRaMqBRbGtRaVq1bS0d3H31pQRFXEw4pm4Iqaj5RWDo9ziFLZnL80h1pb2qmvyfN4/e9zNoXu2qSChpNwhpXdzZHlzeu0+Yu5LB5NX3eDGPMUU9GdDtwl6qeX2Tb+cCBqnpYFeIXirFsRIDPgvuXr4SwClpn0rfkMO5ITeXeF17k0VeKp6ByCkRyaBQkEiMRjzNv4gSO32Ehb91tD9qbmn3wysvPbBpIBe0xnXlLx34qaDTpz7iRe5Kx8WnMhjFc6smINgKnqOotRbYdCVwXph1RtRizRlTEgFh6FMwungWXzmT43WP/5ucPP8XjazrpS6fQbAbJKZoVAhWliceF1uYE01qamarNzM208JrZ8zjwwG3HbSrIMIzKMxaMKGz17SjQUmJbC+HbETUGqvDSP+GJW7wBzYI931rUgNKZDDc/9Si3PvUCD6/qpy8NsWicfefP5JAlszhuu2W0NzXTl0pz18qV/P35lTz+yjpe6tjE+s4+Hu/oQ3UjIvDrtU8Sf/QvtDYnmN7WwqKpk9hjzmwOX7KY+RPLtgk2DMOoGUPJmutX1WOKbLsJaFbVgysfvXCMmRRRMQNaeuRWBlTcfGD7GXEOWTKLE5buSGuyqeRh+nvSPHbvy6x7qYvklBjr5yn/XLeaR19ew0sdnXT09NFbpJJEa3OCaW0TaEtO4EMH7M3ec+dU82wYhlEH1FOK6ALgzyJyL/ATYDWuCvdbgd1wfcc1LiFSQCM1H3cYZfXTm3jygZfRrLLtXjNcjTgRjme7LfbNp6DuWfniZoNa29nL2o71qG7gD1OnmhEZhjEmCDse0Z2+LOhi4FJcI9Ycrh3REap6V/WiOIbJG9DjN0PX6q0MKJ3JcPMT/+HWp17g3y/1058Zuvnkcamg1ax7qZtJ0yew/X6zypYFNSXiHLHNEg5cuJB7n17PXU+soas/y6IpSVonZTh66aIKnQTDMIyREXo8IlW9A9hfRCYAk4ENqjo2OxWrNrncQCWEzQZ0JszenXQux81P/IdbnnQpn5GYD2ydCtpurxnM9amgcvRnslsY0HYzWjls2QwWTi1V1GcYhlEbhjwspDefHhGZLCJLgf8O1uB03FDCgPqn78TNTz/On/91S0XMJ89QU0FgBmQYRv0RdmC8C4Gkqn7cLx8K/A432MpqPzDeI+XCqGuKGFD/rm/m5p4kf/7XKh5etXIL8zlsySyOG6b5QEEqKAfbLZ/J3KWTyqaCCg1o6cxWDtthJgumjs0hGQzDMPKETRG9Gfh8YPnrwF+BC4Ev4MqOTqxs1MYAuRys+ic8fgt0raZ/wgxunn4kt66N8MitqzabzzKf8hmJ+eTp607z+H0DqaAd9p9Fc1vpVFB/Jss9T6/nrsfX0J0yAzIMo/4Ia0RzgKcBRGQ+rqbcu1T1PhH5BvDjKsWvNgQMqL9zNTexgNvSB/DIywn6M5mKmw/4VNBTHTz5wCuoDp4KMgMyDGO8ENaIOoGJfv5QXEWF/JDhfVRtPORRxhtQ/6N/4qZ1/dyWmssjPUvpz8UGzGebWRy3XWXMJ09ftysLWr9q8FSQGZBhGOONsEb0F+DjIpLDDQXxu8C2pWw56mr9kcvRv3IFNz14J7dtaOKRniX0a5JYPMqymXEO3WY2x263rKLmA0NLBZkBGYYxXglrRB8CfgpcB/wL+FRg21txwzXUHf2pfm564M/c9tQLPLKplf7cAuLxGDvMbeLQJdUxnzxbpIJmTGCH/YqngsyADMMY74x4PCIRaQf6VDVVmSgNneF28fOji79NZ1cWARIxoSUZpy3RRGSQNjojJZPKkerNABBvihJPRrfaR1XpSWfp8UOWu/jFiEcrM+hb+6R2DnvHGyoSlmEY9Us9dfFTElXdVImI1IJZEycTz6ynrbm16uYDoDmlvzdDNp0jGouQnBDbaqiGahuQYRjGWGPERlTPnPDeM0blOKrKqqc6eOqBV1Bgm92nM6egLKgvneWep9dx1xNr6Ull2X5mK4ctm8n8KZYFZxjG+KahjWg06OtO89g9q1m/2pcF7T+L5taBsiAzIMMwGh0zoipRmApaunzmFqkgMyDDMAyHGVEVKJcK6ktnufvpdfzVDMgwDAMI39dcFtg/0Ig1uG0v4D5V3brqV4Ohqqx6soOn/uFTQXvPZM52LhVkBmQYhlGcsCmiclXK4kCmAnGpa4KpoMkzXU/Zza2JrQxoh1ltHLrDDDMgwzAMT0kjEpEFwKLAqj1EpLB1ZxNwJvBM5aNWH5RKBfVncvzfY6+YARmGYQxCuRTR24DzAfXT90rs1wucXeF41QV9Xb53hEAqSJJR7nh8jRmQYRhGSMoZ0WXAr3DZcg/hhoJ4qGCfFPB8wwyM5ymWCpqyqI17nlm/2YCWzW7jkO3NgAzDMAajpBGp6hpgDYCILAZeUtX0aEVsrFKYClq0fAb/fHkTf73l8c0GdOgOM5g32QzIMAwjDGErK0wA9gLuARCRZuAzwI7Abap6aXWiN3ZwqaCNPPmPNQAs2nM6z5DhhruepjdtBmQYhjFcwhrRZcDf8UYEfBVXhnQX8GURaVLVr1YhfmOCvq40j96zig0v99A2rZkNM+L89MmXzYAMwzAqQFgj2hk3PDgiEgfOAD6oqj8SkQ8C78KZ07gimArK5pTumUnu6umi95mcGZBhGEaFCGtELUC+l+39/PL1fvkfwMIKx6vm5FNBa1d1szGq/DeZo2dTygzIMAyjwoQ1omdwBnQncDLwT1Vd57dNww0lPi7Ip4IeW/EKqzt6eaYJOidE2HFOO4eYARmGYVScsEb0DeB7IvJGYA9c+VCeg9m6Wndd0teV5uG/vcRjT6znxVyGjhkJdlg40QzIMAyjioQyIlW9QkSeAPYGPq6qtwU2rwe+FfaAInI0cAkQBS5X1S8VbF8A/ASY5Pf5uKreGDb84aCqPPvf9dx950pe2thH5/Q42yybxunLZpoBGYZhVJnQvW+r6p24rLnC9ReEDUNEosB3gSOAF4D7ReQGVf1PYLdPA79Q1e+JyI7AjWzZ1VBF2bihj1tveornn91EX3OEeftO57Q95jB3UnO1DmkYhmEECG1EIjID+DCwHJgPnKyqj4jIubjet+8OEcw+wJOq+rQP8zrgJCBoRAq0+/mJwEth4zhU/n7/Kv5+23NkcsrMHSdxxEELLQVkGIYxyoQdBmIf4M/AK8BfcOVCSb95Ns6g3hAiqLnAysDyC8C+BftcANwiIh/A1c47vESczgHOAViwYEGIQ2/NzBnNTJ7RzKFHL2bxnPbBf2AYhmFUnEjI/b4J3A4sxbUZCg4LcR8upVMpTgOuUtV5wLHAT0Vkq3iq6g9VdbmqLp8+ffqwDrTNwkm84+27mQkZhmHUkLBZc3sCJ6lqTvJjXQ+wDpgRMpwXcdl6eeb5dUHeARwNoKp3+6EnpuFSY4ZhGMY4I2yKqAMolexYArwcMpz7ge1EZLGIJIBTgRsK9nkeOAxARJbhxjxaEzJ8wzAMo84Ia0Q3ABeKyJLAOhWRacBHGOhloSyqmgHeD9wM/BdXO+4REblIRE70u30YeKeIPAhcC5ylqhoynoZhGEadIWHe8SIyGbgN19v2A8D+uNTNtrheFw5R1Zr1rrB8+XJdsWJFrQ5vGIZRt4jIA6q6vJZxCNugdYOI7Ifr7PQwoBvXkPVy4OpGGxjPMAzDqBxDadCaAq7wk2EYhmFUhFBlRCKS9W2Jim3bS0SylY2WYRiG0SiEraxQWGU7SBzIVCAuhmEYRgNSMmvOdz66KLBqD9+mJ0gTcCauwoJhGIZhDJlyZURvA87H9f2mwPdK7NcLnF3heBmGYRgNQjkjugz4FS5b7iHgzWw97lAKeN5qzRmGYRjDpaQRqeoafI8GIrIYWOVrzhmGYRhGxQjbjui5akfEMAzDaEzC1pozDMMwjKpgRmQYhmHUFDMiwzAMo6aYERmGYRg1JXRfc74x64G4wewKG7aqqpZqZ2QYhmEYJQllRCJyAPBrSg+OV67Bq2EYhmGUJGzW3LeBp4E9gKSqRgqmaPWiaBiGYYxnwmbNbQ+8TlUfrGZkDMMwjMYjbIroIWBWNSNiGIZhNCZhjeg9wIdE5KBqRsYwDMNoPMJmzd0KTABuF5EU0Fm4g6rOqGTEDMMwjMYgrBF9F1czzjAMwzAqSthOTy+ocjwMwzCMBsV6VjAMwzBqylB6VtgfeAewlK17VkBV96lgvAzDMIwGIVSKSESOAO7Ede9zAG7AvC5gN2Aq8HC1ImgYhmGMb8JmzV0EXAIc55c/o6qH4lJHaeCOykfNMAzDaATCGtGOwE1ADld7rgU2j9x6AfCpakTOMAzDGP+ENaI+IKKqCqwCtgls24TLsjMMwzCMIRO2ssKDuP7mbgVuAz4hIi8CKVy23b+rEz3DMAxjvBM2RfQtBhq0fhLoBm4G/g+YAbyv4jEzDMMwGoKwDVpvDMy/KCJ7AdsCzcCjqpqqUvwMwzCMcc6QG7SKiACzgWdU9aGhmpCIHC0ij4nIkyLy8SLbvyki//LT4yKycahxNAzDMOqH0EYkIseKyL24igsrgV39+h+JyFtChhHF9Vt3DK4m3mkismNwH1X9kKrurqq7A5cC14eNo2EYhlF/hG3Q+lbgBuBR4BxAApsfx/W4EIZ9gCdV9WmfkroOOKnM/qcB14YM2zAMw6hDwqaIPgV8VVXPBH5WsO0RXOomDHNxqak8L/h1WyEiC4HFwO0hwzYMwzDqkLBGtBBXdbsYfUB7ZaKzBacCv1LVbLGNInKOiKwQkRVr1qypwuENwzCM0SCsEa0E9iixbTnwZMhwXgTmB5bn+XXFOJUy2XKq+kNVXa6qy6dPnx7y8IZhGMZYI6wRXQGc7yslNPt1IiKHAR8DfhQynPuB7URksYgkcGZzQ+FOIrIDMBm4O2S4hmEYRp0StmeFL+NSMj8B8lllfweiwA9U9dthAlHVjIi8H9cYNgpcqaqPiMhFwApVzZvSqcB1vkshwzAMYxwjQ3nXi8g2wOG4oR/WA7er6uNViltoli9fritWrKh1NAzDMOoOEXlAVZfXMg6hB8YDUNWngKeqFBfDMAyjARmSEYnI9rjq1sVGaL1x618YhmEYRnlCGZGI7IKrwbaMLRuz5lFcmY9hGIZhDImwKaIrcSOxHo+rqm2dnBqGYRgVIawRLQNer6o3VzMyhmEYRuMRth3RfcCCakbEMAzDaEzCpojOAa4VkR7cYHgbC3dQ1Z4KxsswDMNoEMIa0VrgWeDqMvtYZQXDMAxjyIQ1op8B+wNfwyorGIZhGBUkrBEdArxTVa+pZmQMwzCMxiNsZYVnASsDMgzDMCpOWCP6KPApEVlUxbgYhmEYDUjYrLkLcdW3HxeRZylea26fykXLMAzDaBTCGtHDfjIMwzCMihLKiFT1bdWOiGEYhtGYhC0jMgzDMIyqYEZkGIZh1BQzIsMwDKOmmBEZhmEYNcWMyDAMw6gpZkSGYRhGTRFVrXUcRoyIrAGeG+bPp+F6F280GlG3aW4MTPPQWKiq0ysZmaEyLoxoJIjIClVdXut4jDaNqNs0Nwamuf6wrDnDMAyjppgRGYZhGDXFjAh+WOsI1IhG1G2aGwPTXGc0fBmRYRiGUVssRWQYhmHUFDMiwzAMo6aYERmGYRg1xYzIMAzDqClmRIZhGEZNMSMyDMMwaooZkWEYhlFTzIgMwzCMmmJGZBiGYdQUMyLDMAyjppgRGYZhGDXFjMgwDMOoKWZEhmEYQ0REpNZxGG2qqdmMyDCMISMic0TkUBGZIiKJWsdnNBCRGSKyi4hEtUGGLRgtzTYMhGEYQ0JEdgJ+D/wLaAHuAH6jqo/WMFpVRUR2Bm4A/g1MBr4P/EVVX6xpxKrIaGo2IzIMY0iIyAXAelX9togcCewDzAYuHY9mJCJR4OvAw6p6uYicBewCvAz873g0o9HWbFlzhmEMlVZgGYCq3oL7al4FnCoiLbWMWDVQ1SxO82K/fBVwCzATOALGX5nRaGs2IzIMY1BEZDsROcgvfg2YLSJnAqjqQ8BfgT2BiTWKYrX5Nk7zkQCqejOwAniriCTHS5mRiLSLSKtfvASYNRqazYgMIyQisqOIXCEi54nISbWOz2ghIjsCvwJ29V/B64CfAa8RkbcCqOodQCfw6lrFs5KIyFIR+Y6IfFJEjvVm+x/gMBE5AkBVrwU6gP1qGddKISKzgEeA00VkOvAs8F/g8GprNiMyjBCIyFzgWuBx4CXgByLyvtrGqvqISBz4IvBtVb3UfwVngBtxWTXHisgl/kX1GuDp2sW2MojIMuDnOMPtBD7pzfjnwFrgZBH5tIjsDyz368YDPX7aGTgKyAI/AdYAJ4nIZ6qlOVbJwAxjHDMHeF5VvwwgIg8BPxURUdXv1DZqo8LNACJyBa7s4B/A7cB5wEXACcAHVfWBmsWwAojIBOBDwGWq+iNvxPOA7VT1dyLyU2AB8AlgKU7zI7WLcWXw9/EmEbkb6AKOxZnwvcDdwF+ATwLbUQXNVmvOMELgsyq+BnwF+I+qqojshntBf0BVfznI76OwuRC4rhCRTwCKK7heC/wWOAdYq6qf8PtEVDXnX2jq180Dkqr6VG1iPjxE5GDgcVV9yS9/Bligqu8M7BPBaesNavbb4kBUVfsKt401RKRVVbsCyx/FpXRjwGeA/YF3q+pvfLZsspguf3+LqmaGo9my5kZI/gVTgXB2EpHPVSKsekFEthWRj4jIySIyu9bxKcTH70wROVlV1wAvAp/CtZ1BVR8E3g/sO0g4OwNXA5d7re1VjvqIyDdQDdzbDwBzgVnAD1X1fuB84BARWQKgqjn/P29C2+OysrYZ3dgPDxFZICJHishOqnpH3oQ8T+KyIxGR5SJygKrmVLUXBjT77TsC1wC/FJHTx7gJ7QRc6j+o8qxiIIt1d+AFv+90dfTBVpqXAT8GrhGR1/uPtCHVqDMjGgEisgtwrYh8L1+DaJjhbA9cAayuWOSqRKUK7AN57tsBrwUOrVAUK4KP37W4NjKnicgpqvpJQIArRST/gs0Ci0SkaDa3L1u6AlfF+cfA+4DPBn4/pvD39D9EZO986s1X0X4Q16jxzSLSBjThXs6pImEsA34BfMP/dkzj4/tr4Azg0yLyOr8+/35cDaz298QVQLJMOFcDfwQuA74qItOqHP1hISLzgd8Af/MfVHn+CRyOy4r7Gi5V9AZKa94B+F/gb8AvcWWnhw/ZgFXVpmFMwAxcVcZ3AUfjWh9/FJg2xHDm4WqnvMUvR3B50G1AvNY6C+I6F/dCOg/30K4G3jeMcKLAd4Ez/fL7gAu97tljQGcM91X7dr/8blzKp80vfx5Xa+w64FHgtWXCejXw28DyibgKDx8EJtRaa0Fcp+CyGu/D9ZqwZ8H2U/x1+hNwD/CGEuF8GugILF+Ie6mdPgY1zwf+DrzZL38ReCsuay2/z+G48pJ/A8eVuWc+DbwrsO4O/044EWiutdaC+L4WuMLPR7zGw/zyd4GPB/adUyKMuL+25wbWvQf4xFDjY5UVhs9kXDXGK9Tliz4DfBn3xfyVIYTTDfQD0/3yb3D58c3Ab0Xkf1V1U+WiPSJmU5kCe8UVeO/il8/AnYdFQLuIfFRVn6xctIdFDuj02VNn4r78DxWRPlU9XUQW4bKdOlX1vjL54i8D630V4Btx2Xr/wL2cngeuHw0xIYkCl6vqL0Xkg7isltNV9R8AqvoL2JyC71fVZ0vovhhIiMjfcff2Q7iX+LuBdlxXMWOFCcDXVDV/HY4GDgD2FpE08BFc9lQH7oV7e7FA/Dvgu6q6AUBErsMV+j8FnIv74LysqkqGxgu4Zw5cDcgNQJuIfBY4RH15H4BumU25GVVNi8j1uCzrPP3Aq/JlhqFjU2tnrtcJSADfA44Hmvy67XFJ27ND/D4OtPr52bgv0FXAF/26U3Ff3MtqrTUQ50XAlcBODFR02Q2XMnrjEMNaiGuzcD2unzJwD+tXwpy/UdD6Bly7kT8Dv/br2nDZiaHjh/vaPA+XffEHXBZGAjgG/0U6liagJTD/QVyKb2+/vDh/3cv8Pn9fRIBLgZ8Eth2ES2211Vpnibi/A7jYz+/qn78j/PIuQX2DnUNgr8DywcD9QHutNQbitK2/FhcB5wXW/wL4XFitRa77HsBP/fyrgGPC/N7KiIaAuB6H9xWRxaqawpnH64DtxLU0fgxX9fNoKdMjcaBA81oReauqrsJ1m3GpunIIVPU6XOpqSXVVlUdcw75zfJyexX3lDafAfrGInJ5fVtXncC3xf4l7SFHVF4BeXFXpmpAv61HVX+Ea7V2JezhR1U6ceQ7a23SgllxOXQryw7jqryf7e6cV6KuGhuEQiG/+KxlV/RbwQ+CHIvIe3D27fbkw1L+BvO4PAGcHdknjPlrCfylXkSIVja5UXwtQXQPWDtzHB6r6b/9/q1RvYTiq2q1bVmPvZ4zoDlznJ3HZah/CfVjmuQmXDVlUa2E4eQL79gGbRORVwFWErYdQa2eul8lfrPtwBZE/BU7367+GK8A8DGccB/iL2VQinGW4sqWzcF/FL+LLRQh8geBe0o8Au9dQc9xrywH/L7D+elxB/jZ++WRcy/tYiXC2wzWKuxnXBiG4bYE/Xx8G9saVQR1aI73R/H9gDz9/MC5ldBTObP81WPwC4URwhdd7FWw/FldOdGyt7+si8d09Px/YfgWuUsbJIcO4Cp+KCmw/0J+7E2qtt5jmoF6/fmd/L+47hHB+XET3a7zuE8eQ5ii+/A+Xo/M08HZcivDfwNHD1ezPZZd/d5UNZ4swa31y6mHCZRn9A3idXz4Ll5ee334eLv/3z8DDlMimonyB5kn4Ak3gEFze8vFjQPtxwDdxecpfCqy/2L9kwxTYnwJ8CzgSVxD6wcC2BK7G3F9w5WMlw6myzuDDdReu0DW/7ixcFeabB4tfQTg3+nujOX9P4GqbfTRwL4XO/hgl3e9mSxNagGvUWDK+ITQv8Nf9pDrRfIq/p8uaZgjdOwKX402olrqLaH4v/sMRl4X2Llxv20eVi2sIzZNw/Q6GNiFVM6KwF3EH4KLAchuu1tBugXUTcV/0Ow9yIScH5q/DlRu8DrgNeK9fvxeuwHAsPLTHAv8P1+vuC8B3cNVywZUPHQrsM4hmwRUKN+FST9/FZQkEayYFy8xGVXPBw/VHXAF1Cy7Vt9Bva2eg1lwpnZFAOH/AGc4EXA8Eby92Xmp8bcvpnu+3zQB2CFxHKQgjlGZgUh1p3hc4fJBwyul+R2C/KbXWXUbzdfn7O2Q4g15rf48sGKrmmt0Q9TYBE/3/JC5p+0d8RQL/sA7pRqMOCjQDen/s5w/E5QH/dQi/L3xxTcCZ0WW42nIHA2fVUF+wcP33wMd8HG8hUIV1COFFgVsD4fwZ+FCtr2M1dTei5nrRXS+aa36ixvpE4Ks9f0H9/5/jKhLsg2sMt2go4RTZvr//ymgdA5qDXz4t3jRej+tz6gvAJuD8oZ67wPpWXHbAjbgC7NfV+hrjarOd5x+u29gy+7DkR0bgQc+nlt6OaxfV4h/Sc8OEU0+6x7nmSJnf1qXueri/a36SxvIEWxfiBrZdhque+g+Glpf8Y+qjQDPCQIH92bheeT/ulxcxeNbFYJqPw9VKOs4vj9qDG3i4dgqsO8w/XLcD/1O47yDhHAl8zj/kbbhsxuuBj4QJp550N6LmetNdj5prepOM5YkSBZqB9T/BVTU+tNyFoL4LNN/j5+cD++W3BfYflma/7e3AKflwRls3rsbiM8DywLr3A58aTF9BOEfgasAdEFiXAF49lHDqSXcjaq433fWmueYnbCxOlC7cu4aBgrijgdcPEk69F2i24hpizivUNEzNRQvsa2BCuwCPMVCFdS4FWaIhXsYRXOWLnzLwUfE6XEr5zLDh1JPuRtRcj7rrUbM1aC3Ad1mS9R0e/g5XrfgyXNXih1T1eb/Pn1T11+IpFpa6bjKiuGq/d+Jqi90A/F5Vrwjst97/1+qqK04ZzdcDD6traJqPa9lGeYNovjJ/vMD+WgPdUdxQBvNF5NM4w7xORJbn41cqToFuT3LqeiK+CfiCiNyIy2J9Ajg139llra5pCYaluxE157dBXequO83W11wBqqr+RXon7gX6HVxtk9+ra2kO7ite8/sHf5+/yCLSpq4l/pm4m+IqBl7IlwT3rbqoQQijebCbdyxrDsRvKq5yxH9wHXyehes14WhcbxF7ACtC6DwI113NP3D9qJ0FrFfVR0VkV1ytwLHQin7EuhtRc0E4Y173uNBciWRVvU/UYeGeaR6yzuMZ6Kr+q7iPsLjftiOuj8DXhAjvWNzDebYP77OBbUfjGjSfVMtrW2ndjai52rpxDT9HXrZSR5rLHrfWN89Ymaizwr3xrBlXtftMXG8Twxo2gC0rVeyDq3o+D/gAbqCzfOPZQ/y2kwYLC5fl8XVch62vwX0xzvHb5uLaRR0znGuMa0/1ftzXZ8tQflsN3aOh2f/mKOD7I7yP6+1aH4fLdRh2m8F60zyonkoGVq8TdVi4N141477s/uUf1CuBL1GiD7syYUzDjSCa78Nvb1w7qDfhesRY7NfviDPMHYvFE5eymxo4XxFcF023+Ic73xL/WJwZDymegeMci/ti/QyucsiQx3iqlO7R0uzDiOLapeXY8st7KPdLvV3rY3Av+INGcN7qSnMoTdUKuJ4mXEd9X8Z9gX/avxT+gE8plHswilzc03GpgxtxfbR9CFfgN6QB8xpRM64rpXsZMMfX4PqzG9KgYrgvxO/g+sObBSwH/usfrvyX4qG42oFTy4SzF27QtHfjqrNvh8s7vxM4x++zn9d+0DCvwzJc5ZBX++X/h2u1vpQhpgYroXs0NBcc70B/z9wNXObXJcfptV6My/b6lF+egh+QjiE0ZK8nzaE1VTPwqkV6pI2nBvJVp+L6EEsAP8LVjDsN16/at4F3hgznIOCzuK/5nXFZS/n+uXb1F3bKWDhnY1kzblymYP9kER/OAUO95j4+X8GNuBnDDU72DO7r8Qxc78CDdiqLG3MqC5zhl2fhsj9uwBnvQ4ygR2l/3nf381Nwnd3+Hlfb8DaGmH1TCd3V1hy8l3x8L8SVTf7FH+PPfjnUNa+jaz3daz0Pl/X8V9xIv7/ElcuGHqepXjSH1lPtA1Qkki7//J3ACQw4ftn2LGXCqovCPdxLuSJJ4XrQHIhjvgfyfLx+ge+TD/flV/JrufDFhTPEbwGf98vvx73grwaOLPabImGejPtifxnYNbB+Ki7Vsl2YcELG9yTgTX4+RmC4kSGGMyLd1dRc4nhX+v8n4IYQ+NNoax4t3bgs8M/45+c9fl0C10D+TeNRc6jzUs3AK3ST5gvCvwz8ADcmUL4BaGgzoo4K93DjyXf53xcd12gcao4WHCf//4e4LLs34r4aZw0SzjHABbgafE24HsK/iRuJst3vEy/z+7wh7or76swb44dxYyrNxGWnnRdWWwjNpXqo+A7wlpBhDVv3aGouuLYxXKromz7ejwBvxo2J8+Vqaq7VtfbzU4HDCrZfRriPjrrRPKTzM5oHG+YF/ALwscDy13GNqkL3RkAdFe75G+EXuGyzP+CGDA+dZ16nmvMv5Dm4sWDigW2X4rLnVhCoal7w+/zDtQduHJmzcQP13YHL7lrmH/Sv4XoTL/oBEwjnOFye+2U4o32VX/8xXNbZiLMrimhOFmx/vde8bZkwRqx7NDUX0Z1P/Z2AG8H0DX55Uv7+HGfXei5wauFzhhuWfgV+oMnxoHnI52g0DzbMi/g24IKCdV/DZQmFquZKHRXu4fLG9/fzJ/ib7VTGaYF94CGdl38AcGVY+Q+NbwLP47MIyoSzNy5LItj+6TLgL37+VcD2JX47MTC/G244jnk4c12Lyyt/jd++cz4chpldUUbzJFwK4Wz/gtg5RFjD0j3amkvoPhE3jtc0Bj6YSn7Nj5NrfWL+Wvv1b8JV0Cl7retJ87DO02gdaAQ37864bKTTC9b/CDh4COHUTeEekAjM583oNL98MCFrU411zQUP6d24/OptcIXWSwLXf4cQYR2Ie3lfScBYcZUxZpT5XdI/mB/1y3H/sB6E+0qdBFwCPDeU+22YmvOp1MUMMqzISHSPtuZBdN9BidRPg1zrBfj+K8eD5mGfq1odOOxF9PMHABvwL2O/7grgzYOEUVeFe5QoM8CZ0R9xXeY8Ue7GHeuaGcgeyGtdgEvdvg73Ar6PgWGpy47h5PfZDf/SBnbCvdjehStX2ht4msGzeXbApRbPDmw7l4GRaI/A1XAqmjVYYc2hyj2Hq3u0NA9VdzU11/O1rgfNlZhqctAQJz/4JXG6nz/In8yLgW/g+lNaGiKsuijco0w5iV//Y5wRhMmuGZOa2TJffJb/vwJXHrLYz59QuG+Z8A4BXsQNTPhNXAWK3XHZhnf4c1YyvIDO7XF57mkGvhxf48P4Bu5L9MBhXteKah6p7tHQbNe6cTRXaqrZgcuc/GJ5qpMDJ/T1uA78lg12Q1AnhXtFNG8uM/DrF+MaiO4+HjTjvuju8fNLcSPd/otw2YT5+LUDH8eVhS308foBbtC+pbjsj3NDhHcALpX5KlzD3Jfw+fC4/vO+DRxdgft62JorrXu0NNu1bhzNIz5ntY5AwUkrlad6J2VqD5UIqy4K98poDuYjNxGiYWM9aMal1m4jkL2Iq4xx0hDCOB5XEeNeBkaRXeYf1qv9+dsf1y7qLWxZjX02W3bE+kbgB4Hl3XFV588tOOZICulHrHkkumuh2a5142iuxFSzAxeeACqUjxwId8wW7g1F81BukLGomS2zK+LAe3HDjm81qGAYrQy0hzoRN+bRb/FVxnFV0T+JL8fCfQ0uLPj9ApyRzsHVUNvPh9HKwIv7B7jqxPOG84BWWvNIdY+GZrvWjaO5GlNtD16d/PPdGMOFe42kuUDrHP8wxHBDkN+IH2Z9COHNx7WxujSw7ne44SYSfrnF/48U+W2wV/GrgO/6+R8B1wGvxqUAf47vzWGE13fEmkeie7Q027VuHM3VmmoeAX+iRpR/Hginbgr3Gkkzbrjw3+C+8N6GM9yz/UMWOn/aa/sMrkHuMYH1twJ/GOS3S7yGz/nlHXEpx/zy+cD38V+jFbinK6J5JLpHW7Nd68bRXOmp9hEYYZ4qdVi410iacf2n3eLn/wRc5een4KqS/xw3mF45k9zN65mL69nhQ7iuf44K7FvyCy8QzkG4fvA+5peX4vLcLwrsG7rHjmporpTu0dZs17pxNFdjGv0DVif/fEwX7jWSZgKNcQPxfBvwEf+gJv36ebiuhSYNEt6xuKr6H/dad8Vlf/wPrufiY0Oer5NweeVX+PDy7aqW4l4W+WyMQdsuVVtzpXRXU7Nd68bRPBrT6B6sOvnnY7pwr5E041Jop+Eaup6A60PrWFw24U0M9Kj9YVyedWKQ8HbGZVcuBt6O60boaa8/gRu7Z5cQ8WrDdVN0sF9e5uP0Cb+8A4GGukO8FhXVXCnd1dRs17pxNI/WVJuDVi5PtW4K98a7ZgaM8AxcO6ZH/HITrov7i3HZj+8EHmSQDkz9/HT/EB2IS9m147oregXYZwhxa8X1TJHvASLq47GGwBAWw7imFdFcDd3V0mzXunE0j+YUYZQRkZOAI1T1ZEBxHWU+g3uR3gq8TUQmiIiECC6C6zJ+bxE5BkBVT8J9IVzvl7v9/1zBb+PA60Xkc375K0CziHxOVd+Jq0F2Bn74ZlV9wDSX1DcdV1kCYBUwGXhBROaqah8uhbYW96W3L66rpkeKhCOqqiJymIi8V1XXqOqjuKzCq1R1E25489txncOWio/4//P9eejCVYm/VkTaVTULvIDLwrw9jMZqaa6U7tHQXEndjXit60nzqFNtp6OCearUSeFeg2qegO+QFdej8gdwKbXd/fad/f+yw0Tg8rofLND1Llxq7X04s9y1VPwCOo/GPdTXAe/AfTVehMv++ASwEjhgqDqroXmkukdTs13rxtE8mlN1A69O/vmYLtxrRM2B8E/GDdkwEZc98ElclsFFuNTa5EF+34bLvlyIS729Clf5YgK+TRPw2hDx2MdrfBVuoLWv4z4CIriPglOAQyp0j49Ic6V0j6Zmu9aNo3m0puoFXMH880CYY7pwrxE1FznO0biq4BP98jn+YQ2jNYGr1v5z4H9xg+I9AlxWsF/JLzzch8A9wK2BdSf6h/yTlBl7qRaaK6G7FprtWjeO5tGYqhOoK4T7nZ8/HNdr9M3AXL9uLi5F8A3gcvzooCXCqovCvUbUXOZYx+KyGPJZfIOOMMtAlsN04PN5bTjDvAY/mF/I4x+Ma+T7wcC61+O6KVpSpXt+yJorqbsWmu1aN47mak/VC7gCeaqBi3cY8N7A+v/Bt3EBzsRlcZVMigbCmR9Ydz6uO538sAjH4JK4B5jmilz/k3ApuQjhx9iJFiyfjOvl+6QQOvf3x9zbL7/Kaw12AFuyv71aaR6O7rGk2a5142iu6vms8sWqRP75SdRR4V4jai6jo+hXXrk4B5abcV+Kg/a7hzPUx3DDmXcDp/j1++KySD9czfs8jOZK6x5Lmu1aN47mqp3LUbhYI8lTrcvCvUbUPIxzdBCuUe/uQJNfV/iwJgPzpcZPWoTrOXwprmPWp/10tt++P/DqWumshu5G1FxvuhtR84jO1yhdlOHmn9dt4V4jag6hLZ9S29dr+iXOLN/HQCPc/D75ih8J/AixRcLJV4NfgDPeB/zyKUAO/+VY66kSuhtRc73pbkTNlZpGpUGrqt6Iq3p8u4hEcD08l8U3/krhapk9AVyiqh/AVYeeJCKtgfC1zLE3+WPvKCIf9OtuwI3/MxOXhVZxGlHzYKiqisg+wIW4Rn9vxDXo3RY4S0Ra/T4xVc2IyCTge8H4BhoFngBcIiITVfV5XP95D/vdnse113px9NSVZqS6G1Ez1J/uRtRcMUbT9RhCzRC/f90X7jWi5kH0HQlk8PnauPZQp+C6qD+Xga/AibiU4cFFwjgEV4Pw0MC6fYCf4roqehSfXUGZlGM96W5EzfWouxE1V+S81ToCpU5m4TrGWeFeI2oOxPVEXNbFaX45gmsEvJNfnoQbB6lobT5cmdl7/Hz+wU7gaiy+Bziy1horrbsRNder7kbUPOJzVusIBE7+QTRY4V4jag7E+VjgAeDMgvVxXM/frynz2y8D3ytYtwswO7A8Jr8Uh6u7ETXXs+5G1Dyi81Xji9VwhXuNqLnMuTgR19XQHLYcL2nKIL+b43/3Sb98AL63iVprqpbuRtRc77obUfOwz1XNI+Benn9ioF3Mqbghr9+HL18JvJAn+Zf2rMDv8y/kE3D5sBP98knAT/z8frixf8ZEqqARNZc5F9OHuH/U/98W16DwatyX53G11lIt3Y2oebzobkTNwzpPNY9AAxbuNaLmYZyjcg118/9bcF2mLKhXnY2uuVF1N6LmclNecE0RkRNxHYJ+XlWv9dWd3wQ8pKqP+GqOvwU+rap/LfL7jwGdqvo9EUmqar+IJHC1xpYBT6nqLaOlJwyNqHmoiMhBuBFn7wYeVdW+QPXWiBaMt1RsXb3RiJqhMXU3ouZSjAkjAhCRY4HPAd9W1Z8E1seB7wA/U9W7Svz2y7j+094TWLcLsFZVV/ll0bEi1tOImgcj8CDuC1yJyyvfBPwDN3hYd2CfqKpm/fmKqhukrO5oRM3QmLobUXMoap0kC040YOFeI2oOcU6GWoZ2JYEytHqcGlFzo+puRM2DTaM+VHg51LX+P0hVX9JAElRV15f6jf9qeAn3Qj9FRK7GdYf+AVW9r+qRHiGNqDkEk3BDaRzhl3+Fy77YBXiHz4rMiMhE3BDOV6vq6prEtHJMovE0Q2PqnkTjaS5PrZ1wCF8RDVe414iaAzpH1CiwHqdG1NyouhtRc9nzUesIDPHiHUSJBqAU76U49JgwY3VqRM0BLcNuFFivUyNqblTdjai51DRmKiuUohEL9xpRcyl87cIv4bIyVqvPvhSRKVom+7KeaUTN0Ji6G1FzMca8EQH4Hm0vAj6mqg+JyKm4ngmexDXg7Cro0fYbuEL8us1XbUTNpRCR6aq6ptbxGE0aUTM0pu5G1FzImKqsUIZJNF7h3iQaT3NRGvEhbUTN0Ji6G1FzIbFaRyAMqnqLiLwOuFhEXlLXAPRXuKG4H1LXmHMSZRqA1huNqNkwjMakLrLm8oykAWi90oiaDcNoLOrKiKAxC/caUbNhGI1D3RkRNGbhXiNqNgyjMahLIzIMwzDGD/VSa84wDMMYp5gRGYZhGDXFjMgwDMOoKWZEhmEYRk0xIzLGLSJylYis8PP7iMgFNYrHOSLy2iLrnxWRr9UgSoYxprBac8a4RUS2AZpV9WEReT9wqapKDeKxAnhYVc8qWL8HsE5Vnx/tOBnGWKIuuvgxjOGgqk9VK2wRaVbV3pGEoar/rFR8DKOesaw5Y9ySz5oTkbOAS/069dMdgf12FpE/ikinn34pIrMC2w/2vzlKRG4QkS5c90qIyIdF5H4R6RCRl0Xk9yKybeC3dwB7AWcGjn2W37ZV1pyInCIi/xaRfhFZKSJfEJFYYPtZPoxdRORWEekWkUd9v4TBcA4QkbtEZJOf/iUib6zUuTWMSmJGZDQCfwS+7uf399N7Abxp/A1oAt4CnAXsBPxeRAqz8a4AHsSNrnmFXzcPZ0onAe/EdUr7d98rOv44jwI3Bo79x2KRFJEjgZ/jxp06CWeeH/HhF3INcANwMvAEcJ2IzPPhtAN/AJ4GXg+8Afgprkd3wxhzWNacMe5R1TUi8qyfv6dg8/nAauAYVU0BiMhDOPM4li1N45eq+pmCsD+UnxeRKHAr8ArOSK5W1f+ISDewpsixC7kIuENVz/TLf/JeeLGIfF5VXwjs+01VvdIf9wHgZeB44PvAUmAi8H5V7fT73zLIsQ2jZliKyGh0Dgd+A+REJOazwZ4BngWWF+y7VUpGRPbzWWTrgAzQA7TizCA03sT2BH5ZsOnnuOd0/4L1m41FVdfhzG+eX/UU0AVcIyIn+eFCDGPMYkZkNDrTgPOAdMG0BJhfsO/LwQURWYAzBAHeBbwa2BtnCk3DiEe88BiB5SkF6zcWLKfyx1TVDbgBFePAL4A1vgxsyRDjZBijgmXNGY3OelyK6PIi29YWLBe2dTgamACcpKrdAD5FVWgaYViLM8AZBetnBuIZGp8NeLSINONSfd/AlSvtN4y4GUZVMSMyGoV8+U+TqvYF1t+Gq5zwgA69UV0zkMNlyeU5ha2fq82plVKoataX9bwR+F5BeDncMPFDxlcx/72I7Ax8YjhhGEa1MSMyGoVH/f9zReR2YJOqPgZcANwH/FFErsSlTObisrauUtU7yoR5O66W3I9F5AqcoX2ErbPNHgWOEpGjgHXAM75cp5DzgZtF5MfAdcAuuNF5f1RQUaEsInIc8HbcMPLPez3v8vE1jDGHlREZjcJdwFeBc4F7gR8AqOrjuOyqHuCHwE3AhUA/8GS5AFX137jq3vviqkufjkvRdBTs+nngv7jymvuBE0qEdwtwKq6SxO+BD+Kqnb8/vEzw8Vbgi7gyrK8Af8KZk2GMOayLH8MwDKOmWIrIMAzDqClmRIZhGEZNMSMyDMMwaooZkWEYhlFTzIgMwzCMmmJGZBiGYdQUMyLDMAyjppgRGYZhGDXl/wPdrsl7x/m/PQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results = pd.DataFrame(rsh.cv_results_)\n",
    "results['params_str'] = results.params.apply(str)\n",
    "\n",
    "results.drop_duplicates(subset=('params_str', 'iter'), \n",
    "                        inplace=True)\n",
    "\n",
    "mean_scores = results.pivot(index='iter', \n",
    "                            columns='params_str',\n",
    "                            values='mean_test_score')\n",
    "\n",
    "ax = mean_scores.plot(legend=False, alpha=.6)\n",
    "\n",
    "labels = [\n",
    "    f'iter={i}\\nn_samples={rsh.n_resources_[i]}\\n'\n",
    "    f'n_candidates={rsh.n_candidates_[i]}'\n",
    "    for i in range(rsh.n_iterations_)\n",
    "]\n",
    "\n",
    "ax.set_xticks(range(rsh.n_iterations_))\n",
    "ax.set_xticklabels(labels, rotation=45, multialignment='left')\n",
    "ax.set_title('Scores of candidates over iterations')\n",
    "ax.set_ylabel('mean test score', fontsize=15)\n",
    "ax.set_xlabel('iterations', fontsize=15)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1st iteration: a small amount of resources (#samples) is used. All candidates are evaluated.\n",
    "\n",
    "- 2nd iteration: the best half of the candidates is evaluated. The number of allocated resources is doubled: candidates are evaluated on twice as many samples.\n",
    "\n",
    "- last iteration: only 2 candidates are left. The best candidate is the candidate that has the best score at the last iteration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips\n",
    "\n",
    "- Parameter search uses the estimator's `score` function by default. They are [accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) and [r2 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score) for classification and regression respectively.\n",
    "\n",
    "- Alternatives can specified using the `scoring` parameter of most search tools.\n",
    "\n",
    "- You can use *multiple metrics* for Grid & Randomized searches. They are specified as a list of strings, or a name:function dict.\n",
    "\n",
    "- When specifying multiple metrics, set `refit` to the metric for which `best_params_` will be used to build the `best_estimator_` for the entire dataset. Otherwise set `refit=False` - the default `refit=None` setting will cause errors when using multiple metrics.\n",
    "\n",
    "- **Parallelism**: param search tools evaluate each data fold independently. Computations can run in parallel vis `n_jobs=-1`.\n",
    "\n",
    "- **Robustness**: some settings can cause `fit` failures. This will cause the entire search to fail by default. Use `error_score=0` or `error_score=np.NaN` to instead issue a warning & set the score for that fold to zero or NaN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Composite Estimators & Parameter Spaces\n",
    "\n",
    "- Grid & Randomized parameter search can use composite or nested estimators (Pipelines, Column Transformers, Voting Classifiers, Calibrated Classifiers). A dedicated syntax is involved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.calibration import CalibratedClassifierCV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X, y = make_moons()\n",
    "\n",
    "calibrated_forest = CalibratedClassifierCV(\n",
    "   base_estimator=RandomForestClassifier(n_estimators=10))\n",
    "\n",
    "param_grid = {\n",
    "   'base_estimator__max_depth': [2, 4, 6, 8]}\n",
    "\n",
    "search = GridSearchCV(calibrated_forest, \n",
    "                      param_grid, \n",
    "                      cv=5).fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- If the meta-estimator is build as a collection (ie, Pipeline), `estimator` refers to the estimator name. There can be multiple nesting levels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "\n",
    "pipe = Pipeline([\n",
    "   ('select', SelectKBest()),\n",
    "   ('model', calibrated_forest)])\n",
    "\n",
    "param_grid = {\n",
    "   'select__k': [1, 2],\n",
    "   'model__base_estimator__max_depth': [2, 4, 6, 8]}\n",
    "\n",
    "search = GridSearchCV(pipe, param_grid, cv=5).fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative to Brute-Force Search\n",
    "\n",
    "- Some models can fit data for a range of param values almost as efficiently as fitting an estimator for a single param value. The most common example of this is *regularization strength*. The following models have this capability.\n",
    "\n",
    "- [ElasticNet CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html#sklearn.linear_model.ElasticNetCV)\n",
    "- [LARS CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LarsCV.html#sklearn.linear_model.LarsCV)\n",
    "- [Lasso CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV)\n",
    "- [LassoLARS CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsCV.html#sklearn.linear_model.LassoLarsCV)\n",
    "- [Logistic Regression CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV)\n",
    "- [ElasticNet (Multitask) CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.MultiTaskElasticNetCV.html#sklearn.linear_model.MultiTaskElasticNetCV)\n",
    "- [Lasso (Multitask) CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.MultiTaskLassoCV.html#sklearn.linear_model.MultiTaskLassoCV)\n",
    "- [OMP CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.OrthogonalMatchingPursuitCV.html#sklearn.linear_model.OrthogonalMatchingPursuitCV)\n",
    "- [Ridge CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV)\n",
    "- [Ridge Classifier CV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifierCV.html#sklearn.linear_model.RidgeClassifierCV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Info Criteria (AIC, BIC) based Regularization\n",
    "\n",
    "- [LassoLARS IC](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsIC.html#sklearn.linear_model.LassoLarsIC) uses either the AIC or BIC info criterions to build optimal regularization estimates with a single regularization path (instead of building several via cross-validation.) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Example: Model comparison w/ Grid Search](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_stats.html#sphx-glr-auto-examples-model-selection-plot-grid-search-stats-py)\n",
    "\n",
    "- TODO"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:working] *",
   "language": "python",
   "name": "conda-env-working-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}