code_recsys/capstone_book3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cecdf254-1efc-4d13-a911-67f2059d922e",
   "metadata": {},
   "source": [
    "# Capstone Project: Wine Recommender\n",
    "---\n",
    "Book 1: Problem Statement & Data Cleaning<br>\n",
    "Book 2: Exploratory Data Analysis, Preprocessing & Feature Engineering<br>\n",
    "**Book 3: Modelling, Conclusion & Recommendation**<br>\n",
    "Author: Lee Wan Xian"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d12e31d9-2117-464f-bca0-880f6d30188f",
   "metadata": {},
   "source": [
    "## Contents:\n",
    "- [Modelling](#Modelling)\n",
    "- [Model Evaluation](#Model-Evaluation)\n",
    "- [Conclusion](#Conclusion)\n",
    "- [Recommendation](#Recommendation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5196f137-7846-4a7a-8a51-fc81ddd1edfd",
   "metadata": {},
   "source": [
    "## Python Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "273a7df8-f006-461d-9396-7810637bf0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "from random import sample, choice, seed\n",
    "\n",
    "from surprise import Dataset, Reader, accuracy\n",
    "from surprise.model_selection import cross_validate, KFold, GridSearchCV\n",
    "from surprise import NormalPredictor, BaselineOnly, KNNBasic, KNNWithMeans, KNNWithZScore, KNNBaseline, SVD, NMF, SlopeOne, CoClustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f2cea6a3-243c-4085-ba02-2b35ab1d4950",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this setting widens pandas column to fit 400 characters\n",
    "pd.options.display.max_colwidth = 400"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405bc0fa-2098-4632-8b8d-f67c1bc3671b",
   "metadata": {},
   "source": [
    "## Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfae9470-c6eb-417b-a8aa-ed523277cf1f",
   "metadata": {},
   "source": [
    "### Model Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "445d9bf0-289c-4334-887b-8886504ea994",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import wine dataframes\n",
    "df_wine_model = pd.read_pickle('../data/df_wine_us_rate.pkl')\n",
    "df_wine_combi = pd.read_pickle('../data/df_wine_combi.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3452fd04-5415-42b0-82a7-91ed653a115c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the parameters for reader, data, algorithms, KFold strategy\n",
    "kf = KFold(random_state=42)\n",
    "reader = Reader(rating_scale=(88, 100))\n",
    "data = Dataset.load_from_df(df_wine_model, reader)\n",
    "algors = {\n",
    "    'Normal Predictor': NormalPredictor(),\n",
    "    'Baseline Predictor': BaselineOnly(),\n",
    "    'KNN Basic': KNNBasic(),\n",
    "    'KNN Means': KNNWithMeans(),\n",
    "    'KNN ZScore': KNNWithZScore(),\n",
    "    'KNN Baseline': KNNBaseline(),\n",
    "    'SVD': SVD(random_state=42),\n",
    "    'NonNegative Matrix Factorization': NMF(random_state=42),\n",
    "    'Slope One': SlopeOne(),\n",
    "    'Co-clustering': CoClustering(random_state=42)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9150b640-4a64-4992-934e-9e01952d63f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 15572 unique wines in data\n",
      "There are 10 unique tasters in data\n"
     ]
    }
   ],
   "source": [
    "# Create lists of unique wines & wine tasters\n",
    "list_wines = df_wine_model['title'].unique().tolist()\n",
    "list_tasters = df_wine_model['taster_name'].unique().tolist()\n",
    "\n",
    "# Show unique count of wine & wine tasters\n",
    "print(f'There are {len(list_wines)} unique wines in data')\n",
    "print(f'There are {len(list_tasters)} unique tasters in data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41f74518-0b8e-4e1d-b85c-927ed62e46e0",
   "metadata": {},
   "source": [
    "In our model dataset, we have 10 unique tasters, 15572 unique wines, and ratings given for each wine. Given that the dataset has explicit ratings, we can use [Scikit-Surprise](https://surpriselib.com/) library to train our recommender system model.<br>\n",
    "\n",
    "Relevant wine definition: We will deem that relevant wines have a points rating of 90 and above. This is because the rating of 90 is right below the median point of our dataset and the recommender should recommend wines that are rated better than or equivalent to the median point."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fcd89c2-194a-445b-b1ec-ca2c7fb145d8",
   "metadata": {},
   "source": [
    "### Baseline Model: Pure Randomized Recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e8495b8-a1c8-452a-8913-1df085d67472",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average precision@k is 0.36\n",
      "Average recall@k is 0.000364\n"
     ]
    }
   ],
   "source": [
    "# Create the baseline model & generate Precision@k and Recall@k results\n",
    "\n",
    "k=10                # Represents the number of recommendations to return\n",
    "precision = []      # Precision@k value list using random experiments\n",
    "recall = []         # Recall@k value list using random experiments\n",
    "\n",
    "for i in range(len(list_tasters)):\n",
    "    seed(42)\n",
    "    random_user = choice(list_tasters)\n",
    "    relevant_wines = df_wine_model[(df_wine_model['taster_name']!=random_user) & (df_wine_model['points']>=90)]['title'].unique().tolist()\n",
    "    count_relevant_wines = len(relevant_wines)\n",
    "    \n",
    "    # Since all titles can be selected for pure randomized recommender model, we select a random k titles\n",
    "    random_10_wines = sample(list_wines, k)    \n",
    "    rec_correct_wines = []\n",
    "    for wine in random_10_wines:\n",
    "        if wine in relevant_wines:\n",
    "            rec_correct_wines.append(wine)\n",
    "\n",
    "    if count_relevant_wines == 0:\n",
    "        recall.append(0)\n",
    "    else:\n",
    "        recall.append(len(rec_correct_wines)/count_relevant_wines)\n",
    "    \n",
    "    precision.append(len(rec_correct_wines)/k)\n",
    "    \n",
    "    list_tasters.remove(random_user)\n",
    "\n",
    "precision_at_k = sum(precision)/len(precision)\n",
    "recall_at_k = sum(recall)/len(recall)\n",
    "\n",
    "print(f'Average precision@k is {round(precision_at_k, 6)}')\n",
    "print(f'Average recall@k is {round(recall_at_k, 6)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "009a30c0-1d9b-404a-9ed4-d10bdd11b0f3",
   "metadata": {},
   "source": [
    "Due to the nature of a randomized recommender model generating different recommendations in each run, the RMSE could range from $0$ to $\\infty$. Thus, it is impossible to calculate a RMSE value with small variation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c240fbce-fb7b-46f3-9191-184ad19135de",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Other Model Algorithm Trials"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6987995a-7ef6-446d-9188-fc79af2f7abf",
   "metadata": {},
   "source": [
    "The Surprise library provides an array of in-built [algorithms](https://surprise.readthedocs.io/en/stable/prediction_algorithms_package.html) that can serve as the model for recommender system.<br>\n",
    "For the purpose of this project, we will test out the below algorithms and choose the most suitable one based on Root Mean Squared Error (RMSE) and Precision@k.\n",
    "1. `Baseline Predictor`: Basic algorithm predicting the baseline estimate for given user and item.\n",
    "2. `Normal Predictor`: Basic algorithm predicting a random rating based on the distribution of the training set, which is assumed to be normal.\n",
    "3. `KNN Basic`: A basic collaborative filtering k-NN based algorithm.\n",
    "4. `KNN with Means`: A basic collaborative filtering k-NN based algorithm, taking into account the mean ratings of each user.\n",
    "5. `KNN with Z score`: A basic collaborative filtering k-NN based algorithm, taking into account the z-score normalization of each user.\n",
    "6. `KNN Baseline`: A basic collaborative filtering k-NN based algorithm taking into account a baseline rating.\n",
    "7. `SVD`: The famous Singular Value Decomposition algorithm which is a Matrix Factorization based algorithm.\n",
    "8. `NonNegative Matrix Factorization`: A collaborative filtering algorithm based on Non-negative Matrix Factorization.\n",
    "9. `Slope One`: A simple yet accurate collaborative filtering algorithm.\n",
    "10. `Co-clustering`: A collaborative filtering algorithm based on co-clustering.\n",
    "\n",
    "RMSE is calculated based on comparing the predicted rating to the true rating for each taster-wine pair with a known label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "68bcd23b-4630-48a9-a1b0-b73a60e1ad24",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:28<00:00,  2.84s/it]\n"
     ]
    }
   ],
   "source": [
    "# Run multiple tests on algorithms list for RMSE results (Similarity used is Mean Squared Difference) \n",
    "msd_rmse_results = []\n",
    "\n",
    "for algo_name, algo in tqdm(algors.items()):\n",
    "    algo_results = []\n",
    "    results = cross_validate(algo, data, measures=['rmse'], cv=kf, n_jobs=-1)\n",
    "    algo_results.append(algo_name)\n",
    "    algo_results.append(results['test_rmse'].mean())\n",
    "    msd_rmse_results.append(algo_results)\n",
    "    \n",
    "# Generate the RMSE results for all algors in dataframe\n",
    "msd_results_df = pd.DataFrame(msd_rmse_results, columns = ['algorithm_name', 'ave_rmse_score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2abd45-2c4d-4319-bcb5-1fbd912e85cb",
   "metadata": {},
   "source": [
    "The formula of Precision@k and Recall@k is based on this [documentation](https://surprise.readthedocs.io/en/stable/FAQ.html#how-to-compute-precision-k-and-recall-k). As stated earlier, we have set the threshold for defining relevant wines as those with ratings above or equal to 90. As for k, we will set it as 10 as it's expected that our recommender system is to recommend 10 wines.<br>\n",
    "In simpler terms, Precision@k is the proportion of recommended wines in the top-k set that are relevant. The higher the Precision@k, the more relevant rated wines will appear in the recommendations. Recall@k is the proportion of relevant wines found in the top-k recommendations. It is hard to define the behaviour of Recall@k as it is susceptible to the number of relevant wines in the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66206b28-87b5-4b87-af9c-f945abefb801",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for calculating precision@k & recall@k (https://surprise.readthedocs.io/en/stable/FAQ.html#how-to-compute-precision-k-and-recall-k)\n",
    "def precision_recall_at_k(predictions, threshold, k=10):\n",
    "    \"\"\"Return precision and recall at k metrics for each user\"\"\"\n",
    "\n",
    "    # First map the predictions to each user.\n",
    "    user_est_true = defaultdict(list)\n",
    "    for uid, _, true_r, est, _ in predictions:\n",
    "        user_est_true[uid].append((est, true_r))\n",
    "\n",
    "    precisions = dict()\n",
    "    recalls = dict()\n",
    "    for uid, user_ratings in user_est_true.items():\n",
    "\n",
    "        # Sort user ratings by estimated value\n",
    "        user_ratings.sort(key=lambda x: x[0], reverse=True)\n",
    "\n",
    "        # Number of relevant items\n",
    "        n_rel = sum((true_r >= threshold) for (_, true_r) in user_ratings)\n",
    "\n",
    "        # Number of recommended items in top k\n",
    "        n_rec_k = sum((est >= threshold) for (est, _) in user_ratings[:k])\n",
    "\n",
    "        # Number of relevant and recommended items in top k\n",
    "        n_rel_and_rec_k = sum(((true_r >= threshold) and (est >= threshold)) for (est, true_r) in user_ratings[:k])\n",
    "\n",
    "        # Precision@K: Proportion of recommended items that are relevant\n",
    "        # When n_rec_k is 0, Precision is undefined. We here set it to 0.\n",
    "        precisions[uid] = n_rel_and_rec_k / n_rec_k if n_rec_k != 0 else 0\n",
    "\n",
    "        # Recall@K: Proportion of relevant items that are recommended\n",
    "        # When n_rel is 0, Recall is undefined. We here set it to 0.\n",
    "        recalls[uid] = n_rel_and_rec_k / n_rel if n_rel != 0 else 0\n",
    "\n",
    "    return precisions, recalls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4d65b60d-a5cf-4ea5-9f80-c2d68ffb25b8",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|████████▎                                                                          | 1/10 [00:00<00:05,  1.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n",
      "Estimating biases using als...\n",
      "Estimating biases using als...\n",
      "Estimating biases using als...\n",
      "Estimating biases using als...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|████████████████▌                                                                  | 2/10 [00:01<00:05,  1.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 30%|████████████████████████▉                                                          | 3/10 [00:01<00:03,  1.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|█████████████████████████████████▏                                                 | 4/10 [00:02<00:03,  1.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|█████████████████████████████████████████▌                                         | 5/10 [00:02<00:02,  1.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 60%|█████████████████████████████████████████████████▊                                 | 6/10 [00:03<00:02,  1.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "Estimating biases using als...\n",
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:48<00:00,  4.83s/it]\n"
     ]
    }
   ],
   "source": [
    "# Run multiple tests on list of algorithms for Precision@k & Recall@k results\n",
    "all_precision_recall = []\n",
    "\n",
    "for algo_name, algo in tqdm(algors.items()):\n",
    "    algo_precision_recall_list = []\n",
    "    precision_list = []\n",
    "    recall_list = []\n",
    "    \n",
    "    for trainset, testset in kf.split(data):\n",
    "        algo.fit(trainset)\n",
    "        predictions = algo.test(testset)\n",
    "        precisions, recalls = precision_recall_at_k(predictions, threshold=90)\n",
    "        precision_list.append(sum(prec for prec in precisions.values()) / len(precisions))\n",
    "        recall_list.append(sum(rec for rec in recalls.values()) / len(recalls))\n",
    "    \n",
    "    precision_ave = sum(precision_list)/len(precision_list)\n",
    "    recall_ave = sum(recall_list)/len(recall_list)\n",
    "    algo_precision_recall_list.append(algo_name)\n",
    "    algo_precision_recall_list.append(precision_ave)\n",
    "    algo_precision_recall_list.append(recall_ave)\n",
    "    all_precision_recall.append(algo_precision_recall_list)\n",
    "    \n",
    "# Generate the results for all algors in dataframe\n",
    "all_precision_recall_df = pd.DataFrame(all_precision_recall, columns = ['algorithm_name', 'ave_precision@k_score', 'ave_recall@k_score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "44d70ab7-58a5-4221-b9a7-dce5b3f6af77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm_name</th>\n",
       "      <th>ave_rmse_score</th>\n",
       "      <th>ave_precision@k_score</th>\n",
       "      <th>ave_recall@k_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Baseline Predictor</td>\n",
       "      <td>1.921808</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>KNN Basic</td>\n",
       "      <td>1.853119</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>KNN Means</td>\n",
       "      <td>1.853373</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>KNN ZScore</td>\n",
       "      <td>1.853372</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>KNN Baseline</td>\n",
       "      <td>1.828589</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Slope One</td>\n",
       "      <td>1.853719</td>\n",
       "      <td>0.751706</td>\n",
       "      <td>0.215194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Co-clustering</td>\n",
       "      <td>1.858452</td>\n",
       "      <td>0.749706</td>\n",
       "      <td>0.215124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SVD</td>\n",
       "      <td>1.830108</td>\n",
       "      <td>0.659683</td>\n",
       "      <td>0.091666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NonNegative Matrix Factorization</td>\n",
       "      <td>2.123324</td>\n",
       "      <td>0.582563</td>\n",
       "      <td>0.209640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Normal Predictor</td>\n",
       "      <td>2.708486</td>\n",
       "      <td>0.510452</td>\n",
       "      <td>0.106167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     algorithm_name  ave_rmse_score  ave_precision@k_score  \\\n",
       "1                Baseline Predictor        1.921808               0.751706   \n",
       "2                         KNN Basic        1.853119               0.751706   \n",
       "3                         KNN Means        1.853373               0.751706   \n",
       "4                        KNN ZScore        1.853372               0.751706   \n",
       "5                      KNN Baseline        1.828589               0.751706   \n",
       "8                         Slope One        1.853719               0.751706   \n",
       "9                     Co-clustering        1.858452               0.749706   \n",
       "6                               SVD        1.830108               0.659683   \n",
       "7  NonNegative Matrix Factorization        2.123324               0.582563   \n",
       "0                  Normal Predictor        2.708486               0.510452   \n",
       "\n",
       "   ave_recall@k_score  \n",
       "1            0.215194  \n",
       "2            0.215194  \n",
       "3            0.215194  \n",
       "4            0.215194  \n",
       "5            0.215194  \n",
       "8            0.215194  \n",
       "9            0.215124  \n",
       "6            0.091666  \n",
       "7            0.209640  \n",
       "0            0.106167  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compile the RMSE, Precision@k & Recall@k results together\n",
    "df_mod_eval = msd_results_df.join(all_precision_recall_df.set_index('algorithm_name'), on='algorithm_name')\n",
    "df_mod_eval.sort_values('ave_precision@k_score', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86521b8f-9378-4f7d-af03-036bd3b84565",
   "metadata": {},
   "source": [
    "So far, **KNN Baseline** has performed better than the rest as it has the highest average Precision@k score and the lowest average RMSE score. Thus, we will continue with the KNN Baseline model for subsequent steps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "392153d2-b6c3-420a-aa06-ee6ed6037ef3",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Hyperparameter Tuning\n",
    "\n",
    "We will perform hyperparameter tuning on the KNN Baseline model to assess whether any improvements can be achieved. We also will adjust the threshold for relevant wines to assess whether the actual ratings threshold of 90 points is optimal or not."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d37d0eac-9ad7-4ce7-88a3-9c334788369e",
   "metadata": {},
   "source": [
    "#### Tuning on Model Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "154bc4a1-375f-48f3-933f-c31ac144fb2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the hyper params for KNN Baseline\n",
    "param_grid_knn = {\n",
    "    'k': [35, 40, 45],\n",
    "    'min_k': [1, 3, 5],\n",
    "    'sim_options': {\n",
    "        'name': ['msd', 'cosine', 'pearson_baseline']\n",
    "    },\n",
    "    'verbose': [False]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c5db67ec-d63c-40a3-bb1d-2639bb786862",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best RMSE score of tuned KNN model is 1.828586\n",
      "Parameters of the tuned KNN model are {'k': 35, 'min_k': 1, 'sim_options': {'name': 'cosine', 'user_based': True}, 'verbose': False}\n"
     ]
    }
   ],
   "source": [
    "# Instantiate & fit the tuned KNN Baseline model\n",
    "grid_knn = GridSearchCV(KNNBaseline, param_grid_knn, measures=[\"rmse\"], cv=kf, n_jobs=-1)\n",
    "grid_knn.fit(data)\n",
    "\n",
    "print(f'Best RMSE score of tuned KNN model is {round(grid_knn.best_score[\"rmse\"],6)}')\n",
    "print(f'Parameters of the tuned KNN model are {grid_knn.best_params[\"rmse\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "397eb69a-672d-47a5-945b-2c7afc3bb967",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Precision@k for tuned KNN model is 0.751706\n",
      "Mean Recall@k for tuned KNN model is 0.215194\n"
     ]
    }
   ],
   "source": [
    "# Obtain Precision@k & Recall@k for tuned KNN Baseline model\n",
    "best_knn = grid_knn.best_estimator['rmse']\n",
    "precision_list = []\n",
    "recall_list = []\n",
    "\n",
    "for trainset, testset in kf.split(data):\n",
    "    best_knn.fit(trainset)\n",
    "    predictions = best_knn.test(testset)\n",
    "    precisions, recalls = precision_recall_at_k(predictions, threshold=90)\n",
    "    precision_list.append(sum(prec for prec in precisions.values()) / len(precisions))\n",
    "    recall_list.append(sum(rec for rec in recalls.values()) / len(recalls))\n",
    "    \n",
    "precision_ave = sum(precision_list)/len(precision_list)\n",
    "recall_ave = sum(recall_list)/len(recall_list)\n",
    "\n",
    "print(f'Mean Precision@k for tuned KNN model is {round(precision_ave, 6)}')\n",
    "print(f'Mean Recall@k for tuned KNN model is {round(recall_ave, 6)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "597573c7-d738-4c57-b99e-f09251987cbb",
   "metadata": {},
   "source": [
    "#### Tuning on Actual Ratings Threshold for Relevant Wines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df3cfc87-8bee-4bf4-a2c3-6f3f5cc87222",
   "metadata": {},
   "source": [
    "We will test out how the changes in the threshold will affect the Precision@k and Recall@k metrics. Based on the [definitions](https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54) and our dataset, the possible range of Precision@k is from 0 (when no relevant wines are in the 10 recommendations) to 1 (when the 10 recommendations are all relevant wines).<br>\n",
    "For this project, we will compute the Precision@k and Recall@k when threshold is set at 89 and 91 points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "66d8e394-63e9-4305-88c5-eb5242731582",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "At threshold = 89, mean Precision@k is 0.892143\n",
      "At threshold = 89, mean Recall@k is 0.313805\n"
     ]
    }
   ],
   "source": [
    "# Compute Precision@k & Recall@k when Threshold = 89\n",
    "precision_list = []\n",
    "recall_list = []\n",
    "\n",
    "for trainset, testset in kf.split(data):\n",
    "    best_knn.fit(trainset)\n",
    "    predictions = best_knn.test(testset)\n",
    "    precisions, recalls = precision_recall_at_k(predictions, threshold=89)\n",
    "    precision_list.append(sum(prec for prec in precisions.values()) / len(precisions))\n",
    "    recall_list.append(sum(rec for rec in recalls.values()) / len(recalls))\n",
    "    \n",
    "precision_ave = sum(precision_list)/len(precision_list)\n",
    "recall_ave = sum(recall_list)/len(recall_list)\n",
    "\n",
    "print(f'At threshold = 89, mean Precision@k is {round(precision_ave, 6)}')\n",
    "print(f'At threshold = 89, mean Recall@k is {round(recall_ave, 6)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "77700428-9511-45f7-88c8-21da43fe7b52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "At threshold = 91, mean Precision@k is 0.633214\n",
      "At threshold = 91, mean Recall@k is 0.02535\n"
     ]
    }
   ],
   "source": [
    "# Compute Precision@k & Recall@k when Threshold = 91\n",
    "precision_list = []\n",
    "recall_list = []\n",
    "\n",
    "for trainset, testset in kf.split(data):\n",
    "    best_knn.fit(trainset)\n",
    "    predictions = best_knn.test(testset)\n",
    "    precisions, recalls = precision_recall_at_k(predictions, threshold=91)\n",
    "    precision_list.append(sum(prec for prec in precisions.values()) / len(precisions))\n",
    "    recall_list.append(sum(rec for rec in recalls.values()) / len(recalls))\n",
    "    \n",
    "precision_ave = sum(precision_list)/len(precision_list)\n",
    "recall_ave = sum(recall_list)/len(recall_list)\n",
    "\n",
    "print(f'At threshold = 91, mean Precision@k is {round(precision_ave, 6)}')\n",
    "print(f'At threshold = 91, mean Recall@k is {round(recall_ave, 6)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fec8ea3-f9ad-4742-b06e-ebac314d6746",
   "metadata": {},
   "source": [
    "When the threshold is at 89, the benchmark for wines to be considered relevant is very low. About 80% of wines are considered relevant. It causes the recommender system to be too lenient and recommend wines that might not be of good quality. For instance, wines with an actual rating of 89 can end up in the top 10 recommendations.<br>\n",
    "When the threshold is at 91, the benchmark for wines to be considered relevant might be too high. About 56% of wines are considered relevant. It might cause the recommender system to recommend the same set of wines for every run or be unable to recommend wines with specific traits.<br>\n",
    "Thus, setting the threshold at 90 is optimal for this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19475085-d41c-49e4-a445-57f75be89ee7",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Model Prediction Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f40b769-578d-4804-9f79-795407a944b8",
   "metadata": {},
   "source": [
    "For this section, we will review the top 10 recommendations given by the chosen model for a new user. We will compare the estimated ratings of recommended wines to their actual rating to critique the model's capability to provide similar recommendations.\n",
    "\n",
    "Now, we will review the top 10 recommendations provided by the model if a new user is indifferent of wine traits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b2b6d767-d7ee-4d61-9dd9-879fa51ea2f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n",
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.knns.KNNBaseline at 0x286b541dd00>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the params for best model\n",
    "sim_options = {'name': 'cosine'}\n",
    "chosen_model = KNNBaseline(k=35, min_k=1, sim_options=sim_options)\n",
    "\n",
    "# Prepare the whole dataset as the trainset\n",
    "train = data.build_full_trainset()\n",
    "\n",
    "# Fir the model with the whole dataset\n",
    "chosen_model.fit(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b118a1bc-e7ef-4390-bf7f-1cd472e4ae4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dataframe with the estimate match score (predicted ratings)\n",
    "recommend_list = []\n",
    "user_wines = df_wine_model[df_wine_model['taster_name'] == 'mock_user']['title'].unique()\n",
    "not_user_wines = []\n",
    "for wine in df_wine_model['title'].unique():\n",
    "    if wine not in user_wines:\n",
    "        not_user_wines.append(wine)\n",
    "\n",
    "for wine in not_user_wines:\n",
    "    wine_compatibility = []\n",
    "    prediction = chosen_model.predict(uid='mock_user', iid=wine)\n",
    "    wine_compatibility.append(prediction.iid)\n",
    "    wine_compatibility.append(prediction.est)\n",
    "    recommend_list.append(wine_compatibility)\n",
    "\n",
    "recommend_df = pd.DataFrame(recommend_list, columns = ['title', 'est_match_pts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "faa27d76-06da-4962-9fc2-c7edd298850b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>est_match_pts</th>\n",
       "      <th>points</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Charles Smith 2006 Royal City Syrah (Columbia Valley (WA))</td>\n",
       "      <td>91.557454</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Cayuse 2008 Bionic Frog Syrah (Walla Walla Valley (WA))</td>\n",
       "      <td>91.557454</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K Vintners 2013 The Hidden Northridge Vineyard Syrah (Wahluke Slope)</td>\n",
       "      <td>91.523150</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cayuse 2011 En Chamberlin Vineyard Syrah (Walla Walla Valley (OR))</td>\n",
       "      <td>91.466545</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Cayuse 2009 En Chamberlin Vineyard Syrah (Walla Walla Valley (OR))</td>\n",
       "      <td>91.466545</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Alpha Omega 2012 Stagecoach Vineyard Cabernet Sauvignon (Atlas Peak)</td>\n",
       "      <td>91.448946</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Alpha Omega 2012 ERA Red (Napa Valley)</td>\n",
       "      <td>91.448946</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Doyenne 2008 Grand Ciel Vineyard Syrah (Red Mountain)</td>\n",
       "      <td>91.429456</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Williams Selyem 2012 Eastside Road Neighbors Pinot Noir (Russian River Valley)</td>\n",
       "      <td>91.397192</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Dutton-Goldfield 2013 Dutton Ranch Cherry Ridge Vineyard Syrah (Russian River Valley)</td>\n",
       "      <td>91.397192</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                    title  \\\n",
       "0                              Charles Smith 2006 Royal City Syrah (Columbia Valley (WA))   \n",
       "1                                 Cayuse 2008 Bionic Frog Syrah (Walla Walla Valley (WA))   \n",
       "2                    K Vintners 2013 The Hidden Northridge Vineyard Syrah (Wahluke Slope)   \n",
       "4                      Cayuse 2011 En Chamberlin Vineyard Syrah (Walla Walla Valley (OR))   \n",
       "5                      Cayuse 2009 En Chamberlin Vineyard Syrah (Walla Walla Valley (OR))   \n",
       "6                    Alpha Omega 2012 Stagecoach Vineyard Cabernet Sauvignon (Atlas Peak)   \n",
       "7                                                  Alpha Omega 2012 ERA Red (Napa Valley)   \n",
       "8                                   Doyenne 2008 Grand Ciel Vineyard Syrah (Red Mountain)   \n",
       "10         Williams Selyem 2012 Eastside Road Neighbors Pinot Noir (Russian River Valley)   \n",
       "12  Dutton-Goldfield 2013 Dutton Ranch Cherry Ridge Vineyard Syrah (Russian River Valley)   \n",
       "\n",
       "    est_match_pts  points  \n",
       "0       91.557454     100  \n",
       "1       91.557454     100  \n",
       "2       91.523150      95  \n",
       "4       91.466545      99  \n",
       "5       91.466545      99  \n",
       "6       91.448946      99  \n",
       "7       91.448946      99  \n",
       "8       91.429456      95  \n",
       "10      91.397192      95  \n",
       "12      91.397192      95  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return top10 recommended wines\n",
    "chosen_rec = recommend_df.sort_values('est_match_pts', ascending=False).head(10)\n",
    "chosen = chosen_rec.merge(df_wine_model, on='title', how='inner')\n",
    "chosen[['title', 'est_match_pts', 'points']].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "13c3863d-8e73-4097-96f1-715ef3e7fd42",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAHFCAYAAAAT5Oa6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFK0lEQVR4nO3de1hVdd7//9eWswQ7ATklopYynm20AJtS8lyKZTNaNqT3mDqZlmOOk9kkNd1aNqM2ODnVmJaHcA5q3tpQmmiZeExKHcesNE8gpgioBCif7x/9WD+3CCIBG13Px3Wt63Kvz3ut9V5bXLxch70dxhgjAAAAG2vg7gYAAADcjUAEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEVNOCBQvkcDisydfXV+Hh4UpISND06dOVk5NTbpnk5GQ5HI6r2s65c+eUnJys9evXX9Vyl9tWs2bN1L9//6taz5UsWbJEs2fPvuyYw+FQcnJyjW6vpn300Ufq0qWL/P395XA4tGLFinI1J06cUIMGDfTYY4+VG3vyySflcDg0efLkcmMjRoyQh4eHcnNzJbnv/Vi/fr3Lz6qHh4caN26sAQMGaPv27dVe72uvvaYFCxaUm3/w4EE5HI7LjgH1lae7GwCudfPnz9dPfvITlZSUKCcnRxs3btTLL7+sP/7xj1q6dKl69uxp1T766KPq27fvVa3/3Llzev755yVJ3bt3r/Jy1dlWdSxZskS7d+/W+PHjy41lZGSoSZMmtd5DdRljNHjwYLVq1UorV66Uv7+/YmJiytU1btxYbdu2VXp6ermx9evXy9/fv8KxTp06qVGjRpLc/35MmzZNCQkJKikp0c6dO/X888+rW7duyszMVMuWLa96fa+99ppCQkI0fPhwl/kRERHKyMjQzTffXEOdA7WPM0TAj9SuXTvFxcXpzjvv1AMPPKBZs2bpiy++kL+/vwYNGqTjx49btU2aNFFcXFyt9nPu3Lk629aVxMXF1etAdOzYMZ06dUr333+/evToobi4OCu8XCohIUH79u1Tdna2Ne/UqVPatWuXHnvsMe3YsUMFBQXW2JEjR/TNN98oISHBmufu96Nly5bWz+oTTzyhWbNm6dy5c1q0aFGNbsfHx0dxcXFq3Lhxja4XqE0EIqAWNG3aVH/6059UUFCg119/3Zp/uctY69atU/fu3RUcHCw/Pz81bdpUDzzwgM6dO6eDBw9av1Sef/5565JH2f/Iy9b32Wef6ec//7kaNWpk/a+8sstzy5cvV4cOHeTr66sWLVroz3/+s8t42eXAgwcPuswvu/RSdvmue/fuWr16tb799luXSzJlLneJaPfu3Ro4cKAaNWokX19fderUSW+//fZlt/Puu+9qypQpioyMVGBgoHr27Kl9+/ZV/MZfZOPGjerRo4cCAgLUsGFDde3aVatXr7bGk5OTrXDyu9/9Tg6HQ82aNatwfWXB5uJLlxs2bJCnp6cmTpwoSfrkk0+ssbIzRhcHokvfj7L3OT09XY899phCQkIUHBysQYMG6dixY+V6WLp0qeLj4+Xv768bbrhBffr00c6dO6v0flxOly5dJMkltEs//KzFxsYqKChIgYGB+ulPf6p58+bp4u8Cb9asmfbs2aMNGzZYf+9l79/lLpmV/Tzu2bNHDz30kJxOp8LCwvSrX/1KeXl5Lts/ffq0RowYoaCgIN1www2699579c0335R7/06cOKFRo0YpKipKPj4+aty4se644w6tXbu22u8J7ItABNSSe+65Rx4eHvr4448rrDl48KDuvfdeeXt766233lJaWppeeukl+fv7q7i4WBEREUpLS5P0w/0oGRkZysjI0O9//3uX9QwaNEi33HKL/vGPf+ivf/1rpX1lZmZq/Pjx+s1vfqPly5era9euevLJJ/XHP/7xqvfxtdde0x133KHw8HCrt4yMjArr9+3bp65du2rPnj3685//rGXLlqlNmzYaPny4ZsyYUa7+mWee0bfffqu//e1veuONN7R//34NGDBAFy5cqLSvDRs26O6771ZeXp7mzZund999VwEBARowYICWLl0q6YdLisuWLZMkjRs3ThkZGVq+fHmF6+zWrZsaNGjgcmksPT1dXbp0UVhYmDp37uwSltLT0+Xh4aE777yz0l7LevHy8tKSJUs0Y8YMrV+/Xr/85S9daqZNm6aHHnpIbdq00d///nctXLhQBQUFuvPOO/Wf//znitu4nAMHDkiSWrVq5TL/4MGDGj16tP7+979r2bJlGjRokMaNG6c//OEPVs3y5cvVokUL3Xrrrdbfe2XvX5kHHnhArVq10r/+9S89/fTTWrJkiX7zm99Y46WlpRowYICWLFmi3/3ud1q+fLliY2Mve/k3KSlJK1as0HPPPacPP/xQf/vb39SzZ0+dPHmyWu8HbM4AqJb58+cbSWbbtm0V1oSFhZnWrVtbr6dOnWou/mf3z3/+00gymZmZFa7jxIkTRpKZOnVqubGy9T333HMVjl0sOjraOByOctvr1auXCQwMNGfPnnXZtwMHDrjUpaenG0kmPT3dmnfvvfea6Ojoy/Z+ad8PPvig8fHxMYcOHXKp69evn2nYsKE5ffq0y3buuecel7q///3vRpLJyMi47PbKxMXFmdDQUFNQUGDNO3/+vGnXrp1p0qSJKS0tNcYYc+DAASPJvPLKK5Wur0ynTp1Mq1atrNft27c3Tz/9tDHGmEmTJpkuXbpYY82bNze33367y/KXvh9l7/OYMWNc6mbMmGEkmaysLGOMMYcOHTKenp5m3LhxLnUFBQUmPDzcDB48uNK+y97PpUuXmpKSEnPu3Dnz6aefmpiYGNOmTRuTm5tb4bIXLlwwJSUl5oUXXjDBwcHWe2eMMW3btjXdunUrt0zZ+zp//nxrXtnP44wZM1xqx4wZY3x9fa31rl692kgyc+fOdambPn16uffvhhtuMOPHj69034Gq4gwRUIvMRZcYLqdTp07y9vbWqFGj9Pbbb+ubb76p1nYeeOCBKte2bdtWHTt2dJk3dOhQ5efn67PPPqvW9qtq3bp16tGjh6KiolzmDx8+XOfOnSt3dikxMdHldYcOHSRJ3377bYXbOHv2rLZs2aKf//znuuGGG6z5Hh4eSkpK0pEjR6p82e1SCQkJ+vLLL3Xs2DGdPHlSu3fvtm5079atm3bu3Km8vDwdOnRIBw4ccLlcVpkr7ecHH3yg8+fP65FHHtH58+etydfXV926davyE4hDhgyRl5eXGjZsqDvuuEP5+flavXq1brzxRpe6devWqWfPnnI6nfLw8JCXl5eee+45nTx58rJPT16Ny+3r999/b613w4YNkqTBgwe71D300EPl1nX77bdrwYIFevHFF7V582aVlJT8qN5gbwQioJacPXtWJ0+eVGRkZIU1N998s9auXavQ0FA9/vjjuvnmm3XzzTfr1VdfvaptRUREVLk2PDy8wnm1fanh5MmTl+217D26dPvBwcEur318fCRJhYWFFW4jNzdXxpir2k5VXXwf0fr16+Xh4aE77rhDkvSzn/1M0g/3EV3u/qHKXGk/y+7xue222+Tl5eUyLV26VN99912VtvPyyy9r27Zt2rBhg6ZMmaLjx4/rvvvuU1FRkVWzdetW9e7dW5L05ptv6tNPP9W2bds0ZcoUl56q60r7evLkSXl6eiooKMilLiwsrNy6li5dqmHDhulvf/ub4uPjFRQUpEceecTlxnegqnjsHqglq1ev1oULF674qPydd96pO++8UxcuXND27duVkpKi8ePHKywsTA8++GCVtnU1n210uV8WZfPKfln5+vpKkssvSklV/sVbkeDgYGVlZZWbX3YDcUhIyI9avyQ1atRIDRo0qJXt3HXXXfLw8ND69evl4+Ojn/70p9ZZqMDAQHXq1Enp6ek6deqUPD09rbD0Y5X1+89//lPR0dHVXk+LFi2sG6nvuusu+fn56dlnn1VKSop1Y3hqaqq8vLy0atUq6+dA0mU/n6k2BAcH6/z58zp16pRLKLrcz21ISIhmz56t2bNn69ChQ1q5cqWefvpp5eTkWPfeAVXFGSKgFhw6dEgTJ06U0+nU6NGjq7SMh4eHYmNj9Ze//EWSrMtXVTkrcjX27Nmjzz//3GXekiVLFBAQoJ/+9KeSZD0t9MUXX7jUrVy5stz6fHx8qtxbjx49tG7dunJPUL3zzjtq2LBhjXxMgL+/v2JjY7Vs2TKXvkpLS7Vo0SI1adKk3E3EVeV0OnXrrbdaZ4guDbvdunVTenq61q9fr9tvv93lkt2P0adPH3l6eurrr79Wly5dLjtVx6RJk3TLLbfopZdesj4ywOFwyNPTUx4eHlZdYWGhFi5cWG75q/m7r6pu3bpJknXze5nU1NRKl2vatKnGjh2rXr161fqlX1yfOEME/Ei7d++27unIycnRJ598ovnz58vDw0PLly+v9LNY/vrXv2rdunW699571bRpU33//fd66623JMn6QMeAgABFR0frvffeU48ePRQUFKSQkJBKHxGvTGRkpBITE5WcnKyIiAgtWrRIa9as0csvv6yGDRtK+uHSTExMjCZOnKjz58+rUaNGWr58uTZu3Fhufe3bt9eyZcs0d+5cde7cWQ0aNKjwF/TUqVO1atUqJSQk6LnnnlNQUJAWL16s1atXa8aMGXI6ndXap0tNnz5dvXr1UkJCgiZOnChvb2+99tpr2r17t959992r/rTwiyUkJOiVV16Rw+HQyy+/7DLWrVs3zZo1S8YYPfzwwz92NyzNmjXTCy+8oClTpuibb75R37591ahRIx0/flxbt26Vv7+/9eGdV8PLy0vTpk3T4MGD9eqrr+rZZ5/Vvffeq5kzZ2ro0KEaNWqUTp48qT/+8Y9WML9Y+/btlZqaqqVLl6pFixby9fVV+/btf9S+9u3bV3fccYeeeuop5efnq3PnzsrIyNA777wjSWrQ4If/x+fl5SkhIUFDhw7VT37yEwUEBGjbtm1KS0vToEGDflQPsCk339QNXLPKnhAqm7y9vU1oaKjp1q2bmTZtmsnJySm3zKVPfmVkZJj777/fREdHGx8fHxMcHGy6detmVq5c6bLc2rVrza233mp8fHyMJDNs2DCX9Z04ceKK2zLmh6fM7r33XvPPf/7TtG3b1nh7e5tmzZqZmTNnllv+yy+/NL179zaBgYGmcePGZty4cdYTQBc/ZXbq1Cnz85//3Nx4443G4XC4bFOXeTpu165dZsCAAcbpdBpvb2/TsWNHl6eRjPn/n4r6xz/+4TL/ck8vVeSTTz4xd999t/H39zd+fn4mLi7O/N///d9l11fVp8yMMeb99983koyHh4fJy8tzGTt16pRp0KCBkWTWrFlTbtlL34+KnlS83NN8xhizYsUKk5CQYAIDA42Pj4+Jjo42P//5z83atWsr7bmi97NMbGysadSokfWU31tvvWViYmKMj4+PadGihZk+fbqZN29euScPDx48aHr37m0CAgKMJOtpw8qeMrv0Z/VyTzSeOnXK/M///I+58cYbTcOGDU2vXr3M5s2bjSTz6quvGmOM+f77782vf/1r06FDBxMYGGj8/PxMTEyMmTp1qvW0JHA1HMZc4TEYAADcbMmSJXr44Yf16aefqmvXru5uB9chAhEAoF559913dfToUbVv314NGjTQ5s2b9corr+jWW2+1HssHahr3EAEA6pWAgAClpqbqxRdf1NmzZxUREaHhw4frxRdfdHdruI5xhggAANgej90DAADbIxABAADbIxABAADb46bqKiotLdWxY8cUEBDwoz7UDQAA1B1jjAoKChQZGWl9sOflEIiq6NixY+W+oRsAAFwbDh8+rCZNmlQ4TiCqooCAAEk/vKGBgYFu7gbAde3sWSky8oc/Hzsm+fu7tx/gGpafn6+oqCjr93hFCERVVHaZLDAwkEAEoHZd9MWqCgwkEAE14Eq3u3BTNQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD1PdzcAVKRTl1hlZWVVOB4REaHM7VvqsCMAwPWKQIR6KysrSwlTl1Y4nv78kDrsBgBwPeOSGQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD0CEQAAsD23BqK5c+eqQ4cOCgwMVGBgoOLj4/Xvf//bGjfGKDk5WZGRkfLz81P37t21Z88el3UUFRVp3LhxCgkJkb+/vxITE3XkyBGXmtzcXCUlJcnpdMrpdCopKUmnT5+ui10EAADXALcGoiZNmuill17S9u3btX37dt19990aOHCgFXpmzJihmTNnas6cOdq2bZvCw8PVq1cvFRQUWOsYP368li9frtTUVG3cuFFnzpxR//79deHCBatm6NChyszMVFpamtLS0pSZmamkpKQ6318AAFA/OYwxxt1NXCwoKEivvPKKfvWrXykyMlLjx4/X7373O0k/nA0KCwvTyy+/rNGjRysvL0+NGzfWwoULNWTIEEnSsWPHFBUVpffff199+vTR3r171aZNG23evFmxsbGSpM2bNys+Pl7//e9/FRMTU6W+8vPz5XQ6lZeXp8DAwNrZebgIu6mpEqYurXA8/fkhOn70UB12VLFOXWKVlZVVaU1ERIQyt2+po45wTTt7Vrrhhh/+fOaM5O/v3n6Aa1hVf3971mFPlbpw4YL+8Y9/6OzZs4qPj9eBAweUnZ2t3r17WzU+Pj7q1q2bNm3apNGjR2vHjh0qKSlxqYmMjFS7du20adMm9enTRxkZGXI6nVYYkqS4uDg5nU5t2rSpyoEIqExWVlal4U36IcABAOontweiXbt2KT4+Xt9//71uuOEGLV++XG3atNGmTZskSWFhYS71YWFh+vbbbyVJ2dnZ8vb2VqNGjcrVZGdnWzWhoaHlthsaGmrVXE5RUZGKioqs1/n5+dXbQQAAUO+5/SmzmJgYZWZmavPmzXrsscc0bNgw/ec//7HGHQ6HS70xpty8S11ac7n6K61n+vTp1k3YTqdTUVFRVd0lAABwjXH7GSJvb2/dcsstkqQuXbpo27ZtevXVV637hrKzsxUREWHV5+TkWGeNwsPDVVxcrNzcXJezRDk5OeratatVc/z48XLbPXHiRLmzTxebPHmyJkyYYL3Oz8+vtVDE/ScAALiX2wPRpYwxKioqUvPmzRUeHq41a9bo1ltvlSQVFxdrw4YNevnllyVJnTt3lpeXl9asWaPBgwdL+uFejt27d2vGjBmSpPj4eOXl5Wnr1q26/fbbJUlbtmxRXl6eFZoux8fHRz4+PrW5qxbuPwEAwL3cGoieeeYZ9evXT1FRUSooKFBqaqrWr1+vtLQ0ORwOjR8/XtOmTVPLli3VsmVLTZs2TQ0bNtTQoUMlSU6nUyNGjNBTTz2l4OBgBQUFaeLEiWrfvr169uwpSWrdurX69u2rkSNH6vXXX5ckjRo1Sv379+eGagAAIMnNgej48eNKSkpSVlaWnE6nOnTooLS0NPXq1UuSNGnSJBUWFmrMmDHKzc1VbGysPvzwQwUEBFjrmDVrljw9PTV48GAVFhaqR48eWrBggTw8PKyaxYsX64knnrCeRktMTNScOXPqdmcBAEC95dZANG/evErHHQ6HkpOTlZycXGGNr6+vUlJSlJKSUmFNUFCQFi1aVN02AQDAdc7tT5kBAAC4G4EIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYnlsD0fTp03XbbbcpICBAoaGhuu+++7Rv3z6XmuHDh8vhcLhMcXFxLjVFRUUaN26cQkJC5O/vr8TERB05csSlJjc3V0lJSXI6nXI6nUpKStLp06drexcBAMA1wK2BaMOGDXr88ce1efNmrVmzRufPn1fv3r119uxZl7q+ffsqKyvLmt5//32X8fHjx2v58uVKTU3Vxo0bdebMGfXv318XLlywaoYOHarMzEylpaUpLS1NmZmZSkpKqpP9BAAA9ZunOzeelpbm8nr+/PkKDQ3Vjh07dNddd1nzfXx8FB4eftl15OXlad68eVq4cKF69uwpSVq0aJGioqK0du1a9enTR3v37lVaWpo2b96s2NhYSdKbb76p+Ph47du3TzExMbW0hwAA4FpQr+4hysvLkyQFBQW5zF+/fr1CQ0PVqlUrjRw5Ujk5OdbYjh07VFJSot69e1vzIiMj1a5dO23atEmSlJGRIafTaYUhSYqLi5PT6bRqLlVUVKT8/HyXCQAAXJ/qTSAyxmjChAn62c9+pnbt2lnz+/Xrp8WLF2vdunX605/+pG3btunuu+9WUVGRJCk7O1ve3t5q1KiRy/rCwsKUnZ1t1YSGhpbbZmhoqFVzqenTp1v3GzmdTkVFRdXUrgIAgHrGrZfMLjZ27Fh98cUX2rhxo8v8IUOGWH9u166dunTpoujoaK1evVqDBg2qcH3GGDkcDuv1xX+uqOZikydP1oQJE6zX+fn5hCIAAK5T9eIM0bhx47Ry5Uqlp6erSZMmldZGREQoOjpa+/fvlySFh4eruLhYubm5LnU5OTkKCwuzao4fP15uXSdOnLBqLuXj46PAwECXCQAAXJ/cGoiMMRo7dqyWLVumdevWqXnz5ldc5uTJkzp8+LAiIiIkSZ07d5aXl5fWrFlj1WRlZWn37t3q2rWrJCk+Pl55eXnaunWrVbNlyxbl5eVZNQAAwL7cesns8ccf15IlS/Tee+8pICDAup/H6XTKz89PZ86cUXJysh544AFFRETo4MGDeuaZZxQSEqL777/fqh0xYoSeeuopBQcHKygoSBMnTlT79u2tp85at26tvn37auTIkXr99dclSaNGjVL//v15wgwAALg3EM2dO1eS1L17d5f58+fP1/Dhw+Xh4aFdu3bpnXfe0enTpxUREaGEhAQtXbpUAQEBVv2sWbPk6empwYMHq7CwUD169NCCBQvk4eFh1SxevFhPPPGE9TRaYmKi5syZU/s7CdSxTl1ilZWVVeF4RESEMrdvqcOOAKD+c2sgMsZUOu7n56cPPvjgiuvx9fVVSkqKUlJSKqwJCgrSokWLrrpH4FqTlZWlhKlLKxxPf35IhWMAYFf14qZqAAAAdyIQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA23NrIJo+fbpuu+02BQQEKDQ0VPfdd5/27dvnUmOMUXJysiIjI+Xn56fu3btrz549LjVFRUUaN26cQkJC5O/vr8TERB05csSlJjc3V0lJSXI6nXI6nUpKStLp06drexcBAMA1wK2BaMOGDXr88ce1efNmrVmzRufPn1fv3r119uxZq2bGjBmaOXOm5syZo23btik8PFy9evVSQUGBVTN+/HgtX75cqamp2rhxo86cOaP+/fvrwoULVs3QoUOVmZmptLQ0paWlKTMzU0lJSXW6v9eCTl1iFXZT00qnTl1i3d0mAAA1ytOdG09LS3N5PX/+fIWGhmrHjh266667ZIzR7NmzNWXKFA0aNEiS9PbbbyssLExLlizR6NGjlZeXp3nz5mnhwoXq2bOnJGnRokWKiorS2rVr1adPH+3du1dpaWnavHmzYmN/+GX+5ptvKj4+Xvv27VNMTEzd7ng9lpWVpYSpSyutSX9+SB11AwBA3ahX9xDl5eVJkoKCgiRJBw4cUHZ2tnr37m3V+Pj4qFu3btq0aZMkaceOHSopKXGpiYyMVLt27ayajIwMOZ1OKwxJUlxcnJxOp1VzqaKiIuXn57tMAADg+lRvApExRhMmTNDPfvYztWvXTpKUnZ0tSQoLC3OpDQsLs8ays7Pl7e2tRo0aVVoTGhpabpuhoaFWzaWmT59u3W/kdDoVFRX143YQAADUW/UmEI0dO1ZffPGF3n333XJjDofD5bUxpty8S11ac7n6ytYzefJk5eXlWdPhw4ershsAAOAaVC8C0bhx47Ry5Uqlp6erSZMm1vzw8HBJKncWJycnxzprFB4eruLiYuXm5lZac/z48XLbPXHiRLmzT2V8fHwUGBjoMgEAgOuTWwORMUZjx47VsmXLtG7dOjVv3txlvHnz5goPD9eaNWusecXFxdqwYYO6du0qSercubO8vLxcarKysrR7926rJj4+Xnl5edq6datVs2XLFuXl5Vk1AADAvtz6lNnjjz+uJUuW6L333lNAQIB1JsjpdMrPz08Oh0Pjx4/XtGnT1LJlS7Vs2VLTpk1Tw4YNNXToUKt2xIgReuqppxQcHKygoCBNnDhR7du3t546a926tfr27auRI0fq9ddflySNGjVK/fv35wkzAADg3kA0d+5cSVL37t1d5s+fP1/Dhw+XJE2aNEmFhYUaM2aMcnNzFRsbqw8//FABAQFW/axZs+Tp6anBgwersLBQPXr00IIFC+Th4WHVLF68WE888YT1NFpiYqLmzJlTuzsIAACuCW4NRMaYK9Y4HA4lJycrOTm5whpfX1+lpKQoJSWlwpqgoCAtWrSoOm0CAIDrXL24qRoAAMCdCEQAAMD2CEQAAMD2CEQAAMD2CEQAAMD2qhWIWrRooZMnT5abf/r0abVo0eJHNwUAAFCXqhWIDh48qAsXLpSbX1RUpKNHj/7opgAAAOrSVX0O0cqVK60/f/DBB3I6ndbrCxcu6KOPPlKzZs1qrDkAAIC6cFWB6L777pP0w4clDhs2zGXMy8tLzZo105/+9Kcaaw4AAKAuXFUgKi0tlfTDl65u27ZNISEhtdIUAABAXarWV3ccOHCgpvsAAABwm2p/l9lHH32kjz76SDk5OdaZozJvvfXWj24MAACgrlQrED3//PN64YUX1KVLF0VERMjhcNR0XwAAAHWmWoHor3/9qxYsWKCkpKSa7gcAAKDOVetziIqLi9W1a9ea7gUAAMAtqhWIHn30US1ZsqSmewEAAHCLal0y+/777/XGG29o7dq16tChg7y8vFzGZ86cWSPNAQAA1IVqBaIvvvhCnTp1kiTt3r3bZYwbrAFIUqcuscrKyqq0JiIiQpnbt9RRRwBQsWoFovT09JruA8B1JisrSwlTl1Zak/78kDrqBgAqV617iAAAAK4n1TpDlJCQUOmlsXXr1lW7IQAAgLpWrUBUdv9QmZKSEmVmZmr37t3lvvQVAACgvqtWIJo1a9Zl5ycnJ+vMmTM/qiEAAIC6VqP3EP3yl7/ke8wAAMA1p0YDUUZGhnx9fWtylQAAALWuWpfMBg0a5PLaGKOsrCxt375dv//972ukMQAAgLpSrUDkdDpdXjdo0EAxMTF64YUX1Lt37xppDAAAoK5UKxDNnz+/pvsAAABwm2oFojI7duzQ3r175XA41KZNG91666011RcAAECdqVYgysnJ0YMPPqj169frxhtvlDFGeXl5SkhIUGpqqho3blzTfQIAANSaaj1lNm7cOOXn52vPnj06deqUcnNztXv3buXn5+uJJ56o6R4BAABqVbXOEKWlpWnt2rVq3bq1Na9Nmzb6y1/+wk3VAADgmlOtM0SlpaXy8vIqN9/Ly0ulpaU/uikAAIC6VK1AdPfdd+vJJ5/UsWPHrHlHjx7Vb37zG/Xo0aPGmgMAAKgL1QpEc+bMUUFBgZo1a6abb75Zt9xyi5o3b66CggKlpKTUdI8AAAC1qlr3EEVFRemzzz7TmjVr9N///lfGGLVp00Y9e/as6f4AAABq3VWdIVq3bp3atGmj/Px8SVKvXr00btw4PfHEE7rtttvUtm1bffLJJ7XSKAAAQG25qkA0e/ZsjRw5UoGBgeXGnE6nRo8erZkzZ9ZYcwAAAHXhqgLR559/rr59+1Y43rt3b+3YseNHNwUAAFCXrioQHT9+/LKP25fx9PTUiRMnfnRTAAAAdemqAtFNN92kXbt2VTj+xRdfKCIi4kc3BQAAUJeuKhDdc889eu655/T999+XGyssLNTUqVPVv3//GmsOAACgLlxVIHr22Wd16tQptWrVSjNmzNB7772nlStX6uWXX1ZMTIxOnTqlKVOmVHl9H3/8sQYMGKDIyEg5HA6tWLHCZXz48OFyOBwuU1xcnEtNUVGRxo0bp5CQEPn7+ysxMVFHjhxxqcnNzVVSUpKcTqecTqeSkpJ0+vTpq9l1AABwHbuqQBQWFqZNmzapXbt2mjx5su6//37dd999euaZZ9SuXTt9+umnCgsLq/L6zp49q44dO2rOnDkV1vTt21dZWVnW9P7777uMjx8/XsuXL1dqaqo2btyoM2fOqH///rpw4YJVM3ToUGVmZiotLU1paWnKzMxUUlLS1ew6AAC4jl31BzNGR0fr/fffV25urr766isZY9SyZUs1atToqjfer18/9evXr9IaHx8fhYeHX3YsLy9P8+bN08KFC60PhVy0aJGioqK0du1a9enTR3v37lVaWpo2b96s2NhYSdKbb76p+Ph47du3TzExMVfdNwAAuL5U66s7JKlRo0a67bbbdPvtt1crDFXV+vXrFRoaqlatWmnkyJHKycmxxnbs2KGSkhL17t3bmhcZGal27dpp06ZNkqSMjAw5nU4rDElSXFycnE6nVQMAAOytWl/dUVf69eunX/ziF4qOjtaBAwf0+9//Xnfffbd27NghHx8fZWdny9vbu1wgCwsLU3Z2tiQpOztboaGh5dYdGhpq1VxOUVGRioqKrNdln84NAACuP/U6EA0ZMsT6c7t27dSlSxdFR0dr9erVGjRoUIXLGWPkcDis1xf/uaKaS02fPl3PP/98NTsHAADXkmpfMnOHiIgIRUdHa//+/ZKk8PBwFRcXKzc316UuJyfHurk7PDxcx48fL7euEydOVHoD+OTJk5WXl2dNhw8frsE9AQAA9ck1FYhOnjypw4cPWx/+2LlzZ3l5eWnNmjVWTVZWlnbv3q2uXbtKkuLj45WXl6etW7daNVu2bFFeXp5Vczk+Pj4KDAx0mQAAwPXJrZfMzpw5o6+++sp6feDAAWVmZiooKEhBQUFKTk7WAw88oIiICB08eFDPPPOMQkJCdP/990v64QtlR4wYoaeeekrBwcEKCgrSxIkT1b59e+ups9atW6tv374aOXKkXn/9dUnSqFGj1L9/f54wAwAAktwciLZv366EhATr9YQJEyRJw4YN09y5c7Vr1y698847On36tCIiIpSQkKClS5cqICDAWmbWrFny9PTU4MGDVVhYqB49emjBggXy8PCwahYvXqwnnnjCehotMTGx0s8+AgAA9uLWQNS9e3cZYyoc/+CDD664Dl9fX6WkpCglJaXCmqCgIC1atKhaPQIAgOvfNXUPEQAAQG0gEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANsjEAEAANtzayD6+OOPNWDAAEVGRsrhcGjFihUu48YYJScnKzIyUn5+furevbv27NnjUlNUVKRx48YpJCRE/v7+SkxM1JEjR1xqcnNzlZSUJKfTKafTqaSkJJ0+fbqW9w4AAFwr3BqIzp49q44dO2rOnDmXHZ8xY4ZmzpypOXPmaNu2bQoPD1evXr1UUFBg1YwfP17Lly9XamqqNm7cqDNnzqh///66cOGCVTN06FBlZmYqLS1NaWlpyszMVFJSUq3vHwAAuDZ4unPj/fr1U79+/S47ZozR7NmzNWXKFA0aNEiS9PbbbyssLExLlizR6NGjlZeXp3nz5mnhwoXq2bOnJGnRokWKiorS2rVr1adPH+3du1dpaWnavHmzYmNjJUlvvvmm4uPjtW/fPsXExNTNzgKo1zp1iVVWVlaF4xEREcrcvqUOOwJQl9waiCpz4MABZWdnq3fv3tY8Hx8fdevWTZs2bdLo0aO1Y8cOlZSUuNRERkaqXbt22rRpk/r06aOMjAw5nU4rDElSXFycnE6nNm3aVGEgKioqUlFRkfU6Pz+/FvYSQH2RlZWlhKlLKxxPf35IHXYDoK7V25uqs7OzJUlhYWEu88PCwqyx7OxseXt7q1GjRpXWhIaGllt/aGioVXM506dPt+45cjqdioqK+lH7AwAA6q96G4jKOBwOl9fGmHLzLnVpzeXqr7SeyZMnKy8vz5oOHz58lZ0DAIBrRb0NROHh4ZJU7ixOTk6OddYoPDxcxcXFys3NrbTm+PHj5dZ/4sSJcmefLubj46PAwECXCQAAXJ/qbSBq3ry5wsPDtWbNGmtecXGxNmzYoK5du0qSOnfuLC8vL5earKws7d6926qJj49XXl6etm7datVs2bJFeXl5Vg0AALA3t95UfebMGX311VfW6wMHDigzM1NBQUFq2rSpxo8fr2nTpqlly5Zq2bKlpk2bpoYNG2ro0KGSJKfTqREjRuipp55ScHCwgoKCNHHiRLVv39566qx169bq27evRo4cqddff12SNGrUKPXv358nzAAAgCQ3B6Lt27crISHBej1hwgRJ0rBhw7RgwQJNmjRJhYWFGjNmjHJzcxUbG6sPP/xQAQEB1jKzZs2Sp6enBg8erMLCQvXo0UMLFiyQh4eHVbN48WI98cQT1tNoiYmJFX72EQAAsB+3BqLu3bvLGFPhuMPhUHJyspKTkyus8fX1VUpKilJSUiqsCQoK0qJFi35MqwAA4DpWb+8hAgAAqCsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHv1OhAlJyfL4XC4TOHh4da4MUbJycmKjIyUn5+funfvrj179riso6ioSOPGjVNISIj8/f2VmJioI0eO1PWuAACAeqxeByJJatu2rbKysqxp165d1tiMGTM0c+ZMzZkzR9u2bVN4eLh69eqlgoICq2b8+PFavny5UlNTtXHjRp05c0b9+/fXhQsX3LE7AACgHvJ0dwNX4unp6XJWqIwxRrNnz9aUKVM0aNAgSdLbb7+tsLAwLVmyRKNHj1ZeXp7mzZunhQsXqmfPnpKkRYsWKSoqSmvXrlWfPn3qdF8AAED9VO/PEO3fv1+RkZFq3ry5HnzwQX3zzTeSpAMHDig7O1u9e/e2an18fNStWzdt2rRJkrRjxw6VlJS41ERGRqpdu3ZWTUWKioqUn5/vMgEAgOtTvQ5EsbGxeuedd/TBBx/ozTffVHZ2trp27aqTJ08qOztbkhQWFuayTFhYmDWWnZ0tb29vNWrUqMKaikyfPl1Op9OaoqKianDPAABAfVKvA1G/fv30wAMPqH379urZs6dWr14t6YdLY2UcDofLMsaYcvMuVZWayZMnKy8vz5oOHz5czb0AAAD1Xb0ORJfy9/dX+/bttX//fuu+okvP9OTk5FhnjcLDw1VcXKzc3NwKayri4+OjwMBAlwkAAFyfrqlAVFRUpL179yoiIkLNmzdXeHi41qxZY40XFxdrw4YN6tq1qySpc+fO8vLycqnJysrS7t27rRoAAIB6/ZTZxIkTNWDAADVt2lQ5OTl68cUXlZ+fr2HDhsnhcGj8+PGaNm2aWrZsqZYtW2ratGlq2LChhg4dKklyOp0aMWKEnnrqKQUHBysoKEgTJ060LsEBAABI9TwQHTlyRA899JC+++47NW7cWHFxcdq8ebOio6MlSZMmTVJhYaHGjBmj3NxcxcbG6sMPP1RAQIC1jlmzZsnT01ODBw9WYWGhevTooQULFsjDw8NduwUAAOqZeh2IUlNTKx13OBxKTk5WcnJyhTW+vr5KSUlRSkpKDXcHAACuF9fUPUQAAAC1gUAEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsj0AEAABsr15/dQcA2E2nLrHKO3pUB/6/181vidG5Bq7/d42IiFDm9i113xxwHSMQAUA9kpWVpb7PvCM90UOSdNcz76jIx8+lJv35Ie5oDbiucckMAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYnqe7GwAA1E+dusQqKyur0pqIiAhlbt9SRx0BtYdABAC4rKysLCVMXVppTfrzQ+qoG6B2cckMAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYnq0C0WuvvabmzZvL19dXnTt31ieffOLulgAAQD1gm0C0dOlSjR8/XlOmTNHOnTt15513ql+/fjp06JC7WwMAAG5mm88hmjlzpkaMGKFHH31UkjR79mx98MEHmjt3rqZPn+7m7gAAP9aVPkiSD5FEZWwRiIqLi7Vjxw49/fTTLvN79+6tTZs2uakrAEBNutIHSdbVh0jyCd/XJlsEou+++04XLlxQWFiYy/ywsDBlZ2dfdpmioiIVFRVZr/Py8iRJ+fn5Nd5faWmpSgrPXrGmNrZ9LfdSV31UBe/b1fdRl71URX1634oLz6psS8WFZ1VSWuq2Xvg7vHpHjx7VXU8vqLTm45eG10kvd3S7u8Lfc2XCw8P16YZ1td6Lu5S9z8aYyguNDRw9etRIMps2bXKZ/+KLL5qYmJjLLjN16lQjiYmJiYmJiek6mA4fPlxpVrDFGaKQkBB5eHiUS8k5OTnlzhqVmTx5siZMmGC9Li0t1alTpxQcHCyHw1FjveXn5ysqKkqHDx9WYGBgja33x6qPfdXHniT6uhr1sSepfvZVH3uS6Otq1MeepPrZV232ZIxRQUGBIiMjK62zRSDy9vZW586dtWbNGt1///3W/DVr1mjgwIGXXcbHx0c+Pj4u82688cZa6zEwMLDe/GBerD72VR97kujratTHnqT62Vd97Emir6tRH3uS6mdftdWT0+m8Yo0tApEkTZgwQUlJSerSpYvi4+P1xhtv6NChQ/r1r3/t7tYAAICb2SYQDRkyRCdPntQLL7ygrKwstWvXTu+//76io6Pd3RoAAHAz2wQiSRozZozGjBnj7jZc+Pj4aOrUqeUuz7lbfeyrPvYk0dfVqI89SfWzr/rYk0RfV6M+9iTVz77qQ08OY670HBoAAMD1zTZf3QEAAFARAhEAALA9AhEAALA9AhEAALA9AlEdOH/+vJ599lk1b95cfn5+atGihV544QWVXvT9RGfOnNHYsWPVpEkT+fn5qXXr1po7d67b+zp+/LiGDx+uyMhINWzYUH379tX+/ftrta+CggKNHz9e0dHR8vPzU9euXbVt2zZr3Bij5ORkRUZGys/PT927d9eePXtqtaeq9LVs2TL16dNHISEhcjgcyszMrPWertRXSUmJfve736l9+/by9/dXZGSkHnnkER07dsxtPUlScnKyfvKTn8jf31+NGjVSz549tWVL7X/R5ZX6utjo0aPlcDg0e/Zst/Y0fPhwORwOlykuLq5We6pKX5K0d+9eJSYmyul0KiAgQHFxcTp06JBb+7r0vSqbXnnlFbf15I7je1X6qu3j+8cff6wBAwYoMjJSDodDK1ascBmvyrG8qKhI48aNU0hIiPz9/ZWYmKgjR47UWI+XNoRa9uKLL5rg4GCzatUqc+DAAfOPf/zD3HDDDWb27NlWzaOPPmpuvvlmk56ebg4cOGBef/114+HhYVasWOG2vkpLS01cXJy58847zdatW81///tfM2rUKNO0aVNz5syZWutr8ODBpk2bNmbDhg1m//79ZurUqSYwMNAcOXLEGGPMSy+9ZAICAsy//vUvs2vXLjNkyBATERFh8vPza62nqvT1zjvvmOeff968+eabRpLZuXNnrfZTlb5Onz5tevbsaZYuXWr++9//moyMDBMbG2s6d+7stp6MMWbx4sVmzZo15uuvvza7d+82I0aMMIGBgSYnJ8etfZVZvny56dixo4mMjDSzZs1ya0/Dhg0zffv2NVlZWdZ08uTJWu2pKn199dVXJigoyPz2t781n332mfn666/NqlWrzPHjx93a18XvU1ZWlnnrrbeMw+EwX3/9tdt6csfx/Up91cXx/f333zdTpkwx//rXv4wks3z5cpfxqhzLf/3rX5ubbrrJrFmzxnz22WcmISHBdOzY0Zw/f75GerwYgagO3HvvveZXv/qVy7xBgwaZX/7yl9brtm3bmhdeeMGl5qc//al59tln3dbXvn37jCSze/dua/z8+fMmKCjIvPnmm7XS07lz54yHh4dZtWqVy/yOHTuaKVOmmNLSUhMeHm5eeukla+z77783TqfT/PWvf62VnqrS18UOHDhQZ4Hoavoqs3XrViPJfPvtt/Wmp7y8PCPJrF27tlZ6upq+jhw5Ym666Saze/duEx0dXauBqCo9DRs2zAwcOLDWeqhuX0OGDHE5htWXvi41cOBAc/fdd7u1J3cc36/UV10f3y8NRFU5lp8+fdp4eXmZ1NRUq+bo0aOmQYMGJi0trcZ75JJZHfjZz36mjz76SF9++aUk6fPPP9fGjRt1zz33uNSsXLlSR48elTFG6enp+vLLL9WnTx+39VVUVCRJ8vX1tZbx8PCQt7e3Nm7cWCs9nT9/XhcuXHDZpiT5+flp48aNOnDggLKzs9W7d29rzMfHR926ddOmTZtqpaeq9OUu1ekrLy9PDoej1r6b72p7Ki4u1htvvCGn06mOHTvWSk9V7au0tFRJSUn67W9/q7Zt29ZaL1fTkyStX79eoaGhatWqlUaOHKmcnBy39lVaWqrVq1erVatW6tOnj0JDQxUbG1vukkhd93Wp48ePa/Xq1RoxYoRbe3LH8f1Kfbnj+H6xqhzLd+zYoZKSEpeayMhItWvXrnaO9zUesVBOaWmpefrpp43D4TCenp7G4XCYadOmudQUFRWZRx55xEgynp6extvb27zzzjtu7au4uNhER0ebX/ziF+bUqVOmqKjITJ8+3UgyvXv3rrW+4uPjTbdu3czRo0fN+fPnzcKFC43D4TCtWrUyn376qZFkjh496rLMyJEja7WnK/V1sbo8Q3Q1fRljTGFhoencubN5+OGH3d7T//3f/xl/f3/jcDhMZGSk2bp1a632VJW+pk2bZnr16mVKS0uNMabWzxBVpafU1FSzatUqs2vXLrNy5UrTsWNH07ZtW/P999+7ra+srCwjyTRs2NDMnDnT7Ny500yfPt04HA6zfv16t/V1qZdfftk0atTIFBYWurUndxzfr9RXXR/fdckZoqocyxcvXmy8vb3LratXr15m1KhRNd9jja8R5bz77rumSZMm5t133zVffPGFeeedd0xQUJBZsGCBVfPKK6+YVq1amZUrV5rPP//cpKSkmBtuuMGsWbPGrX1t377ddOzY0UgyHh4epk+fPqZfv36mX79+tdbXV199Ze666y5rm7fddpt5+OGHTevWra1/RMeOHXNZ5tFHHzV9+vSptZ6u1NfF6joQVbWv4uJiM3DgQHPrrbeavLw8t/d05swZs3//fpORkWF+9atfmWbNmtX6/SeV9bV9+3YTFhbmcoCui0BU1b+/MseOHTNeXl7mX//6l9v6Onr0qJFkHnroIZdlBgwYYB588EG39XWpmJgYM3bs2Frtpyo9ueP4XpW+6vL4XlEgquxYXlEg6tmzpxk9enTN91jja0Q5TZo0MXPmzHGZ94c//MHExMQYY3641uvl5VXuWu+IESNq9Zf8lfq62OnTp60bXm+//XYzZsyYWuurzJkzZ6x/LIMHDzb33HOP+frrr40k89lnn7nUJiYmmkceeaTWe6qor4vVdSCqSl/FxcXmvvvuMx06dDDfffddvejpUrfccku5M6d12desWbOMw+EwHh4e1iTJNGjQwERHR7ulp4rccsstLvde1HVfRUVFxtPT0/zhD39wqZ00aZLp2rWr2/q62Mcff2wkmczMzDrpp6Ke3HV8v1JfF6uL4/ulgagqx/KPPvrISDKnTp1yqenQoYN57rnnarxH7iGqA+fOnVODBq5vtYeHh/V4e0lJiUpKSiqtcUdfF3M6nWrcuLH279+v7du3a+DAgbXWVxl/f39FREQoNzdXH3zwgQYOHKjmzZsrPDxca9asseqKi4u1YcMGde3atdZ7qqiv+qCivkpKSjR48GDt379fa9euVXBwsNt7uhxjjHVfgzv6SkpK0hdffKHMzExrioyM1G9/+1t98MEHbunpck6ePKnDhw8rIiKi1nuqqC9vb2/ddttt2rdvn0vtl19+qejoaLf1dbF58+apc+fOtXpfWlV6ctfx/Up9Xcwdx/eqHMs7d+4sLy8vl5qsrCzt3r27do73NR6xUM6wYcPMTTfdZD3evmzZMhMSEmImTZpk1XTr1s20bdvWpKenm2+++cbMnz/f+Pr6mtdee82tff3973836enp5uuvvzYrVqww0dHRZtCgQbXWkzHGpKWlmX//+9/mm2++MR9++KHp2LGjuf32201xcbEx5odHNZ1Op1m2bJnZtWuXeeihh+rksfsr9XXy5Emzc+dOs3r1aiPJpKammp07d5qsrCy39VVSUmISExNNkyZNTGZmpsvjyEVFRW7p6cyZM2by5MkmIyPDHDx40OzYscOMGDHC+Pj4uDzxUtd9XU5dXDKrrKeCggLz1FNPmU2bNpkDBw6Y9PR0Ex8fb2666Sa3/7wvW7bMeHl5mTfeeMPs37/fpKSkGA8PD/PJJ5+4tS9jfnhqsWHDhmbu3Lm12ktVe3LH8b0qfdX28b2goMDs3LnT7Ny500iy7jcre8K1KsfyX//616ZJkyZm7dq15rPPPjN33303j91fy/Lz882TTz5pmjZtanx9fU2LFi3MlClTXH4hZWVlmeHDh5vIyEjj6+trYmJizJ/+9Cfr5k539fXqq6+aJk2aGC8vL9O0aVPz7LPP1uovUmOMWbp0qWnRooXx9vY24eHh5vHHHzenT5+2xktLS83UqVNNeHi48fHxMXfddZfZtWtXrfZUlb7mz59vJJWbpk6d6ra+yi7fXW5KT093S0+FhYXm/vvvN5GRkcbb29tERESYxMTEOrmp+kp/h5eqi0BUWU/nzp0zvXv3No0bN7b+DQ4bNswcOnSoVnu6Ul9l5s2bZ2655Rbj6+trOnbsWOufq1PVvl5//XXj5+dX6d9tXfbkjuN7Vfqq7eN7enr6ZY89w4YNM8ZU7VheWFhoxo4da4KCgoyfn5/p379/rf38O4wxpubPOwEAAFw7uIcIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIAADYHoEIgC0tWLBAN954o7vbAFBP8MGMAGypsLBQBQUFCg0NrfIy3bt3V6dOnTR79uzaawyAW3i6uwEAcAc/Pz/5+fm5uw0A9QSXzABck7p3766xY8dq7NixuvHGGxUcHKxnn31WZSe9c3Nz9cgjj6hRo0Zq2LCh+vXrp/3791vLX3rJLDk5WZ06ddLChQvVrFkzOZ1OPfjggyooKJAkDR8+XBs2bNCrr74qh8Mhh8OhgwcPKjc3Vw8//LAaN24sPz8/tWzZUvPnz6/T9wLAj0cgAnDNevvtt+Xp6aktW7boz3/+s2bNmqW//e1vkn4IMNu3b9fKlSuVkZEhY4zuuecelZSUVLi+r7/+WitWrNCqVau0atUqbdiwQS+99JIk6dVXX1V8fLxGjhyprKwsZWVlKSoqSr///e/1n//8R//+97+1d+9ezZ07VyEhIXWy/wBqDpfMAFyzoqKiNGvWLDkcDsXExGjXrl2aNWuWunfvrpUrV+rTTz9V165dJUmLFy9WVFSUVqxYoV/84heXXV9paakWLFiggIAASVJSUpI++ugj/e///q+cTqe8vb3VsGFDhYeHW8scOnRIt956q7p06SJJatasWe3uNIBawRkiANesuLg4ORwO63V8fLz279+v//znP/L09FRsbKw1FhwcrJiYGO3du7fC9TVr1swKQ5IUERGhnJycSnt47LHHlJqaqk6dOmnSpEnatGnTj9gjAO5CIAJgG8YYlwB1KS8vL5fXDodDpaWlla6zX79++vbbbzV+/HgdO3ZMPXr00MSJE2ukXwB1h0AE4Jq1efPmcq9btmypNm3a6Pz589qyZYs1dvLkSX355Zdq3bp1tbfn7e2tCxculJvfuHFjDR8+XIsWLdLs2bP1xhtvVHsbANyDQATgmnX48GFNmDBB+/bt07vvvquUlBQ9+eSTatmypQYOHKiRI0dq48aN+vzzz/XLX/5SN910kwYOHFjt7TVr1kxbtmzRwYMH9d1336m0tFTPPfec3nvvPX311Vfas2ePVq1a9aNCFwD3IBABuGY98sgjKiws1O23367HH39c48aN06hRoyRJ8+fPV+fOndW/f3/Fx8fLGKP333+/3GWxqzFx4kR5eHioTZs2aty4sQ4dOiRvb29NnjxZHTp00F133SUPDw+lpqbW1C4CqCN8UjWAaxKfGg2gJnGGCAAA2B6BCAAA2B6XzAAAgO1xhggAANgegQgAANgegQgAANgegQgAANgegQgAANgegQgAANgegQgAANgegQgAANgegQgAANje/wNOkbRaehxlFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot histogram of points in dataframe\n",
    "ax = sns.histplot(df_wine_model, x='points')\n",
    "\n",
    "plt.xticks(df_wine_model.points.unique())\n",
    "plt.axvline(x=95, c='red')\n",
    "plt.title('Distribution of Wine Ratings');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bcea20a-da03-4289-8a5d-4fd0b7dd5408",
   "metadata": {},
   "source": [
    "About 60% of the recommended wines have actual ratings above 95 points which signifies that the model is very good at recommending high quality and similar wines.\n",
    "\n",
    "Now, we will access the top 10 recommendations provided by the model if a new user wants to choose a wine with specific traits like `cherry` or `pencil_lead`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d2090844-4fc9-41b7-b422-18edaed097bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>est_match_pts</th>\n",
       "      <th>points</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3797</th>\n",
       "      <td>Alpha Omega 2012 ERA Red (Napa Valley)</td>\n",
       "      <td>91.448946</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3589</th>\n",
       "      <td>Williams Selyem 2012 Eastside Road Neighbors Pinot Noir (Russian River Valley)</td>\n",
       "      <td>91.397192</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1501</th>\n",
       "      <td>Cayuse 2011 En Cerise Vineyard Syrah (Walla Walla Valley (OR))</td>\n",
       "      <td>91.375636</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3798</th>\n",
       "      <td>Alpha Omega 2012 Beckstoffer Missouri Hopper Cabernet Sauvignon (Oakville)</td>\n",
       "      <td>91.358037</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3588</th>\n",
       "      <td>Williams Selyem 2013 Westside Road Neighbors Pinot Noir (Russian River Valley)</td>\n",
       "      <td>91.358037</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>852</th>\n",
       "      <td>Shafer 2012 Hillside Select Cabernet Sauvignon (Stags Leap District)</td>\n",
       "      <td>91.358037</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>491</th>\n",
       "      <td>Freeman 2012 Gloria Estate Pinot Noir (Russian River Valley)</td>\n",
       "      <td>91.334430</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1502</th>\n",
       "      <td>Wayfarer 2012 The Traveler Pinot Noir (Fort Ross-Seaview)</td>\n",
       "      <td>91.267128</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1424</th>\n",
       "      <td>Cobb 2012 Diane Cobb Coastlands Vineyard Pinot Noir (Sonoma Coast)</td>\n",
       "      <td>91.267128</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3752</th>\n",
       "      <td>Donkey &amp; Goat 2010 Fenaughty Vineyard Syrah (El Dorado)</td>\n",
       "      <td>91.267128</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                               title  \\\n",
       "3797                                          Alpha Omega 2012 ERA Red (Napa Valley)   \n",
       "3589  Williams Selyem 2012 Eastside Road Neighbors Pinot Noir (Russian River Valley)   \n",
       "1501                  Cayuse 2011 En Cerise Vineyard Syrah (Walla Walla Valley (OR))   \n",
       "3798      Alpha Omega 2012 Beckstoffer Missouri Hopper Cabernet Sauvignon (Oakville)   \n",
       "3588  Williams Selyem 2013 Westside Road Neighbors Pinot Noir (Russian River Valley)   \n",
       "852             Shafer 2012 Hillside Select Cabernet Sauvignon (Stags Leap District)   \n",
       "491                     Freeman 2012 Gloria Estate Pinot Noir (Russian River Valley)   \n",
       "1502                       Wayfarer 2012 The Traveler Pinot Noir (Fort Ross-Seaview)   \n",
       "1424              Cobb 2012 Diane Cobb Coastlands Vineyard Pinot Noir (Sonoma Coast)   \n",
       "3752                         Donkey & Goat 2010 Fenaughty Vineyard Syrah (El Dorado)   \n",
       "\n",
       "      est_match_pts  points  \n",
       "3797      91.448946      99  \n",
       "3589      91.397192      95  \n",
       "1501      91.375636      98  \n",
       "3798      91.358037      98  \n",
       "3588      91.358037      98  \n",
       "852       91.358037      98  \n",
       "491       91.334430      94  \n",
       "1502      91.267128      97  \n",
       "1424      91.267128      97  \n",
       "3752      91.267128      97  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return the top10 recommended wines when a new user chooses wine with either cherry or pencil lead\n",
    "# Create df that merge estimated ratings with full wine details df\n",
    "df_wine_rec = df_wine_model.merge(recommend_df, on='title', how='inner')\n",
    "\n",
    "# Create df which consists of wines fulfiling either traits\n",
    "df_temp_traits = df_wine_combi.drop(columns=['taster_name', 'points', 'variety', 'designation', 'winery', 'country', 'province', 'region_1', 'region_2', 'price', 'description',\n",
    "                                             'desc_wd_count', 'traits'])\n",
    "trait_filter = ['title', 'cherry', 'pencil_lead']\n",
    "df_temp_traits = df_temp_traits[trait_filter]\n",
    "df_temp_traits['sum'] = df_temp_traits.sum(axis = 1, numeric_only=True)\n",
    "df_temp_traits = df_temp_traits[df_temp_traits['sum'] != 0]\n",
    "\n",
    "# Merge both dfs and show the top 10 by estimated ratings\n",
    "df_recommend_details = df_temp_traits.merge(df_wine_rec, on='title', how='left')\n",
    "df_recommend_final = df_recommend_details.sort_values('est_match_pts', ascending=False).drop_duplicates()\n",
    "df_recommend_final[['title', 'est_match_pts', 'points']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74ec32f-a7b7-4821-adba-4f09cff76190",
   "metadata": {},
   "source": [
    "80% of the recommended wines have actual ratings above 95 points which signifies that the model is very good at recommending high quality wines."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62c04721-1f45-4e57-a8be-8027a0dff402",
   "metadata": {},
   "source": [
    "## Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d15d8351-fea6-4991-a0c6-494beb9811e6",
   "metadata": {},
   "source": [
    "Model|Root Mean Squared Error (RMSE)|Precision@k\n",
    "---|---|---\n",
    "Pure Randomized Recommender (Baseline)|n/a|0.36\n",
    "**KNN Baseline (Tuned)**|**1.828586**|**0.751706**\n",
    "Baseline Predictor|1.921808|0.751706\n",
    "KNN Basic|1.853119|0.751706\n",
    "KNN Means|1.853373|0.751706\n",
    "KNN ZScore|1.853372|0.751706\n",
    "KNN Baseline|1.828589|0.751706\n",
    "Slope One|1.853719|0.751706\n",
    "Co-clustering|1.858452|0.749706\n",
    "SVD|1.830108|0.659683\n",
    "Normal Predictor|2.67958|0.605464\n",
    "NonNegative Matrix Factorization|2.123324|0.582563"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d799392b-aaf3-454f-97b8-63306bafb106",
   "metadata": {},
   "source": [
    "From the table above, the baseline model performed very poorly. With high volatility in its RMSE, the differences in its predicted rating to the actual rating for each taster-wine pair can fluctuate exponentially. It also has the lowest Precision@k score, which means it performs poorly in recommending the relevant wines in its top 10 recommendations.\n",
    "\n",
    "The tuned k-NN Baseline with k=35 and cosine similarity measure performed the best out of all the models. Even though its Precision@k score is the same as six other models, it has the lowest RMSE score thus far. Thus, this model is very good at recommending relevant wines in its top 10 recommendations. Its predicted ratings are also much closer to the actual ratings for each taster-wine pair with a known label, compared to other models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfeb6c2d-29f5-454c-a668-bfc2963ddf76",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90dc412f-8166-491f-8ca9-1e10bac84822",
   "metadata": {},
   "source": [
    "**Recommendation System Model**\n",
    "\n",
    "[k-NN Baseline](https://surprise.readthedocs.io/en/stable/knn_inspired.html#surprise.prediction_algorithms.knns.KNNBaseline) is the most suitable model to use for recommending wines. This model is capable of suggesting good quality wines better than leaving the recommendations by chance.<br>\n",
    "Reasons are stated below:\n",
    "1. Precision@k score is above 0.75 and higher than that of other models.\n",
    "2. RMSE score is lowest amongst all other models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c13e6a89-6220-41b8-ba8b-9ede4d1b7fb7",
   "metadata": {},
   "source": [
    "**Limitation/Further Improvements**\n",
    "\n",
    "A limitation of k-NN algorithm models is that they cannot handle higher dimensional data well. It implies that if the number of users or number of wines in the database increase, the chosen model might not perform well and end up giving poor recommendations. This leads to scalability issues for the client if they want to update the database.<br>\n",
    "\n",
    "Our model is limited to the dataset used to train the recommender system. This is because the model can't recommend wines that are not available in the actual database in the first place. To add on, it is limited by the user reviews recorded in the dataset and any new users would not be able to update their actual ratings in the dataset. It would be good to update the ratings of new users in the database and retrain the recommender system from time to time. That way, our recommender system can stay relevant and continue to give good recommendations going forward. The recommender system can be further enhanced to exclude wines that have gone out of stock by linking the inventory data back to the dataset used by the recommender."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20328587-7908-4ad7-b8da-b5f9c91d44c7",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Recommendation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5861f099-5150-4dba-a6cd-fdc7b6aa1270",
   "metadata": {},
   "source": [
    "Going back to the problem statement, we recommend that the k-NN Baseline model should be used as a recommender system to suggest suitable wines to customers.\n",
    "\n",
    "*Please proceed to the Streamlit app version of this wine recommender to try it out for yourself*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:capstone]",
   "language": "python",
   "name": "conda-env-capstone-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}