Analyze_ab_test_results_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze A/B Test Results \n",
    "\n",
    "This project will assure you have mastered the subjects covered in the statistics lessons. We have organized the current notebook into the following sections: \n",
    "\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Final Check](#finalcheck)\n",
    "- [Submission](#submission)\n",
    "\n",
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists. For this project, you will be working to understand the results of an A/B test run by an e-commerce website.  Your goal is to work through this notebook to help the company understand if they should:\n",
    "- Implement the new webpage, \n",
    "- Keep the old webpage, or \n",
    "- Perhaps run the experiment longer to make their decision.\n",
    "\n",
    "Each **ToDo** task below has an associated quiz present in the classroom.  Though the classroom quizzes are **not necessary** to complete the project, they help ensure you are on the right track as you work through the project, and you can feel more confident in your final submission meeting the [rubric](https://review.udacity.com/#!/rubrics/1214/view) specification. \n",
    "\n",
    "<a id='probability'></a>\n",
    "## Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.1\n",
    "Now, read in the `ab_data.csv` data. Store it in `df`. Below is the description of the data, there are a total of 5 columns:\n",
    "\n",
    "<center>\n",
    "\n",
    "|Data columns|Purpose|Valid values|\n",
    "| ------------- |:-------------| -----:|\n",
    "|user_id|Unique ID|Int64 values|\n",
    "|timestamp|Time stamp when the user visited the webpage|-|\n",
    "|group|In the current A/B experiment, the users are categorized into two broad groups. <br>The `control` group users are expected to be served with `old_page`; and `treatment` group users are matched with the `new_page`. <br>However, **some inaccurate rows** are present in the initial data, such as a `control` group user is matched with a `new_page`. |`['control', 'treatment']`|\n",
    "|landing_page|It denotes whether the user visited the old or new webpage.|`['old_page', 'new_page']`|\n",
    "|converted|It denotes whether the user decided to pay for the company's product. Here, `1` means yes, the user bought the product.|`[0, 1]`|\n",
    "</center>\n",
    "Use your dataframe to answer the questions in Quiz 1 of the classroom.\n",
    "\n",
    "**a.** Read in the dataset from the `ab_data.csv` file and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Use the cell below to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        user_id                   timestamp      group landing_page  converted\n",
      "294476   697314  2017-01-15 01:20:28.957438    control     old_page          0\n",
      "294477   715931  2017-01-16 12:40:24.467417  treatment     new_page          0\n"
     ]
    }
   ],
   "source": [
    "print(df.tail(2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allu = df.user_id.nunique()\n",
    "allu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**d.** The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.12104245244060237\n"
     ]
    }
   ],
   "source": [
    "cnv = df.query('converted == 1').user_id.nunique()\n",
    "print(cnv/allu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**e.** The number of times when the \"group\" is `treatment` but \"landing_page\" is not a `new_page`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1965"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TposNPneg = df.query(\"group == 'treatment' and landing_page != 'new_page'\").shape[0]\n",
    "TposNPneg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1928"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TnegNPpos = df.query('group == \"control\" and landing_page == \"new_page\"').shape[0]\n",
    "TnegNPpos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TposNPneg+TnegNPpos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         294478 non-null int64\n",
      "timestamp       294478 non-null object\n",
      "group           294478 non-null object\n",
      "landing_page    294478 non-null object\n",
      "converted       294478 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.2  \n",
    "In a particular row, the **group** and **landing_page** columns should have either of the following acceptable values:\n",
    "\n",
    "|user_id| timestamp|group|landing_page|converted|\n",
    "|---|---|---|---|---|\n",
    "|XXXX|XXXX|`control`| `old_page`|X |\n",
    "|XXXX|XXXX|`treatment`|`new_page`|X |\n",
    "\n",
    "\n",
    "It means, the `control` group users should match with `old_page`; and `treatment` group users should matched with the `new_page`. \n",
    "\n",
    "However, for the rows where `treatment` does not match with `new_page` or `control` does not match with `old_page`, we cannot be sure if such rows truly received the new or old wepage.  \n",
    "\n",
    "\n",
    "Use **Quiz 2** in the classroom to figure out how should we handle the rows where the group and landing_page columns don't match?\n",
    "\n",
    "**a.** Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>644214</td>\n",
       "      <td>2017-01-22 02:05:21.719434</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>847721</td>\n",
       "      <td>2017-01-17 14:01:00.090575</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>650559</td>\n",
       "      <td>2017-01-24 11:55:51.084801</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>935734</td>\n",
       "      <td>2017-01-17 20:33:37.428378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>746742</td>\n",
       "      <td>2017-01-23 11:38:29.592148</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>913579</td>\n",
       "      <td>2017-01-24 09:11:39.164256</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>690284</td>\n",
       "      <td>2017-01-13 17:22:57.182769</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>710349</td>\n",
       "      <td>2017-01-11 22:24:44.226492</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>677533</td>\n",
       "      <td>2017-01-23 17:48:50.491821</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>831737</td>\n",
       "      <td>2017-01-11 21:18:20.911015</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>771087</td>\n",
       "      <td>2017-01-16 00:05:29.983919</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>896163</td>\n",
       "      <td>2017-01-22 09:10:20.753218</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>862225</td>\n",
       "      <td>2017-01-08 14:49:37.335432</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>939593</td>\n",
       "      <td>2017-01-05 09:15:31.984283</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>702260</td>\n",
       "      <td>2017-01-18 13:55:31.488221</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>670941</td>\n",
       "      <td>2017-01-05 08:16:41.306478</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>850231</td>\n",
       "      <td>2017-01-18 17:18:04.790584</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>685794</td>\n",
       "      <td>2017-01-20 14:54:58.150621</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>714733</td>\n",
       "      <td>2017-01-03 08:22:37.904146</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>710967</td>\n",
       "      <td>2017-01-10 02:19:22.842142</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>680201</td>\n",
       "      <td>2017-01-11 10:38:45.952887</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>790863</td>\n",
       "      <td>2017-01-19 11:02:39.220320</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>717595</td>\n",
       "      <td>2017-01-23 18:19:08.148166</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>779854</td>\n",
       "      <td>2017-01-11 21:28:30.735359</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>916307</td>\n",
       "      <td>2017-01-19 17:27:38.676600</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290555</th>\n",
       "      <td>924332</td>\n",
       "      <td>2017-01-15 19:38:52.858024</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290556</th>\n",
       "      <td>849625</td>\n",
       "      <td>2017-01-06 17:54:07.563311</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290557</th>\n",
       "      <td>929723</td>\n",
       "      <td>2017-01-10 15:13:48.352399</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290558</th>\n",
       "      <td>774769</td>\n",
       "      <td>2017-01-03 06:01:36.251836</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290559</th>\n",
       "      <td>733871</td>\n",
       "      <td>2017-01-21 17:54:08.810964</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290560</th>\n",
       "      <td>844588</td>\n",
       "      <td>2017-01-16 20:48:19.567178</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290561</th>\n",
       "      <td>641244</td>\n",
       "      <td>2017-01-07 16:57:26.193171</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290562</th>\n",
       "      <td>676072</td>\n",
       "      <td>2017-01-14 17:26:02.495442</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290563</th>\n",
       "      <td>886374</td>\n",
       "      <td>2017-01-07 13:43:39.202634</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290564</th>\n",
       "      <td>676732</td>\n",
       "      <td>2017-01-03 23:06:45.459467</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290565</th>\n",
       "      <td>862218</td>\n",
       "      <td>2017-01-04 10:43:07.846494</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290566</th>\n",
       "      <td>798826</td>\n",
       "      <td>2017-01-23 16:50:13.788528</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290567</th>\n",
       "      <td>836721</td>\n",
       "      <td>2017-01-12 17:37:50.966955</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290568</th>\n",
       "      <td>844901</td>\n",
       "      <td>2017-01-15 17:46:36.622726</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290569</th>\n",
       "      <td>653124</td>\n",
       "      <td>2017-01-14 13:44:51.745491</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290570</th>\n",
       "      <td>909437</td>\n",
       "      <td>2017-01-18 14:49:49.064452</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290571</th>\n",
       "      <td>776137</td>\n",
       "      <td>2017-01-12 05:53:12.386730</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290572</th>\n",
       "      <td>883344</td>\n",
       "      <td>2017-01-22 23:15:58.645325</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290573</th>\n",
       "      <td>825594</td>\n",
       "      <td>2017-01-06 12:37:08.897784</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290574</th>\n",
       "      <td>937338</td>\n",
       "      <td>2017-01-19 03:23:22.236666</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290575</th>\n",
       "      <td>733101</td>\n",
       "      <td>2017-01-23 12:52:58.711914</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290576</th>\n",
       "      <td>679096</td>\n",
       "      <td>2017-01-02 16:43:49.237940</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290577</th>\n",
       "      <td>691699</td>\n",
       "      <td>2017-01-09 23:42:35.963486</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290578</th>\n",
       "      <td>807595</td>\n",
       "      <td>2017-01-22 10:43:09.285426</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290579</th>\n",
       "      <td>846225</td>\n",
       "      <td>2017-01-16 15:24:46.705903</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290580</th>\n",
       "      <td>677163</td>\n",
       "      <td>2017-01-03 19:41:51.902148</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290581</th>\n",
       "      <td>925675</td>\n",
       "      <td>2017-01-07 20:38:26.346410</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290582</th>\n",
       "      <td>643562</td>\n",
       "      <td>2017-01-02 19:20:05.460595</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290583</th>\n",
       "      <td>822004</td>\n",
       "      <td>2017-01-04 03:36:46.071379</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290584</th>\n",
       "      <td>715931</td>\n",
       "      <td>2017-01-16 12:40:24.467417</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>290585 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "0        851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1        804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2        864975  2017-01-21 01:52:26.210827    control     old_page          1\n",
       "3        936923  2017-01-10 15:20:49.083499    control     old_page          0\n",
       "4        719014  2017-01-17 01:48:29.539573    control     old_page          0\n",
       "5        644214  2017-01-22 02:05:21.719434    control     old_page          1\n",
       "6        847721  2017-01-17 14:01:00.090575    control     old_page          0\n",
       "7        650559  2017-01-24 11:55:51.084801    control     old_page          0\n",
       "8        935734  2017-01-17 20:33:37.428378    control     old_page          0\n",
       "9        746742  2017-01-23 11:38:29.592148    control     old_page          0\n",
       "10       913579  2017-01-24 09:11:39.164256    control     old_page          1\n",
       "11       690284  2017-01-13 17:22:57.182769    control     old_page          0\n",
       "12       710349  2017-01-11 22:24:44.226492    control     old_page          0\n",
       "13       677533  2017-01-23 17:48:50.491821    control     old_page          0\n",
       "14       831737  2017-01-11 21:18:20.911015    control     old_page          1\n",
       "15       771087  2017-01-16 00:05:29.983919    control     old_page          0\n",
       "16       896163  2017-01-22 09:10:20.753218    control     old_page          0\n",
       "17       862225  2017-01-08 14:49:37.335432    control     old_page          1\n",
       "18       939593  2017-01-05 09:15:31.984283    control     old_page          0\n",
       "19       702260  2017-01-18 13:55:31.488221    control     old_page          0\n",
       "20       670941  2017-01-05 08:16:41.306478    control     old_page          0\n",
       "21       850231  2017-01-18 17:18:04.790584    control     old_page          1\n",
       "22       685794  2017-01-20 14:54:58.150621    control     old_page          0\n",
       "23       714733  2017-01-03 08:22:37.904146    control     old_page          0\n",
       "24       710967  2017-01-10 02:19:22.842142    control     old_page          0\n",
       "25       680201  2017-01-11 10:38:45.952887    control     old_page          0\n",
       "26       790863  2017-01-19 11:02:39.220320    control     old_page          0\n",
       "27       717595  2017-01-23 18:19:08.148166    control     old_page          0\n",
       "28       779854  2017-01-11 21:28:30.735359    control     old_page          0\n",
       "29       916307  2017-01-19 17:27:38.676600    control     old_page          0\n",
       "...         ...                         ...        ...          ...        ...\n",
       "290555   924332  2017-01-15 19:38:52.858024  treatment     new_page          0\n",
       "290556   849625  2017-01-06 17:54:07.563311  treatment     new_page          0\n",
       "290557   929723  2017-01-10 15:13:48.352399  treatment     new_page          0\n",
       "290558   774769  2017-01-03 06:01:36.251836  treatment     new_page          0\n",
       "290559   733871  2017-01-21 17:54:08.810964  treatment     new_page          1\n",
       "290560   844588  2017-01-16 20:48:19.567178  treatment     new_page          0\n",
       "290561   641244  2017-01-07 16:57:26.193171  treatment     new_page          0\n",
       "290562   676072  2017-01-14 17:26:02.495442  treatment     new_page          0\n",
       "290563   886374  2017-01-07 13:43:39.202634  treatment     new_page          0\n",
       "290564   676732  2017-01-03 23:06:45.459467  treatment     new_page          0\n",
       "290565   862218  2017-01-04 10:43:07.846494  treatment     new_page          0\n",
       "290566   798826  2017-01-23 16:50:13.788528  treatment     new_page          0\n",
       "290567   836721  2017-01-12 17:37:50.966955  treatment     new_page          0\n",
       "290568   844901  2017-01-15 17:46:36.622726  treatment     new_page          0\n",
       "290569   653124  2017-01-14 13:44:51.745491  treatment     new_page          0\n",
       "290570   909437  2017-01-18 14:49:49.064452  treatment     new_page          0\n",
       "290571   776137  2017-01-12 05:53:12.386730  treatment     new_page          0\n",
       "290572   883344  2017-01-22 23:15:58.645325  treatment     new_page          0\n",
       "290573   825594  2017-01-06 12:37:08.897784  treatment     new_page          0\n",
       "290574   937338  2017-01-19 03:23:22.236666  treatment     new_page          0\n",
       "290575   733101  2017-01-23 12:52:58.711914  treatment     new_page          0\n",
       "290576   679096  2017-01-02 16:43:49.237940  treatment     new_page          0\n",
       "290577   691699  2017-01-09 23:42:35.963486  treatment     new_page          0\n",
       "290578   807595  2017-01-22 10:43:09.285426  treatment     new_page          0\n",
       "290579   846225  2017-01-16 15:24:46.705903  treatment     new_page          0\n",
       "290580   677163  2017-01-03 19:41:51.902148  treatment     new_page          0\n",
       "290581   925675  2017-01-07 20:38:26.346410  treatment     new_page          0\n",
       "290582   643562  2017-01-02 19:20:05.460595  treatment     new_page          0\n",
       "290583   822004  2017-01-04 03:36:46.071379  treatment     new_page          0\n",
       "290584   715931  2017-01-16 12:40:24.467417  treatment     new_page          0\n",
       "\n",
       "[290585 rows x 5 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove the inaccurate rows, and store the result in a new dataframe df2\n",
    "dfco = df.query('group ==\"control\" and landing_page ==\"old_page\"')\n",
    "dftn = df.query('group ==\"treatment\" and landing_page ==\"new_page\"')\n",
    "df2 = pd.concat([dfco,dftn], ignore_index=True)\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the incorrect rows were removed from df2 - \n",
    "# Output of the statement below should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.3  \n",
    "Use **df2** and the cells below to answer questions for **Quiz 3** in the classroom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allu2 = df2.user_id.nunique()\n",
    "allu2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**b.** There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>146678</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "146678   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.user_id.duplicated()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Display the rows for the duplicate **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>146212</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146678</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "146212   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "146678   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('user_id == 773192')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Remove **one** of the rows with a duplicate **user_id**, from the **df2** dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>146678</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "146678   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove one of the rows with a duplicate user_id..\n",
    "# Hint: The dataframe.drop_duplicates() may not work in this case because the rows with duplicate user_id are not entirely identical. \n",
    "df2 = df2.drop(labels=[146212], axis=0)\n",
    "df2.query('user_id == 773192')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.4  \n",
    "Use **df2** in the cells below to answer the quiz questions related to **Quiz 4** in the classroom.\n",
    "\n",
    "**a.** What is the probability of an individual converting regardless of the page they receive?<br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv2 = df2.query('converted == 1').user_id.nunique()\n",
    "pop2 = df2.user_id.nunique()\n",
    "nconv2 = df2.query('converted == 0').user_id.nunique()\n",
    "P_pop = conv2/pop2\n",
    "P_pop\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ctrlconv = df2.query('group == \"control\" and converted == 1')\n",
    "ctrlnconv = df2.query('group == \"control\" and converted == 0')\n",
    "treatconv = df2.query('group == \"treatment\" and converted == 1')\n",
    "treatnconv = df2.query('group == \"treatment\" and converted == 0')\n",
    "P_ctrlconv =ctrlconv.user_id.nunique() / (ctrlconv.user_id.nunique()+ctrlnconv.user_id.nunique())\n",
    "P_ctrlconv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_treatconv = treatconv.user_id.nunique()/(treatconv.user_id.nunique()+treatnconv.user_id.nunique())\n",
    "P_treatconv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the actual difference (obs_diff) between the conversion rates for the two groups.\n",
    "obs_diff = P_treatconv-P_ctrlconv\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Pcc = ctrlconv.user_id.nunique()/pop2\n",
    "Pcnc = ctrlnconv.user_id.nunique()/pop2\n",
    "Ptc = treatconv.user_id.nunique()/pop2\n",
    "Ptnc = treatnconv.user_id.nunique()/pop2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ptc + Ptnc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** Consider your results from parts (a) through (d) above, and explain below whether the new `treatment` group users lead to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**No, they didn't lead to more conversions.The probability of new page conversions are slightly lower (almost equal) than propability of old page users.**\n",
    ">>**obs_diff = -0.0015782**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "## Part II - A/B Test\n",
    "\n",
    "Since a timestamp is associated with each event, you could run a hypothesis test continuously as long as you observe the events. \n",
    "\n",
    "However, then the hard questions would be: \n",
    "- Do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  \n",
    "- How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "### ToDo 2.1\n",
    "For now, consider you need to make the decision just based on all the data provided.  \n",
    "\n",
    "> Recall that you just calculated that the \"converted\" probability (or rate) for the old page is *slightly* higher than that of the new page (ToDo 1.4.c). \n",
    "\n",
    "If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should be your null and alternative hypotheses (**$H_0$** and **$H_1$**)?  \n",
    "\n",
    "You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the \"converted\" probability (or rate) for the old and new pages respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**$H_0$: $p_{new}$ <= $p_{old}$**\n",
    "\n",
    "**$H_1$: $p_{new}$ > $p_{old}$**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 2.2 - Null Hypothesis $H_0$ Testing\n",
    "Under the null hypothesis $H_0$, assume that $p_{new}$ and $p_{old}$ are equal. Furthermore, assume that $p_{new}$ and $p_{old}$ both are equal to the **converted** success rate in the `df2` data regardless of the page. So, our assumption is: <br><br>\n",
    "<center>\n",
    "$p_{new}$ = $p_{old}$ = $p_{population}$\n",
    "</center>\n",
    "\n",
    "In this section, you will: \n",
    "\n",
    "- Simulate (bootstrap) sample data set for both groups, and compute the  \"converted\" probability $p$ for those samples. \n",
    "\n",
    "\n",
    "- Use a sample size for each group equal to the ones in the `df2` data.\n",
    "\n",
    "\n",
    "- Compute the difference in the \"converted\" probability for the two samples above. \n",
    "\n",
    "\n",
    "- Perform the sampling distribution for the \"difference in the converted probability\" between the two simulated-samples over 10,000 iterations; and calculate an estimate. \n",
    "\n",
    "\n",
    "\n",
    "Use the cells below to provide the necessary parts of this simulation.  You can use **Quiz 5** in the classroom to make sure you are on the right track."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** What is the **conversion rate** for $p_{new}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_new = df2.query('converted == 1').user_id.nunique()/df2.user_id.nunique()\n",
    "P_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** What is the **conversion rate** for $p_{old}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_old = df2.query('converted == 1').user_id.nunique()/df2.user_id.nunique()\n",
    "P_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** What is $n_{new}$, the number of individuals in the treatment group? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new = df2.query('group == \"treatment\"').user_id.nunique()\n",
    "n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is $n_{old}$, the number of individuals in the control group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old = df2.query('group == \"control\"').user_id.nunique()\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e. Simulate Sample for the `treatment` Group**<br> \n",
    "Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null hypothesis.  <br>\n",
    "Store these $n_{new}$ 1's and 0's in the `new_page_converted` numpy array.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate a Sample for the treatment Group\n",
    "new_page_converted = np.random.choice([1,0], size = n_new, p=[P_new,1-P_new])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f. Simulate Sample for the `control` Group** <br>\n",
    "Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null hypothesis. <br> Store these $n_{old}$ 1's and 0's in the `old_page_converted` numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate a Sample for the control Group\n",
    "old_page_converted = np.random.choice([1,0], size = n_old, p=[P_old,1-P_old])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g.** Find the difference in the \"converted\" probability $(p{'}_{new}$ - $p{'}_{old})$ for your simulated samples from the parts (e) and (f) above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0023214247121787213"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_page_converted.mean()-old_page_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**h. Sampling distribution** <br>\n",
    "Re-create `new_page_converted` and `old_page_converted` and find the $(p{'}_{new}$ - $p{'}_{old})$ value 10,000 times using the same simulation process you used in parts (a) through (g) above. \n",
    "\n",
    "<br>\n",
    "Store all  $(p{'}_{new}$ - $p{'}_{old})$  values in a NumPy array called `p_diffs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sampling distribution \n",
    "p_diffs = []\n",
    "new_converted_simulation = np.random.binomial(n_new, P_new, 10000)/n_new \n",
    "old_converted_simulation = np.random.binomial(n_old, P_old, 10000)/n_old \n",
    "p_diffs = new_converted_simulation - old_converted_simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**i. Histogram**<br> \n",
    "Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here.<br><br>\n",
    "\n",
    "Also, use `plt.axvline()` method to mark the actual difference observed  in the `df2` data (recall `obs_diff`), in the chart.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAENlJREFUeJzt3X+s3XV9x/Hna61gNnWUtbCurSuazgz+GLIGWdwfXZhQiqH4hwkk0wZNajJINHNZqvyB0ZCAzh8jc5iqjSVDkU2NjXTDSiTGZEALQ6DUrleocm1HqzXoYuICvvfH+Xactqe9597ec8+tn+cj+eZ8z/v7+X6/n++nN33d749zbqoKSVJ7fmvcHZAkjYcBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWrUwnF34FQWL15cK1euHHc3NA579/Ze3/CG8fZDOgM9+uijP6mqJVO1m9cBsHLlSnbt2jXubmgc1qzpvT744Dh7IZ2RkvxwmHZeApKkRhkAktQoA0CSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1ygCQpEbN608CS/PZyk33jWW/+2+7eiz71W8ezwAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktSoKQMgyYok306yJ8nuJO/t6h9K8uMkj3fTur51PpBkIsneJFf21dd2tYkkm0ZzSJKkYSwcos2LwPur6rEkrwYeTbKjW/bJqvr7/sZJLgSuAy4C/gD4VpI/6hZ/GngLMAnsTLKtqp6ejQORJE3PlAFQVQeBg938L5LsAZadYpX1wD1V9Svg2SQTwKXdsomqegYgyT1dWwNAksZgWvcAkqwE3gg83JVuSvJEki1JFnW1ZcBzfatNdrWT1SVJYzB0ACR5FfAV4H1V9XPgTuD1wMX0zhA+frTpgNXrFPXj97Mxya4kuw4fPjxs9yRJ0zRUACR5Bb3//O+uqq8CVNXzVfVSVf0a+CwvX+aZBFb0rb4cOHCK+jGqanNVra6q1UuWLJnu8UiShjTMU0ABPg/sqapP9NWX9jV7G/BUN78NuC7J2UkuAFYBjwA7gVVJLkhyFr0bxdtm5zAkSdM1zFNAbwbeATyZ5PGu9kHg+iQX07uMsx94D0BV7U5yL72buy8CN1bVSwBJbgLuBxYAW6pq9yweiyRpGoZ5Cui7DL5+v/0U69wK3Dqgvv1U60mS5o6fBJakRhkAktQoA0CSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSo4b5NlBJ88jKTfeNbd/7b7t6bPvW7PMMQJIaZQBIUqO8BKQz2jgvh0hnOs8AJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRk0ZAElWJPl2kj1Jdid5b1c/N8mOJPu610VdPUnuSDKR5Ikkl/Rta0PXfl+SDaM7LEnSVIY5A3gReH9V/TFwGXBjkguBTcADVbUKeKB7D3AVsKqbNgJ3Qi8wgFuANwGXArccDQ1J0tybMgCq6mBVPdbN/wLYAywD1gNbu2ZbgWu7+fXAXdXzEHBOkqXAlcCOqjpSVT8DdgBrZ/VoJElDm9Y9gCQrgTcCDwPnV9VB6IUEcF7XbBnwXN9qk13tZHVJ0hgMHQBJXgV8BXhfVf38VE0H1OoU9eP3szHJriS7Dh8+PGz3JEnTNFQAJHkFvf/8766qr3bl57tLO3Svh7r6JLCib/XlwIFT1I9RVZuranVVrV6yZMl0jkWSNA3DPAUU4PPAnqr6RN+ibcDRJ3k2AF/vq7+zexroMuCF7hLR/cAVSRZ1N3+v6GqSpDEY5k9Cvhl4B/Bkkse72geB24B7k7wb+BHw9m7ZdmAdMAH8ErgBoKqOJPkIsLNr9+GqOjIrRyFJmrYpA6Cqvsvg6/cAlw9oX8CNJ9nWFmDLdDooSRoNPwksSY0yACSpUQaAJDXKAJCkRhkAktQoA0CSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRk0ZAEm2JDmU5Km+2oeS/DjJ4920rm/ZB5JMJNmb5Mq++tquNpFk0+wfiiRpOoY5A/gCsHZA/ZNVdXE3bQdIciFwHXBRt84/JVmQZAHwaeAq4ELg+q6tJGlMFk7VoKq+k2TlkNtbD9xTVb8Cnk0yAVzaLZuoqmcAktzTtX162j2WJM2K07kHcFOSJ7pLRIu62jLgub42k13tZHVJ0pjMNADuBF4PXAwcBD7e1TOgbZ2ifoIkG5PsSrLr8OHDM+yeJGkqMwqAqnq+ql6qql8Dn+XlyzyTwIq+psuBA6eoD9r25qpaXVWrlyxZMpPuSZKGMKMASLK07+3bgKNPCG0DrktydpILgFXAI8BOYFWSC5KcRe9G8baZd1uSdLqmvAmc5EvAGmBxkkngFmBNkovpXcbZD7wHoKp2J7mX3s3dF4Ebq+qlbjs3AfcDC4AtVbV71o9GkjS0YZ4Cun5A+fOnaH8rcOuA+nZg+7R6J0kaGT8JLEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktQoA0CSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1asoASLIlyaEkT/XVzk2yI8m+7nVRV0+SO5JMJHkiySV962zo2u9LsmE0hyNJGtYwZwBfANYeV9sEPFBVq4AHuvcAVwGrumkjcCf0AgO4BXgTcClwy9HQkCSNx5QBUFXfAY4cV14PbO3mtwLX9tXvqp6HgHOSLAWuBHZU1ZGq+hmwgxNDRZI0h2Z6D+D8qjoI0L2e19WXAc/1tZvsaierS5LGZOEsby8DanWK+okbSDbSu3zEa1/72tnrmUZq5ab7ZnV79zzzUwCum+XtSnrZTAPg+SRLq+pgd4nnUFefBFb0tVsOHOjqa46rPzhow1W1GdgMsHr16oEhIWk8Zjvoh7X/tqvHst/fdDO9BLQNOPokzwbg6331d3ZPA10GvNBdIrofuCLJou7m7xVdTZI0JlOeAST5Er3f3hcnmaT3NM9twL1J3g38CHh713w7sA6YAH4J3ABQVUeSfATY2bX7cFUdf2NZkjSHpgyAqrr+JIsuH9C2gBtPsp0twJZp9U6SNDJ+EliSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktQoA0CSGmUASFKjDABJapQBIEmNOq0ASLI/yZNJHk+yq6udm2RHkn3d66KuniR3JJlI8kSSS2bjACRJMzMbZwB/UVUXV9Xq7v0m4IGqWgU80L0HuApY1U0bgTtnYd+SpBkaxSWg9cDWbn4rcG1f/a7qeQg4J8nSEexfkjSE0w2AAr6Z5NEkG7va+VV1EKB7Pa+rLwOe61t3sqsdI8nGJLuS7Dp8+PBpdk+SdDILT3P9N1fVgSTnATuSfP8UbTOgVicUqjYDmwFWr159wnJJ0uw4rTOAqjrQvR4CvgZcCjx/9NJO93qoaz4JrOhbfTlw4HT2L0mauRkHQJLfSfLqo/PAFcBTwDZgQ9dsA/D1bn4b8M7uaaDLgBeOXiqSJM2907kEdD7wtSRHt/PFqvr3JDuBe5O8G/gR8Pau/XZgHTAB/BK44TT2LUk6TTMOgKp6BviTAfWfApcPqBdw40z3J0maXX4SWJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktQoA0CSGnW63waqeWblpvvG3QVJZwjPACSpUZ4BSJr3xnlmu/+2q8e271HzDECSGmUASFKjDABJapQBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGzfnfA0iyFvgHYAHwuaq6ba77IEnDGtffIpiLv0MwpwGQZAHwaeAtwCSwM8m2qnp6Lvsxav5ZRklngrm+BHQpMFFVz1TV/wL3AOvnuA+SJOb+EtAy4Lm+95PAm0a1M38Tl6STm+sAyIBaHdMg2Qhs7N7+T5K9I+/V7FoM/GTcnZhnpj0mf3Z05va3znpn5gl/Tk7kmPTJ7cDMx+QPh2k01wEwCazoe78cONDfoKo2A5vnslOzKcmuqlo97n7MJ47JiRyTEzkmJxr1mMz1PYCdwKokFyQ5C7gO2DbHfZAkMcdnAFX1YpKbgPvpPQa6pap2z2UfJEk9c/45gKraDmyf6/3OoTP28tUIOSYnckxO5JicaKRjkqqaupUk6TeOXwUhSY0yAIaU5NwkO5Ls614XnaTdhq7NviQb+up/muTJJBNJ7kiS49b72ySVZPGoj2W2jGpMknwsyfeTPJHka0nOmatjmokka5Ps7Y5j04DlZyf5crf84SQr+5Z9oKvvTXLlsNuc72Z7TJKsSPLtJHuS7E7y3rk7mtkxip+TbtmCJP+Z5BvT7lRVOQ0xAR8FNnXzm4DbB7Q5F3ime13UzS/qlj1C7/H2AP8GXNW33gp6N8Z/CCwe97GOe0yAK4CF3fztg7Y7XyZ6DzP8AHgdcBbwPeDC49r8NfCZbv464Mvd/IVd+7OBC7rtLBhmm/N5GtGYLAUu6dq8Gviv1sekb72/Ab4IfGO6/fIMYHjrga3d/Fbg2gFtrgR2VNWRqvoZsANYm2Qp8Jqq+o/q/Yvdddz6nwT+juM+FHcGGMmYVNU3q+rFbv2H6H1eZL4a5utN+sfpX4HLu7Od9cA9VfWrqnoWmOi2d6Z/Zcqsj0lVHayqxwCq6hfAHnrfLHCmGMXPCUmWA1cDn5tJpwyA4Z1fVQcButfzBrQZ9FUXy7ppckCdJNcAP66q742i0yM2kjE5zrvonR3MVyc7voFtumB7Afi9U6w7zDbns1GMyf/rLo28EXh4Fvs8aqMak0/R++Xx1zPp1Jw/BjqfJfkW8PsDFt087CYG1Opk9SS/3W37iiG3P+fmekyO2/fNwIvA3UPuaxymPI5TtDlZfdAvZmfS2eEoxqS3UvIq4CvA+6rq5zPu4dyb9TFJ8lbgUFU9mmTNTDplAPSpqr882bIkzydZWlUHu8sXhwY0mwTW9L1fDjzY1ZcfVz8AvJ7eNb3vdfc/lwOPJbm0qv77NA5l1oxhTI5uewPwVuDy7hLRfDXl15v0tZlMshD4XeDIFOtOtc35bCRjkuQV9P7zv7uqvjqaro/MKMbkGuCaJOuAVwKvSfLPVfVXQ/dq3DdHzpQJ+BjH3vD86IA25wLP0rvZuaibP7dbthO4jJdveK4bsP5+zqybwCMZE2At8DSwZNzHOMQYLKR3Y/sCXr65d9FxbW7k2Jt793bzF3Hszb1n6N0snHKb83ka0ZiE3n2iT437+ObLmBy37hpmcBN47ANzpkz0rsU9AOzrXo/+J7aa3l82O9ruXfRu0kwAN/TVVwNP0buD/490H8I7bh9nWgCMZEy6ds8Bj3fTZ8Z9rFOMwzp6T6X8ALi5q30YuKabfyXwL91xPQK8rm/dm7v19nLsk2EnbPNMmmZ7TIA/p3c55Im+n4sTfomaz9Mofk76ls8oAPwksCQ1yqeAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY36P4NyBDt9BTRpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f56721e8e48>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(p_diffs);\n",
    "plt.axvline(x=obs_diff,color='red');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**j.** What proportion of the **p_diffs** are greater than the actual difference observed in the `df2` data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90400000000000003"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(p_diffs > obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**k.** Please explain in words what you have just computed in part **j** above.  \n",
    " - What is this value called in scientific studies?  \n",
    " - What does this value signify in terms of whether or not there is a difference between the new and old pages?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**This value is called as P-value. When p_diffs are higher than obs_diff, P value is higher than 0.05, so we can not reject the null hypothesis.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**l. Using Built-in Methods for Hypothesis Testing**<br>\n",
    "We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. \n",
    "\n",
    "Fill in the statements below to calculate the:\n",
    "- `convert_old`: number of conversions with the old_page\n",
    "- `convert_new`: number of conversions with the new_page\n",
    "- `n_old`: number of individuals who were shown the old_page\n",
    "- `n_new`: number of individuals who were shown the new_page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>290580</th>\n",
       "      <td>677163</td>\n",
       "      <td>2017-01-03 19:41:51.902148</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290581</th>\n",
       "      <td>925675</td>\n",
       "      <td>2017-01-07 20:38:26.346410</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290582</th>\n",
       "      <td>643562</td>\n",
       "      <td>2017-01-02 19:20:05.460595</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290583</th>\n",
       "      <td>822004</td>\n",
       "      <td>2017-01-04 03:36:46.071379</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290584</th>\n",
       "      <td>715931</td>\n",
       "      <td>2017-01-16 12:40:24.467417</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "290580   677163  2017-01-03 19:41:51.902148  treatment     new_page          0\n",
       "290581   925675  2017-01-07 20:38:26.346410  treatment     new_page          0\n",
       "290582   643562  2017-01-02 19:20:05.460595  treatment     new_page          0\n",
       "290583   822004  2017-01-04 03:36:46.071379  treatment     new_page          0\n",
       "290584   715931  2017-01-16 12:40:24.467417  treatment     new_page          0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "# number of conversions with the old_page\n",
    "convert_old = df2.query('landing_page == \"old_page\" and converted == 1').user_id.nunique()\n",
    "\n",
    "# number of conversions with the new_page\n",
    "convert_new = df2.query('landing_page == \"new_page\" and converted == 1').user_id.nunique()\n",
    "\n",
    "# number of individuals who were shown the old_page\n",
    "n_old = df2.query('landing_page == \"old_page\"').user_id.nunique()\n",
    "\n",
    "# number of individuals who received new_page\n",
    "n_new = df2.query('landing_page == \"new_page\"').user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**m.** Now use `sm.stats.proportions_ztest()` to compute your test statistic and p-value.  [Here](https://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportions_ztest.html) is a helpful link on using the built in.\n",
    "\n",
    "The syntax is: \n",
    "```bash\n",
    "proportions_ztest(count_array, nobs_array, alternative='larger')\n",
    "```\n",
    "where, \n",
    "- `count_array` = represents the number of \"converted\" for each group\n",
    "- `nobs_array` = represents the total number of observations (rows) in each group\n",
    "- `alternative` = choose one of the values from `[‘two-sided’, ‘smaller’, ‘larger’]` depending upon two-tailed, left-tailed, or right-tailed respectively. \n",
    ">**Hint**: <br>\n",
    "It's a two-tailed if you defined $H_1$ as $(p_{new} = p_{old})$. <br>\n",
    "It's a left-tailed if you defined $H_1$ as $(p_{new} < p_{old})$. <br>\n",
    "It's a right-tailed if you defined $H_1$ as $(p_{new} > p_{old})$. \n",
    "\n",
    "The built-in function above will return the z_score, p_value. \n",
    "\n",
    "---\n",
    "### About the two-sample z-test\n",
    "Recall that you have plotted a distribution `p_diffs` representing the\n",
    "difference in the \"converted\" probability  $(p{'}_{new}-p{'}_{old})$  for your two simulated samples 10,000 times. \n",
    "\n",
    "Another way for comparing the mean of two independent and normal distribution is a **two-sample z-test**. You can perform the Z-test to calculate the Z_score, as shown in the equation below:\n",
    "\n",
    "$$\n",
    "Z_{score} = \\frac{ (p{'}_{new}-p{'}_{old}) - (p_{new}  -  p_{old})}{ \\sqrt{ \\frac{\\sigma^{2}_{new} }{n_{new}} + \\frac{\\sigma^{2}_{old} }{n_{old}}  } }\n",
    "$$\n",
    "\n",
    "where,\n",
    "- $p{'}$ is the \"converted\" success rate in the sample\n",
    "- $p_{new}$ and $p_{old}$ are the \"converted\" success rate for the two groups in the population. \n",
    "- $\\sigma_{new}$ and $\\sigma_{new}$ are the standard deviation for the two groups in the population. \n",
    "- $n_{new}$ and $n_{old}$ represent the size of the two groups or samples (it's same in our case)\n",
    "\n",
    "\n",
    ">Z-test is performed when the sample size is large, and the population variance is known. The z-score represents the distance between the two \"converted\" success rates in terms of the standard error. \n",
    "\n",
    "Next step is to make a decision to reject or fail to reject the null hypothesis based on comparing these two values: \n",
    "- $Z_{score}$\n",
    "- $Z_{\\alpha}$ or $Z_{0.05}$, also known as critical value at 95% confidence interval.  $Z_{0.05}$ is 1.645 for one-tailed tests,  and 1.960 for two-tailed test. You can determine the $Z_{\\alpha}$ from the z-table manually. \n",
    "\n",
    "Decide if your hypothesis is either a two-tailed, left-tailed, or right-tailed test. Accordingly, reject OR fail to reject the  null based on the comparison between $Z_{score}$ and $Z_{\\alpha}$. \n",
    ">Hint:<br>\n",
    "For a right-tailed test, reject null if $Z_{score}$ > $Z_{\\alpha}$. <br>\n",
    "For a left-tailed test, reject null if $Z_{score}$ < $Z_{\\alpha}$. \n",
    "\n",
    "\n",
    "In other words, we determine whether or not the $Z_{score}$ lies in the \"rejection region\" in the distribution. A \"rejection region\" is an interval where the null hypothesis is rejected iff the $Z_{score}$ lies in that region.\n",
    "\n",
    "\n",
    "\n",
    "Reference: \n",
    "- Example 9.1.2 on this [page](https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Book%3A_Introductory_Statistics_(Shafer_and_Zhang)/09%3A_Two-Sample_Problems/9.01%3A_Comparison_of_Two_Population_Means-_Large_Independent_Samples), courtesy www.stats.libretexts.org\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.31092419842 0.094941687241\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "# ToDo: Complete the sm.stats.proportions_ztest() method arguments\n",
    "z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new], alternative='larger')\n",
    "print(z_score, p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**n.** What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?<br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Z-score shows the difference between conversion rates statistics and null hypothesis. It is lower than 1.64 for a confidence level of 95%. The result of p-value is different from the one that we calculated in part j. However, it is still higher than 0.05. We fail to reject null hypothesis.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "### ToDo 3.1 \n",
    "In this final part, you will see that the result you achieved in the A/B test in Part II above can also be achieved by performing regression.<br><br> \n",
    "\n",
    "**a.** Since each row in the `df2` data is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**We should perform logistic regression since the data is either a conversion or no conversion.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** The goal is to use **statsmodels** library to fit the regression model you specified in part **a.** above to see if there is a significant difference in conversion based on the page-type a customer receives. However, you first need to create the following two columns in the `df2` dataframe:\n",
    " 1. `intercept` - It should be `1` in the entire column. \n",
    " 2. `ab_page` - It's a dummy variable column, having a value `1` when an individual receives the **treatment**, otherwise `0`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "\n",
       "   intercept  ab_page  \n",
       "0          1        0  \n",
       "1          1        0  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['intercept'] = 1\n",
    "df2[['a_page','ab_page']] = pd.get_dummies(df2['group'])\n",
    "df2 = df2.drop('a_page',axis=1)\n",
    "df2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Use **statsmodels** to instantiate your regression model on the two columns you created in part (b). above, then fit the model to predict whether or not an individual converts. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "logit_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']])\n",
    "results = logit_mod.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-05-19 21:00</td>       <td>AIC:</td>        <td>212780.3502</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212801.5095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>1</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290582</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3109</td>  <td>0.1899</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-05-19 21:00 AIC:              212780.3502\n",
       "No. Observations:   290584           BIC:              212801.5095\n",
       "Df Model:           1                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290582           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "ab_page      -0.0150    0.0114    -1.3109  0.1899  -0.0374   0.0074\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**P-Value associated with ab_page is 0.1899. The main difference is while the part II has a one-tailed test, part III has a two tailed test.**\n",
    "\n",
    ">**Null hypothesis is ab_page = 1 and converted = 0**\n",
    "\n",
    ">**Alternative hypothesis is ab_page = 1 and converted = 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**It can not be said that treatment or control page has an influence on conversion. Thus, it might be a good idea to use other factors into this regression model. Adding many parameters into the regression model might lead to multicollinearity. Multicollinearity occurs when two or more parameters are collinear with each other.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g. Adding countries**<br> \n",
    "Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in. \n",
    "\n",
    "1. You will need to read in the **countries.csv** dataset and merge together your `df2` datasets on the appropriate rows. You call the resulting dataframe `df_merged`. [Here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html) are the docs for joining tables. \n",
    "\n",
    "2. Does it appear that country had an impact on conversion?  To answer this question, consider the three unique values, `['UK', 'US', 'CA']`, in the `country` column. Create dummy variables for these country columns. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = pd.read_csv('countries.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>851104</th>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>804228</th>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864975</th>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>936923</th>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>719014</th>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          timestamp    group landing_page  converted  \\\n",
       "user_id                                                                \n",
       "851104   2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "804228   2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "864975   2017-01-21 01:52:26.210827  control     old_page          1   \n",
       "936923   2017-01-10 15:20:49.083499  control     old_page          0   \n",
       "719014   2017-01-17 01:48:29.539573  control     old_page          0   \n",
       "\n",
       "         intercept  ab_page country  \n",
       "user_id                              \n",
       "851104           1        0      US  \n",
       "804228           1        0      US  \n",
       "864975           1        0      US  \n",
       "936923           1        0      US  \n",
       "719014           1        0      US  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df2.set_index('user_id').join(countries.set_index('user_id'))\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.country.unique()\n",
    "df2[['US','CA','UK']] = pd.get_dummies(df2['country'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-05-19 21:00</td>       <td>AIC:</td>        <td>212781.1253</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212823.4439</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>3</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290580</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-2.0300</td>  <td>0.0266</td>  <td>-76.2488</td> <td>0.0000</td> <td>-2.0822</td> <td>-1.9778</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0149</td>  <td>0.0114</td>   <td>-1.3069</td> <td>0.1912</td> <td>-0.0374</td> <td>0.0075</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>0.0506</td>   <td>0.0284</td>   <td>1.7835</td>  <td>0.0745</td> <td>-0.0050</td> <td>0.1063</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0408</td>   <td>0.0269</td>   <td>1.5161</td>  <td>0.1295</td> <td>-0.0119</td> <td>0.0934</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-05-19 21:00 AIC:              212781.1253\n",
       "No. Observations:   290584           BIC:              212823.4439\n",
       "Df Model:           3                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290580           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept     -2.0300    0.0266  -76.2488  0.0000  -2.0822  -1.9778\n",
       "ab_page       -0.0149    0.0114   -1.3069  0.1912  -0.0374   0.0075\n",
       "CA             0.0506    0.0284    1.7835  0.0745  -0.0050   0.1063\n",
       "UK             0.0408    0.0269    1.5161  0.1295  -0.0119   0.0934\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page', 'CA', 'UK']])\n",
    "results = logit_mod.fit()\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion**\n",
    ">**Countries have not any significant impact on conversion since P-Values are higher than 0.05.**\n",
    "*(P-CA = 0.0745)*\n",
    "*(P-UK = 0.1295)*\n",
    "\n",
    ">**Page type has no impact on conversion.**\n",
    "*(P-ab_page = 0.1912)*\n",
    "\n",
    ">**In conclusion, we fail to reject the null hypothesis.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**h. Fit your model and obtain the results**<br> \n",
    "Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if are there significant effects on conversion.  **Create the necessary additional columns, and fit the new model.** \n",
    "\n",
    "\n",
    "Provide the summary results (statistical output), and your conclusions (written response) based on the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df2['ab_UK'] = df2['ab_page'] * df2['UK']\n",
    "df2['ab_CA'] = df2['ab_page'] * df2['CA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-05-19 21:00</td>       <td>AIC:</td>        <td>212782.6602</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212846.1381</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>5</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290578</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-2.0040</td>  <td>0.0364</td>  <td>-55.0077</td> <td>0.0000</td> <td>-2.0754</td> <td>-1.9326</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0674</td>  <td>0.0520</td>   <td>-1.2967</td> <td>0.1947</td> <td>-0.1694</td> <td>0.0345</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0175</td>   <td>0.0377</td>   <td>0.4652</td>  <td>0.6418</td> <td>-0.0563</td> <td>0.0914</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>0.0118</td>   <td>0.0398</td>   <td>0.2957</td>  <td>0.7674</td> <td>-0.0663</td> <td>0.0899</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_UK</th>     <td>0.0469</td>   <td>0.0538</td>   <td>0.8718</td>  <td>0.3833</td> <td>-0.0585</td> <td>0.1523</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_CA</th>     <td>0.0783</td>   <td>0.0568</td>   <td>1.3783</td>  <td>0.1681</td> <td>-0.0330</td> <td>0.1896</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-05-19 21:00 AIC:              212782.6602\n",
       "No. Observations:   290584           BIC:              212846.1381\n",
       "Df Model:           5                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290578           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept     -2.0040    0.0364  -55.0077  0.0000  -2.0754  -1.9326\n",
       "ab_page       -0.0674    0.0520   -1.2967  0.1947  -0.1694   0.0345\n",
       "UK             0.0175    0.0377    0.4652  0.6418  -0.0563   0.0914\n",
       "CA             0.0118    0.0398    0.2957  0.7674  -0.0663   0.0899\n",
       "ab_UK          0.0469    0.0538    0.8718  0.3833  -0.0585   0.1523\n",
       "ab_CA          0.0783    0.0568    1.3783  0.1681  -0.0330   0.1896\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit_mod2 = sm.Logit(df2['converted'], df2[['intercept','ab_page','UK','CA','ab_UK', 'ab_CA']])\n",
    "results2 = logit_mod2.fit()\n",
    "results2.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After testing an interaction between country and pages, there is still no evidence to neglect null hypothesis. P values are still higher than 0.05.**\n",
    "\n",
    "***P-ab_UK = 0.3833***\n",
    "\n",
    "***P_ab_CA = 0.1681***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='finalcheck'></a>\n",
    "## Final Check!\n",
    "\n",
    "Congratulations!  You have reached the end of the A/B Test Results project!  You should be very proud of all you have accomplished!\n",
    "\n",
    "<a id='submission'></a>\n",
    "## Submission\n",
    "You may either submit your notebook through the \"SUBMIT PROJECT\" button at the bottom of this workspace, or you may work from your local machine and submit on  the last page of this project lesson.  \n",
    "\n",
    "1. Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).\n",
    "\n",
    "\n",
    "2. Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.\n",
    "\n",
    "\n",
    "3. Once you've done this, you can submit your project by clicking on the \"Submit Project\" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from subprocess import call\n",
    "call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}