2.1. DataManipulation_Check scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "865bd7e5",
   "metadata": {},
   "source": [
    "# 2. Data manipulation\n",
    "## 2.1. Checking the accuracy of data scraping\n",
    "\n",
    "The primary data source for our project is data scraped from the Channel One's website. The goal of this notebook is to ensure that the scraping algorithm is functioning properly. The notebook provides tests that may be used to detect problems in the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "33a85e25",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8e88480f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run \"2.0. DataManipulation_Functions.ipynb\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1050b950",
   "metadata": {},
   "outputs": [],
   "source": [
    "file = 'data/0-15_output_scraping_cleaned.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ac689c",
   "metadata": {},
   "source": [
    "## Data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "78f73e98",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_scraped = pd.read_csv(file, sep='|')\n",
    "df_scraped.rename(columns={'Unnamed: 0': 'id'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d396f386",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>tags_top</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "      <th>body</th>\n",
       "      <th>tags_bottom</th>\n",
       "      <th>video_duration_seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>2023-01-28 12:27:00</td>\n",
       "      <td>Общество</td>\n",
       "      <td>Выпуск новостей в 12:00 от 28.01.2023</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446131-vypu...</td>\n",
       "      <td>Смотрите в этом выпуске: танковый бой — задача...</td>\n",
       "      <td>Общество</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>2023-01-28 12:18:00</td>\n",
       "      <td>Общество,Культура</td>\n",
       "      <td>Смотрите на Первом канале программы и фильмы в...</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446141-smot...</td>\n",
       "      <td>Сегодня на Первом канале программы и фильмы с ...</td>\n",
       "      <td>Общество,Культура,Музыка,Кино,Телевидение,Доку...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>2023-01-28 12:17:00</td>\n",
       "      <td>Общество,Погода</td>\n",
       "      <td>В столичном регионе на предстоящей неделе темп...</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446139-v_st...</td>\n",
       "      <td>В России перепады температуры больше 40 градус...</td>\n",
       "      <td>Общество,Погода</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                 date           tags_top  \\\n",
       "0  25  2023-01-28 12:27:00           Общество   \n",
       "1   7  2023-01-28 12:18:00  Общество,Культура   \n",
       "2   8  2023-01-28 12:17:00    Общество,Погода   \n",
       "\n",
       "                                               title  \\\n",
       "0              Выпуск новостей в 12:00 от 28.01.2023   \n",
       "1  Смотрите на Первом канале программы и фильмы в...   \n",
       "2  В столичном регионе на предстоящей неделе темп...   \n",
       "\n",
       "                                                 url  \\\n",
       "0  https://www.1tv.ru/news/2023-01-28/446131-vypu...   \n",
       "1  https://www.1tv.ru/news/2023-01-28/446141-smot...   \n",
       "2  https://www.1tv.ru/news/2023-01-28/446139-v_st...   \n",
       "\n",
       "                                                body  \\\n",
       "0  Смотрите в этом выпуске: танковый бой — задача...   \n",
       "1  Сегодня на Первом канале программы и фильмы с ...   \n",
       "2  В России перепады температуры больше 40 градус...   \n",
       "\n",
       "                                         tags_bottom  video_duration_seconds  \n",
       "0                                           Общество                     NaN  \n",
       "1  Общество,Культура,Музыка,Кино,Телевидение,Доку...                     NaN  \n",
       "2                                    Общество,Погода                     NaN  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scraped.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ebe26551",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15011, 8)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scraped.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40903a52",
   "metadata": {},
   "source": [
    "## Adding columns necessary for scraping algorithm testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "df5c2b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_scraped = add_columns(df_scraped)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "307ece98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding time of a newscast extracted from its title\n",
    "df_scraped['newscast title time'] = df_scraped['title'].str.extract('^[а-яА-Я«» ]*([0-9]*):[0-9][0-9].*')\n",
    "df_scraped['newscast title time1'] = df_scraped['title'].str.extract('^[а-яА-Я«» ]*([0-9]*) час от .*')\n",
    "df_scraped['newscast title time'] = np.where(df_scraped['newscast title time1'].isnull(),\n",
    "                                             df_scraped['newscast title time'],\n",
    "                                             df_scraped['newscast title time1'])\n",
    "df_scraped['newscast title time'] = pd.to_numeric(df_scraped['newscast title time'], \n",
    "                                                  errors='coerce').astype('Int64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3f22247f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>tags_top</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "      <th>body</th>\n",
       "      <th>tags_bottom</th>\n",
       "      <th>video_duration_seconds</th>\n",
       "      <th>datetime</th>\n",
       "      <th>dat</th>\n",
       "      <th>year</th>\n",
       "      <th>year_month</th>\n",
       "      <th>hour</th>\n",
       "      <th>weekday</th>\n",
       "      <th>whole newscast</th>\n",
       "      <th>newscast title time</th>\n",
       "      <th>newscast title time1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>2023-01-28 12:27:00</td>\n",
       "      <td>Общество</td>\n",
       "      <td>Выпуск новостей в 12:00 от 28.01.2023</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446131-vypu...</td>\n",
       "      <td>Смотрите в этом выпуске: танковый бой — задача...</td>\n",
       "      <td>Общество</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2023-01-28 12:27:00</td>\n",
       "      <td>2023-01-28</td>\n",
       "      <td>2023</td>\n",
       "      <td>2023-01</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>True</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>2023-01-28 12:18:00</td>\n",
       "      <td>Общество,Культура</td>\n",
       "      <td>Смотрите на Первом канале программы и фильмы в...</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446141-smot...</td>\n",
       "      <td>Сегодня на Первом канале программы и фильмы с ...</td>\n",
       "      <td>Общество,Культура,Музыка,Кино,Телевидение,Доку...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2023-01-28 12:18:00</td>\n",
       "      <td>2023-01-28</td>\n",
       "      <td>2023</td>\n",
       "      <td>2023-01</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>2023-01-28 12:17:00</td>\n",
       "      <td>Общество,Погода</td>\n",
       "      <td>В столичном регионе на предстоящей неделе темп...</td>\n",
       "      <td>https://www.1tv.ru/news/2023-01-28/446139-v_st...</td>\n",
       "      <td>В России перепады температуры больше 40 градус...</td>\n",
       "      <td>Общество,Погода</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2023-01-28 12:17:00</td>\n",
       "      <td>2023-01-28</td>\n",
       "      <td>2023</td>\n",
       "      <td>2023-01</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                 date           tags_top  \\\n",
       "0  25  2023-01-28 12:27:00           Общество   \n",
       "1   7  2023-01-28 12:18:00  Общество,Культура   \n",
       "2   8  2023-01-28 12:17:00    Общество,Погода   \n",
       "\n",
       "                                               title  \\\n",
       "0              Выпуск новостей в 12:00 от 28.01.2023   \n",
       "1  Смотрите на Первом канале программы и фильмы в...   \n",
       "2  В столичном регионе на предстоящей неделе темп...   \n",
       "\n",
       "                                                 url  \\\n",
       "0  https://www.1tv.ru/news/2023-01-28/446131-vypu...   \n",
       "1  https://www.1tv.ru/news/2023-01-28/446141-smot...   \n",
       "2  https://www.1tv.ru/news/2023-01-28/446139-v_st...   \n",
       "\n",
       "                                                body  \\\n",
       "0  Смотрите в этом выпуске: танковый бой — задача...   \n",
       "1  Сегодня на Первом канале программы и фильмы с ...   \n",
       "2  В России перепады температуры больше 40 градус...   \n",
       "\n",
       "                                         tags_bottom  video_duration_seconds  \\\n",
       "0                                           Общество                     NaN   \n",
       "1  Общество,Культура,Музыка,Кино,Телевидение,Доку...                     NaN   \n",
       "2                                    Общество,Погода                     NaN   \n",
       "\n",
       "             datetime         dat  year year_month  hour  weekday  \\\n",
       "0 2023-01-28 12:27:00  2023-01-28  2023    2023-01    12        5   \n",
       "1 2023-01-28 12:18:00  2023-01-28  2023    2023-01    12        5   \n",
       "2 2023-01-28 12:17:00  2023-01-28  2023    2023-01    12        5   \n",
       "\n",
       "   whole newscast  newscast title time newscast title time1  \n",
       "0            True                   12                  NaN  \n",
       "1           False                 <NA>                  NaN  \n",
       "2           False                 <NA>                  NaN  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scraped.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f28fe06",
   "metadata": {},
   "source": [
    "## Removing the first and last days from the dataframe\n",
    "\n",
    "Because the loading for the first and last day is not complete, we must delete them in order to test the accuracy of data scraping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cb32e36c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2022, 6, 3)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_date = df_scraped['dat'].max()\n",
    "min_date = df_scraped['dat'].min()\n",
    "min_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "28dc437c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_scraped.drop(df_scraped[df_scraped['dat'].isin([max_date, min_date])].index, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "22b5d47b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14988, 17)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scraped.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5800166",
   "metadata": {},
   "source": [
    "## Data checks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a639dd",
   "metadata": {},
   "source": [
    "### Duplicates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "785e27a1",
   "metadata": {},
   "source": [
    "#### Row duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "94b1b152",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    14988\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols_subset = ['date', 'tags_top', 'title', 'url', 'body', 'tags_bottom',\n",
    "          'video_duration_seconds']\n",
    "row_duplicates = df_scraped.duplicated(subset=cols_subset, keep=False)\n",
    "row_duplicates.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970cb31c",
   "metadata": {},
   "source": [
    "#### Duplicates in column 'url'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "22c98a14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    14988\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols_subset = ['url']\n",
    "url_duplicates = df_scraped.duplicated(subset=cols_subset, keep=False)\n",
    "url_duplicates.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88d293b7",
   "metadata": {},
   "source": [
    "There are no duplicates in the examined file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3af88c2d",
   "metadata": {},
   "source": [
    "### Missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9478ff38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 14988 entries, 15 to 15002\n",
      "Data columns (total 17 columns):\n",
      " #   Column                  Non-Null Count  Dtype         \n",
      "---  ------                  --------------  -----         \n",
      " 0   id                      14988 non-null  int64         \n",
      " 1   date                    14988 non-null  object        \n",
      " 2   tags_top                14753 non-null  object        \n",
      " 3   title                   14988 non-null  object        \n",
      " 4   url                     14988 non-null  object        \n",
      " 5   body                    14953 non-null  object        \n",
      " 6   tags_bottom             14988 non-null  object        \n",
      " 7   video_duration_seconds  14977 non-null  float64       \n",
      " 8   datetime                14988 non-null  datetime64[ns]\n",
      " 9   dat                     14988 non-null  object        \n",
      " 10  year                    14988 non-null  int64         \n",
      " 11  year_month              14988 non-null  period[M]     \n",
      " 12  hour                    14988 non-null  int64         \n",
      " 13  weekday                 14988 non-null  int64         \n",
      " 14  whole newscast          14988 non-null  bool          \n",
      " 15  newscast title time     1443 non-null   Int64         \n",
      " 16  newscast title time1    0 non-null      object        \n",
      "dtypes: Int64(1), bool(1), datetime64[ns](1), float64(1), int64(4), object(8), period[M](1)\n",
      "memory usage: 2.0+ MB\n"
     ]
    }
   ],
   "source": [
    "df_scraped.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c775ec04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0\n",
       "date                          0\n",
       "tags_top                    235\n",
       "title                         0\n",
       "url                           0\n",
       "body                         35\n",
       "tags_bottom                   0\n",
       "video_duration_seconds       11\n",
       "datetime                      0\n",
       "dat                           0\n",
       "year                          0\n",
       "year_month                    0\n",
       "hour                          0\n",
       "weekday                       0\n",
       "whole newscast                0\n",
       "newscast title time       13545\n",
       "newscast title time1      14988\n",
       "dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of missing cells in columns\n",
    "df_scraped.isnull().sum(axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02b4a5ab",
   "metadata": {},
   "source": [
    "The columns \"body\" and \"video_duration_seconds\" are crucial for the analysis. We need to ensure that the algorithm scrapes them appropriately. We will carefully verify that their missing values are actually missing and that this is not an algorithm error. For this check, we will go to the Channel One website and examine the information offered there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d5581ba5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3738     https://www.1tv.ru/news/2022-11-25/442233-80_let_nazad_podpisano_soglashenie_o_formirovanii_frantsuzskoy_aviatsionnoy_eskadrili_normandiya_neman\n",
       "4691       https://www.1tv.ru/news/2022-11-10/441245-prezident_poruchil_privesti_normativy_obespechennosti_vs_rf_v_sootvetstvie_s_realnymi_potrebnostyami\n",
       "10241                             https://www.1tv.ru/news/2022-08-11/435453-s_territorii_ukrainy_v_osvobozhdennye_rayony_vozvraschayutsya_tselymi_semyami\n",
       "11087                                                                     https://www.1tv.ru/news/2022-07-30/434570-vypusk_novostey_v_10_00_ot_30_07_2022\n",
       "11411                                                                     https://www.1tv.ru/news/2022-07-25/434233-vypusk_novostey_v_18_00_ot_25_07_2022\n",
       "11817                                                             https://www.1tv.ru/news/2022-07-19/433804-vypusk_programmy_vremya_v_21_00_ot_19_07_2022\n",
       "11835                                                                     https://www.1tv.ru/news/2022-07-19/433795-vypusk_novostey_v_18_00_ot_19_07_2022\n",
       "12086                                                             https://www.1tv.ru/news/2022-07-15/433523-vypusk_programmy_vremya_v_21_00_ot_15_07_2022\n",
       "12107                                                                     https://www.1tv.ru/news/2022-07-15/433509-vypusk_novostey_v_18_00_ot_15_07_2022\n",
       "12384                                                             https://www.1tv.ru/news/2022-07-11/433212-vypusk_programmy_vremya_v_21_00_ot_11_07_2022\n",
       "12403                                                                     https://www.1tv.ru/news/2022-07-11/433196-vypusk_novostey_v_18_00_ot_11_07_2022\n",
       "12426                                                                     https://www.1tv.ru/news/2022-07-11/433183-vypusk_novostey_v_15_00_ot_11_07_2022\n",
       "12594                                                             https://www.1tv.ru/news/2022-07-08/432985-vypusk_programmy_vremya_v_21_00_ot_08_07_2022\n",
       "12610                                                                     https://www.1tv.ru/news/2022-07-08/432974-vypusk_novostey_v_18_00_ot_08_07_2022\n",
       "12689                                                             https://www.1tv.ru/news/2022-07-07/432900-vypusk_programmy_vremya_v_21_00_ot_07_07_2022\n",
       "12690                                                                     https://www.1tv.ru/news/2022-07-07/432893-vypusk_novostey_v_18_00_ot_07_07_2022\n",
       "12744                                                             https://www.1tv.ru/news/2022-07-06/432822-vypusk_programmy_vremya_v_21_00_ot_06_07_2022\n",
       "12762                                                                     https://www.1tv.ru/news/2022-07-06/432808-vypusk_novostey_v_18_00_ot_06_07_2022\n",
       "12819                                                             https://www.1tv.ru/news/2022-07-05/432737-vypusk_programmy_vremya_v_21_00_ot_05_07_2022\n",
       "12844                                                                     https://www.1tv.ru/news/2022-07-05/432728-vypusk_novostey_v_18_00_ot_05_07_2022\n",
       "13061                                                                     https://www.1tv.ru/news/2022-07-02/432495-vypusk_novostey_v_15_00_ot_02_07_2022\n",
       "13072                                                                     https://www.1tv.ru/news/2022-07-02/432481-vypusk_novostey_v_12_00_ot_02_07_2022\n",
       "13091                                                                     https://www.1tv.ru/news/2022-07-02/432471-vypusk_novostey_v_10_00_ot_02_07_2022\n",
       "13147                                                                     https://www.1tv.ru/news/2022-07-01/432408-vypusk_novostey_v_14_00_ot_01_07_2022\n",
       "13148                                                                     https://www.1tv.ru/news/2022-07-01/432385-vypusk_novostey_v_13_00_ot_01_07_2022\n",
       "13157                                                                     https://www.1tv.ru/news/2022-07-01/432369-vypusk_novostey_v_12_00_ot_01_07_2022\n",
       "13166                                                                     https://www.1tv.ru/news/2022-07-01/432358-vypusk_novostey_v_09_00_ot_01_07_2022\n",
       "13250                                                             https://www.1tv.ru/news/2022-06-29/432267-vypusk_programmy_vremya_v_21_00_ot_29_06_2022\n",
       "13266                                                                     https://www.1tv.ru/news/2022-06-29/432252-vypusk_novostey_v_18_00_ot_29_06_2022\n",
       "13277                                                                     https://www.1tv.ru/news/2022-06-29/432243-vypusk_novostey_v_15_00_ot_29_06_2022\n",
       "13405                                                                     https://www.1tv.ru/news/2022-06-27/432106-vypusk_novostey_v_18_00_ot_27_06_2022\n",
       "13575                                                             https://www.1tv.ru/news/2022-06-24/431923-vypusk_programmy_vremya_v_21_00_ot_24_06_2022\n",
       "13605                                                                     https://www.1tv.ru/news/2022-06-24/431895-vypusk_novostey_v_15_00_ot_24_06_2022\n",
       "13620                                                                     https://www.1tv.ru/news/2022-06-24/431873-vypusk_novostey_v_13_00_ot_24_06_2022\n",
       "13626                                                                     https://www.1tv.ru/news/2022-06-24/431860-vypusk_novostey_v_12_00_ot_24_06_2022\n",
       "Name: url, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check rows were body is missing\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "df_scraped.loc[df_scraped['body'].isnull(), 'url']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bb480e15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "798                                                 https://www.1tv.ru/news/2023-01-15/445302-skonchalsya_sovetskiy_i_gruzinskiy_pevets_akter_i_rezhisser_vahtang_kikabidze\n",
       "835                                                                        https://www.1tv.ru/news/2023-01-14/445265-ushla_iz_zhizni_narodnaya_artistka_sssr_inna_churikova\n",
       "1000                                                   https://www.1tv.ru/news/2023-01-11/445088-valeriy_gerasimov_naznachen_komanduyuschim_ob_edinennoy_gruppirovkoy_voysk\n",
       "1583                                                                               https://www.1tv.ru/news/2022-12-29/444474-umer_trehkratnyy_chempion_mira_po_futbolu_pele\n",
       "2064                     https://www.1tv.ru/news/2022-12-22/443973-dmitriy_peskov_vyrazil_sozhalenie_chto_vo_vremya_vizita_zelenskogo_v_ssha_ne_prozvuchalo_prizyvov_k_miru\n",
       "4154    https://www.1tv.ru/news/2022-11-18/441799-perspektivy_dalneyshego_rasshireniya_torgovo_ekonomicheskih_svyazey_moskvy_i_baku_obsudili_mihail_mishustin_i_ilham_aliev\n",
       "5768                                                               https://www.1tv.ru/news/2022-10-22/440119-v_pekine_zavershilsya_xx_s_ezd_kommunisticheskoy_partii_kitaya\n",
       "5884                                                                   https://www.1tv.ru/news/2022-10-20/439997-premier_ministr_velikobritanii_liz_trass_uhodit_v_otstavku\n",
       "7215                                              https://www.1tv.ru/news/2022-09-28/438589-bolee_700_spetsialistov_so_vsey_strany_ob_edinil_vserossiyskiy_forum_psihologov\n",
       "8594                                          https://www.1tv.ru/news/2022-09-05/437147-troe_zastryavshih_na_klyuchevskom_vulkane_turistov_vyshli_na_svyaz_s_rodstvennikami\n",
       "8911                                                                                                        https://www.1tv.ru/news/2022-08-30/436818-umer_mihail_gorbachev\n",
       "Name: url, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check rows were video duration is missing\n",
    "df_scraped.loc[df_scraped['video_duration_seconds'].isnull(), 'url']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "96ece993",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.reset_option(\"max_colwidth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66ae1790",
   "metadata": {},
   "source": [
    "All of the URLs given above were manually checked. The missing values are also missing on the Channel One website. So the algorithm works properly. The amount of missing values is negligible and will have no effect on the analysis's findings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc62568",
   "metadata": {},
   "source": [
    "### Anomalies in data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01260ee",
   "metadata": {},
   "source": [
    "To detect algorithm problems, we chose to compare the duration of the video with the entire newscast and the sum of the durations of individual stories from this newscast. Typically, the newscast lasts no more than 10 minutes longer than the aggregate of individual stories (these 10 minutes are for the introduction and a brief overview of the news release). If the news broadcast lasts longer than 10 minutes, this might indicate that the algorithm is not downloading all of the news stories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d4cece6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dat</th>\n",
       "      <th>hour</th>\n",
       "      <th>whole newscast</th>\n",
       "      <th>video_duration_seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>10</td>\n",
       "      <td>False</td>\n",
       "      <td>834.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>10</td>\n",
       "      <td>True</td>\n",
       "      <td>935.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>12</td>\n",
       "      <td>False</td>\n",
       "      <td>809.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>12</td>\n",
       "      <td>True</td>\n",
       "      <td>890.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>15</td>\n",
       "      <td>False</td>\n",
       "      <td>786.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2941</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>15</td>\n",
       "      <td>True</td>\n",
       "      <td>1006.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2942</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>18</td>\n",
       "      <td>False</td>\n",
       "      <td>2003.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2943</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>18</td>\n",
       "      <td>True</td>\n",
       "      <td>2391.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2944</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>21</td>\n",
       "      <td>False</td>\n",
       "      <td>2307.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2945</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>21</td>\n",
       "      <td>True</td>\n",
       "      <td>2422.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2946 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             dat  hour  whole newscast  video_duration_seconds\n",
       "0     2022-06-04    10           False                   834.0\n",
       "1     2022-06-04    10            True                   935.0\n",
       "2     2022-06-04    12           False                   809.0\n",
       "3     2022-06-04    12            True                   890.0\n",
       "4     2022-06-04    15           False                   786.0\n",
       "...          ...   ...             ...                     ...\n",
       "2941  2023-01-27    15            True                  1006.0\n",
       "2942  2023-01-27    18           False                  2003.0\n",
       "2943  2023-01-27    18            True                  2391.0\n",
       "2944  2023-01-27    21           False                  2307.0\n",
       "2945  2023-01-27    21            True                  2422.0\n",
       "\n",
       "[2946 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframe with duration of newscast and sum of duration of individual stories\n",
    "video_dur = df_scraped.groupby(['dat', 'hour', 'whole newscast'])[['video_duration_seconds']].sum()\n",
    "video_dur.reset_index(inplace=True)\n",
    "video_dur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "dccab7d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>whole newscast</th>\n",
       "      <th>dat</th>\n",
       "      <th>hour</th>\n",
       "      <th>sum duration of stories</th>\n",
       "      <th>newscast duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>10</td>\n",
       "      <td>834.0</td>\n",
       "      <td>935.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>12</td>\n",
       "      <td>809.0</td>\n",
       "      <td>890.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>15</td>\n",
       "      <td>786.0</td>\n",
       "      <td>873.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>18</td>\n",
       "      <td>1173.0</td>\n",
       "      <td>1253.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>21</td>\n",
       "      <td>1954.0</td>\n",
       "      <td>2073.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1547</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>13</td>\n",
       "      <td>673.0</td>\n",
       "      <td>864.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>14</td>\n",
       "      <td>583.0</td>\n",
       "      <td>1012.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1549</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>15</td>\n",
       "      <td>737.0</td>\n",
       "      <td>1006.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1550</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>18</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2391.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1551</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>21</td>\n",
       "      <td>2307.0</td>\n",
       "      <td>2422.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1552 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "whole newscast         dat  hour  sum duration of stories  newscast duration\n",
       "0               2022-06-04    10                    834.0              935.0\n",
       "1               2022-06-04    12                    809.0              890.0\n",
       "2               2022-06-04    15                    786.0              873.0\n",
       "3               2022-06-04    18                   1173.0             1253.0\n",
       "4               2022-06-04    21                   1954.0             2073.0\n",
       "...                    ...   ...                      ...                ...\n",
       "1547            2023-01-27    13                    673.0              864.0\n",
       "1548            2023-01-27    14                    583.0             1012.0\n",
       "1549            2023-01-27    15                    737.0             1006.0\n",
       "1550            2023-01-27    18                   2003.0             2391.0\n",
       "1551            2023-01-27    21                   2307.0             2422.0\n",
       "\n",
       "[1552 rows x 4 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.pivot_table(video_dur, values='video_duration_seconds', index = ['dat', 'hour'],\n",
    "                    columns=['whole newscast'], aggfunc=min)\n",
    "df.reset_index(inplace=True)\n",
    "df.rename(columns={False: 'sum duration of stories',\n",
    "                   True: 'newscast duration'},\n",
    "          inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7664bb09",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['diff'] = df['newscast duration'] - df['sum duration of stories']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f4d73583",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataframe with only newscasts (no individual news stories)\n",
    "newscasts = df_scraped[df_scraped['whole newscast']==True].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "79563970",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.merge(newscasts[['dat', 'hour', 'newscast title time']],\n",
    "              how='left',\n",
    "              left_on = ['dat', 'hour'],\n",
    "              right_on = ['dat', 'hour'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "46de0f91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dat</th>\n",
       "      <th>hour</th>\n",
       "      <th>sum duration of stories</th>\n",
       "      <th>newscast duration</th>\n",
       "      <th>diff</th>\n",
       "      <th>newscast title time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>10</td>\n",
       "      <td>834.0</td>\n",
       "      <td>935.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>12</td>\n",
       "      <td>809.0</td>\n",
       "      <td>890.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>15</td>\n",
       "      <td>786.0</td>\n",
       "      <td>873.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>18</td>\n",
       "      <td>1173.0</td>\n",
       "      <td>1253.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>21</td>\n",
       "      <td>1954.0</td>\n",
       "      <td>2073.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>13</td>\n",
       "      <td>673.0</td>\n",
       "      <td>864.0</td>\n",
       "      <td>191.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1549</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>14</td>\n",
       "      <td>583.0</td>\n",
       "      <td>1012.0</td>\n",
       "      <td>429.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1550</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>15</td>\n",
       "      <td>737.0</td>\n",
       "      <td>1006.0</td>\n",
       "      <td>269.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1551</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>18</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2391.0</td>\n",
       "      <td>388.0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1552</th>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>21</td>\n",
       "      <td>2307.0</td>\n",
       "      <td>2422.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1553 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             dat  hour  sum duration of stories  newscast duration   diff  \\\n",
       "0     2022-06-04    10                    834.0              935.0  101.0   \n",
       "1     2022-06-04    12                    809.0              890.0   81.0   \n",
       "2     2022-06-04    15                    786.0              873.0   87.0   \n",
       "3     2022-06-04    18                   1173.0             1253.0   80.0   \n",
       "4     2022-06-04    21                   1954.0             2073.0  119.0   \n",
       "...          ...   ...                      ...                ...    ...   \n",
       "1548  2023-01-27    13                    673.0              864.0  191.0   \n",
       "1549  2023-01-27    14                    583.0             1012.0  429.0   \n",
       "1550  2023-01-27    15                    737.0             1006.0  269.0   \n",
       "1551  2023-01-27    18                   2003.0             2391.0  388.0   \n",
       "1552  2023-01-27    21                   2307.0             2422.0  115.0   \n",
       "\n",
       "      newscast title time  \n",
       "0                      10  \n",
       "1                      12  \n",
       "2                      15  \n",
       "3                      18  \n",
       "4                      21  \n",
       "...                   ...  \n",
       "1548                   13  \n",
       "1549                   14  \n",
       "1550                   15  \n",
       "1551                   18  \n",
       "1552                   21  \n",
       "\n",
       "[1553 rows x 6 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e772eafe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hour</th>\n",
       "      <th>sum duration of stories</th>\n",
       "      <th>newscast duration</th>\n",
       "      <th>diff</th>\n",
       "      <th>newscast title time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1553.000000</td>\n",
       "      <td>1506.000000</td>\n",
       "      <td>1442.000000</td>\n",
       "      <td>1395.000000</td>\n",
       "      <td>1442.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>15.112041</td>\n",
       "      <td>1249.328685</td>\n",
       "      <td>1492.648405</td>\n",
       "      <td>288.993548</td>\n",
       "      <td>14.630374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>4.153196</td>\n",
       "      <td>852.050045</td>\n",
       "      <td>1116.134635</td>\n",
       "      <td>746.953920</td>\n",
       "      <td>3.858891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>358.000000</td>\n",
       "      <td>-1169.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>12.000000</td>\n",
       "      <td>798.250000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>12.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>926.000000</td>\n",
       "      <td>1032.500000</td>\n",
       "      <td>112.000000</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>18.000000</td>\n",
       "      <td>1433.750000</td>\n",
       "      <td>1510.500000</td>\n",
       "      <td>360.000000</td>\n",
       "      <td>18.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>23.000000</td>\n",
       "      <td>5903.000000</td>\n",
       "      <td>9346.000000</td>\n",
       "      <td>7472.000000</td>\n",
       "      <td>21.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              hour  sum duration of stories  newscast duration         diff  \\\n",
       "count  1553.000000              1506.000000        1442.000000  1395.000000   \n",
       "mean     15.112041              1249.328685        1492.648405   288.993548   \n",
       "std       4.153196               852.050045        1116.134635   746.953920   \n",
       "min       7.000000                 0.000000         358.000000 -1169.000000   \n",
       "25%      12.000000               798.250000         918.000000    61.000000   \n",
       "50%      14.000000               926.000000        1032.500000   112.000000   \n",
       "75%      18.000000              1433.750000        1510.500000   360.000000   \n",
       "max      23.000000              5903.000000        9346.000000  7472.000000   \n",
       "\n",
       "       newscast title time  \n",
       "count          1442.000000  \n",
       "mean             14.630374  \n",
       "std               3.858891  \n",
       "min               1.000000  \n",
       "25%              12.000000  \n",
       "50%              14.000000  \n",
       "75%              18.000000  \n",
       "max              21.000000  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854d987b",
   "metadata": {},
   "source": [
    "We looked at several rows that had a negative difference between the newcast duration and the total of individual stories durations. In these instances, the newscast video only contained a portion of the newscast. This information will have no bearing on our study because we will only utilize the scripts of individual stories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "db35ff55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dat</th>\n",
       "      <th>hour</th>\n",
       "      <th>sum duration of stories</th>\n",
       "      <th>newscast duration</th>\n",
       "      <th>diff</th>\n",
       "      <th>newscast title time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2022-06-05</td>\n",
       "      <td>23</td>\n",
       "      <td>584.0</td>\n",
       "      <td>7327.0</td>\n",
       "      <td>6743.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2022-06-07</td>\n",
       "      <td>14</td>\n",
       "      <td>981.0</td>\n",
       "      <td>2069.0</td>\n",
       "      <td>1088.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2022-06-07</td>\n",
       "      <td>14</td>\n",
       "      <td>981.0</td>\n",
       "      <td>2069.0</td>\n",
       "      <td>1088.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>2022-06-13</td>\n",
       "      <td>23</td>\n",
       "      <td>136.0</td>\n",
       "      <td>7521.0</td>\n",
       "      <td>7385.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>2022-06-16</td>\n",
       "      <td>22</td>\n",
       "      <td>409.0</td>\n",
       "      <td>4088.0</td>\n",
       "      <td>3679.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>2022-06-17</td>\n",
       "      <td>22</td>\n",
       "      <td>1307.0</td>\n",
       "      <td>5033.0</td>\n",
       "      <td>3726.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>2022-06-19</td>\n",
       "      <td>23</td>\n",
       "      <td>295.0</td>\n",
       "      <td>7498.0</td>\n",
       "      <td>7203.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>2022-06-21</td>\n",
       "      <td>13</td>\n",
       "      <td>219.0</td>\n",
       "      <td>859.0</td>\n",
       "      <td>640.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>2022-06-26</td>\n",
       "      <td>22</td>\n",
       "      <td>1459.0</td>\n",
       "      <td>7629.0</td>\n",
       "      <td>6170.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>2022-07-10</td>\n",
       "      <td>23</td>\n",
       "      <td>466.0</td>\n",
       "      <td>7894.0</td>\n",
       "      <td>7428.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            dat  hour  sum duration of stories  newscast duration    diff  \\\n",
       "11   2022-06-05    23                    584.0             7327.0  6743.0   \n",
       "24   2022-06-07    14                    981.0             2069.0  1088.0   \n",
       "25   2022-06-07    14                    981.0             2069.0  1088.0   \n",
       "71   2022-06-13    23                    136.0             7521.0  7385.0   \n",
       "93   2022-06-16    22                    409.0             4088.0  3679.0   \n",
       "101  2022-06-17    22                   1307.0             5033.0  3726.0   \n",
       "113  2022-06-19    23                    295.0             7498.0  7203.0   \n",
       "123  2022-06-21    13                    219.0              859.0   640.0   \n",
       "160  2022-06-26    22                   1459.0             7629.0  6170.0   \n",
       "260  2022-07-10    23                    466.0             7894.0  7428.0   \n",
       "\n",
       "     newscast title time  \n",
       "11                    21  \n",
       "24                    15  \n",
       "25                    14  \n",
       "71                    21  \n",
       "93                    21  \n",
       "101                   21  \n",
       "113                   21  \n",
       "123                   13  \n",
       "160                   21  \n",
       "260                   21  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check cases were > 10 min\n",
    "df[df['diff']>600].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13b5bb46",
   "metadata": {},
   "source": [
    "In most cases, the discrapency was caused by an inaccurate time in the news broadcast's \"date\" field (compare \"hour\" that was extracted from the field \"date\" and \"newscast title time\" that was extracted from the newscast title). This is another another example of inaccurate information on the Channel One website.\n",
    "\n",
    "This will have no effect on the outcome of our project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2f80822b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dat</th>\n",
       "      <th>hour</th>\n",
       "      <th>sum duration of stories</th>\n",
       "      <th>newscast duration</th>\n",
       "      <th>diff</th>\n",
       "      <th>newscast title time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2022-06-07</td>\n",
       "      <td>14</td>\n",
       "      <td>981.0</td>\n",
       "      <td>2069.0</td>\n",
       "      <td>1088.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>2022-06-21</td>\n",
       "      <td>13</td>\n",
       "      <td>219.0</td>\n",
       "      <td>859.0</td>\n",
       "      <td>640.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279</th>\n",
       "      <td>2022-07-13</td>\n",
       "      <td>14</td>\n",
       "      <td>217.0</td>\n",
       "      <td>972.0</td>\n",
       "      <td>755.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>2022-07-14</td>\n",
       "      <td>13</td>\n",
       "      <td>249.0</td>\n",
       "      <td>935.0</td>\n",
       "      <td>686.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>2022-07-18</td>\n",
       "      <td>14</td>\n",
       "      <td>249.0</td>\n",
       "      <td>907.0</td>\n",
       "      <td>658.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>424</th>\n",
       "      <td>2022-08-03</td>\n",
       "      <td>13</td>\n",
       "      <td>346.0</td>\n",
       "      <td>1082.0</td>\n",
       "      <td>736.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>2022-08-08</td>\n",
       "      <td>9</td>\n",
       "      <td>831.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>2022-08-08</td>\n",
       "      <td>14</td>\n",
       "      <td>343.0</td>\n",
       "      <td>994.0</td>\n",
       "      <td>651.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>546</th>\n",
       "      <td>2022-08-22</td>\n",
       "      <td>13</td>\n",
       "      <td>290.0</td>\n",
       "      <td>980.0</td>\n",
       "      <td>690.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>600</th>\n",
       "      <td>2022-08-30</td>\n",
       "      <td>14</td>\n",
       "      <td>206.0</td>\n",
       "      <td>856.0</td>\n",
       "      <td>650.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>2022-09-14</td>\n",
       "      <td>14</td>\n",
       "      <td>245.0</td>\n",
       "      <td>933.0</td>\n",
       "      <td>688.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800</th>\n",
       "      <td>2022-09-29</td>\n",
       "      <td>14</td>\n",
       "      <td>308.0</td>\n",
       "      <td>964.0</td>\n",
       "      <td>656.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>842</th>\n",
       "      <td>2022-10-05</td>\n",
       "      <td>14</td>\n",
       "      <td>188.0</td>\n",
       "      <td>886.0</td>\n",
       "      <td>698.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>858</th>\n",
       "      <td>2022-10-07</td>\n",
       "      <td>14</td>\n",
       "      <td>198.0</td>\n",
       "      <td>989.0</td>\n",
       "      <td>791.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>917</th>\n",
       "      <td>2022-10-17</td>\n",
       "      <td>14</td>\n",
       "      <td>320.0</td>\n",
       "      <td>1084.0</td>\n",
       "      <td>764.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>923</th>\n",
       "      <td>2022-10-18</td>\n",
       "      <td>13</td>\n",
       "      <td>408.0</td>\n",
       "      <td>1036.0</td>\n",
       "      <td>628.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>931</th>\n",
       "      <td>2022-10-19</td>\n",
       "      <td>14</td>\n",
       "      <td>245.0</td>\n",
       "      <td>882.0</td>\n",
       "      <td>637.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938</th>\n",
       "      <td>2022-10-20</td>\n",
       "      <td>14</td>\n",
       "      <td>355.0</td>\n",
       "      <td>958.0</td>\n",
       "      <td>603.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>966</th>\n",
       "      <td>2022-10-25</td>\n",
       "      <td>13</td>\n",
       "      <td>188.0</td>\n",
       "      <td>995.0</td>\n",
       "      <td>807.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>967</th>\n",
       "      <td>2022-10-25</td>\n",
       "      <td>14</td>\n",
       "      <td>179.0</td>\n",
       "      <td>902.0</td>\n",
       "      <td>723.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>980</th>\n",
       "      <td>2022-10-27</td>\n",
       "      <td>13</td>\n",
       "      <td>367.0</td>\n",
       "      <td>996.0</td>\n",
       "      <td>629.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>981</th>\n",
       "      <td>2022-10-27</td>\n",
       "      <td>14</td>\n",
       "      <td>359.0</td>\n",
       "      <td>979.0</td>\n",
       "      <td>620.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1004</th>\n",
       "      <td>2022-10-31</td>\n",
       "      <td>14</td>\n",
       "      <td>291.0</td>\n",
       "      <td>897.0</td>\n",
       "      <td>606.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>2022-11-01</td>\n",
       "      <td>14</td>\n",
       "      <td>243.0</td>\n",
       "      <td>984.0</td>\n",
       "      <td>741.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1018</th>\n",
       "      <td>2022-11-02</td>\n",
       "      <td>14</td>\n",
       "      <td>336.0</td>\n",
       "      <td>962.0</td>\n",
       "      <td>626.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1032</th>\n",
       "      <td>2022-11-04</td>\n",
       "      <td>18</td>\n",
       "      <td>55.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>844.0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1052</th>\n",
       "      <td>2022-11-08</td>\n",
       "      <td>14</td>\n",
       "      <td>211.0</td>\n",
       "      <td>876.0</td>\n",
       "      <td>665.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1059</th>\n",
       "      <td>2022-11-09</td>\n",
       "      <td>14</td>\n",
       "      <td>374.0</td>\n",
       "      <td>1007.0</td>\n",
       "      <td>633.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1067</th>\n",
       "      <td>2022-11-10</td>\n",
       "      <td>14</td>\n",
       "      <td>367.0</td>\n",
       "      <td>990.0</td>\n",
       "      <td>623.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1074</th>\n",
       "      <td>2022-11-11</td>\n",
       "      <td>14</td>\n",
       "      <td>298.0</td>\n",
       "      <td>957.0</td>\n",
       "      <td>659.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1117</th>\n",
       "      <td>2022-11-18</td>\n",
       "      <td>14</td>\n",
       "      <td>321.0</td>\n",
       "      <td>977.0</td>\n",
       "      <td>656.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1131</th>\n",
       "      <td>2022-11-21</td>\n",
       "      <td>13</td>\n",
       "      <td>349.0</td>\n",
       "      <td>988.0</td>\n",
       "      <td>639.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1132</th>\n",
       "      <td>2022-11-21</td>\n",
       "      <td>14</td>\n",
       "      <td>346.0</td>\n",
       "      <td>963.0</td>\n",
       "      <td>617.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1176</th>\n",
       "      <td>2022-11-28</td>\n",
       "      <td>14</td>\n",
       "      <td>353.0</td>\n",
       "      <td>970.0</td>\n",
       "      <td>617.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1189</th>\n",
       "      <td>2022-11-30</td>\n",
       "      <td>13</td>\n",
       "      <td>414.0</td>\n",
       "      <td>1137.0</td>\n",
       "      <td>723.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1231</th>\n",
       "      <td>2022-12-06</td>\n",
       "      <td>14</td>\n",
       "      <td>249.0</td>\n",
       "      <td>889.0</td>\n",
       "      <td>640.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1246</th>\n",
       "      <td>2022-12-08</td>\n",
       "      <td>14</td>\n",
       "      <td>330.0</td>\n",
       "      <td>963.0</td>\n",
       "      <td>633.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1268</th>\n",
       "      <td>2022-12-12</td>\n",
       "      <td>13</td>\n",
       "      <td>306.0</td>\n",
       "      <td>934.0</td>\n",
       "      <td>628.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1275</th>\n",
       "      <td>2022-12-13</td>\n",
       "      <td>13</td>\n",
       "      <td>291.0</td>\n",
       "      <td>909.0</td>\n",
       "      <td>618.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1276</th>\n",
       "      <td>2022-12-13</td>\n",
       "      <td>14</td>\n",
       "      <td>349.0</td>\n",
       "      <td>956.0</td>\n",
       "      <td>607.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1298</th>\n",
       "      <td>2022-12-16</td>\n",
       "      <td>14</td>\n",
       "      <td>267.0</td>\n",
       "      <td>952.0</td>\n",
       "      <td>685.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1313</th>\n",
       "      <td>2022-12-19</td>\n",
       "      <td>14</td>\n",
       "      <td>176.0</td>\n",
       "      <td>846.0</td>\n",
       "      <td>670.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1319</th>\n",
       "      <td>2022-12-20</td>\n",
       "      <td>13</td>\n",
       "      <td>219.0</td>\n",
       "      <td>990.0</td>\n",
       "      <td>771.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1334</th>\n",
       "      <td>2022-12-22</td>\n",
       "      <td>14</td>\n",
       "      <td>302.0</td>\n",
       "      <td>1014.0</td>\n",
       "      <td>712.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1358</th>\n",
       "      <td>2022-12-26</td>\n",
       "      <td>14</td>\n",
       "      <td>236.0</td>\n",
       "      <td>937.0</td>\n",
       "      <td>701.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1373</th>\n",
       "      <td>2022-12-28</td>\n",
       "      <td>15</td>\n",
       "      <td>364.0</td>\n",
       "      <td>1004.0</td>\n",
       "      <td>640.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1435</th>\n",
       "      <td>2023-01-10</td>\n",
       "      <td>13</td>\n",
       "      <td>297.0</td>\n",
       "      <td>988.0</td>\n",
       "      <td>691.0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1443</th>\n",
       "      <td>2023-01-11</td>\n",
       "      <td>14</td>\n",
       "      <td>197.0</td>\n",
       "      <td>978.0</td>\n",
       "      <td>781.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1520</th>\n",
       "      <td>2023-01-23</td>\n",
       "      <td>14</td>\n",
       "      <td>224.0</td>\n",
       "      <td>838.0</td>\n",
       "      <td>614.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             dat  hour  sum duration of stories  newscast duration    diff  \\\n",
       "25    2022-06-07    14                    981.0             2069.0  1088.0   \n",
       "123   2022-06-21    13                    219.0              859.0   640.0   \n",
       "279   2022-07-13    14                    217.0              972.0   755.0   \n",
       "285   2022-07-14    13                    249.0              935.0   686.0   \n",
       "313   2022-07-18    14                    249.0              907.0   658.0   \n",
       "424   2022-08-03    13                    346.0             1082.0   736.0   \n",
       "455   2022-08-08     9                    831.0             1467.0   636.0   \n",
       "458   2022-08-08    14                    343.0              994.0   651.0   \n",
       "546   2022-08-22    13                    290.0              980.0   690.0   \n",
       "600   2022-08-30    14                    206.0              856.0   650.0   \n",
       "701   2022-09-14    14                    245.0              933.0   688.0   \n",
       "800   2022-09-29    14                    308.0              964.0   656.0   \n",
       "842   2022-10-05    14                    188.0              886.0   698.0   \n",
       "858   2022-10-07    14                    198.0              989.0   791.0   \n",
       "917   2022-10-17    14                    320.0             1084.0   764.0   \n",
       "923   2022-10-18    13                    408.0             1036.0   628.0   \n",
       "931   2022-10-19    14                    245.0              882.0   637.0   \n",
       "938   2022-10-20    14                    355.0              958.0   603.0   \n",
       "966   2022-10-25    13                    188.0              995.0   807.0   \n",
       "967   2022-10-25    14                    179.0              902.0   723.0   \n",
       "980   2022-10-27    13                    367.0              996.0   629.0   \n",
       "981   2022-10-27    14                    359.0              979.0   620.0   \n",
       "1004  2022-10-31    14                    291.0              897.0   606.0   \n",
       "1011  2022-11-01    14                    243.0              984.0   741.0   \n",
       "1018  2022-11-02    14                    336.0              962.0   626.0   \n",
       "1032  2022-11-04    18                     55.0              899.0   844.0   \n",
       "1052  2022-11-08    14                    211.0              876.0   665.0   \n",
       "1059  2022-11-09    14                    374.0             1007.0   633.0   \n",
       "1067  2022-11-10    14                    367.0              990.0   623.0   \n",
       "1074  2022-11-11    14                    298.0              957.0   659.0   \n",
       "1117  2022-11-18    14                    321.0              977.0   656.0   \n",
       "1131  2022-11-21    13                    349.0              988.0   639.0   \n",
       "1132  2022-11-21    14                    346.0              963.0   617.0   \n",
       "1176  2022-11-28    14                    353.0              970.0   617.0   \n",
       "1189  2022-11-30    13                    414.0             1137.0   723.0   \n",
       "1231  2022-12-06    14                    249.0              889.0   640.0   \n",
       "1246  2022-12-08    14                    330.0              963.0   633.0   \n",
       "1268  2022-12-12    13                    306.0              934.0   628.0   \n",
       "1275  2022-12-13    13                    291.0              909.0   618.0   \n",
       "1276  2022-12-13    14                    349.0              956.0   607.0   \n",
       "1298  2022-12-16    14                    267.0              952.0   685.0   \n",
       "1313  2022-12-19    14                    176.0              846.0   670.0   \n",
       "1319  2022-12-20    13                    219.0              990.0   771.0   \n",
       "1334  2022-12-22    14                    302.0             1014.0   712.0   \n",
       "1358  2022-12-26    14                    236.0              937.0   701.0   \n",
       "1373  2022-12-28    15                    364.0             1004.0   640.0   \n",
       "1435  2023-01-10    13                    297.0              988.0   691.0   \n",
       "1443  2023-01-11    14                    197.0              978.0   781.0   \n",
       "1520  2023-01-23    14                    224.0              838.0   614.0   \n",
       "\n",
       "      newscast title time  \n",
       "25                     14  \n",
       "123                    13  \n",
       "279                    14  \n",
       "285                    13  \n",
       "313                    14  \n",
       "424                    13  \n",
       "455                     9  \n",
       "458                    14  \n",
       "546                    13  \n",
       "600                    14  \n",
       "701                    14  \n",
       "800                    14  \n",
       "842                    14  \n",
       "858                    14  \n",
       "917                    14  \n",
       "923                    13  \n",
       "931                    14  \n",
       "938                    14  \n",
       "966                    13  \n",
       "967                    14  \n",
       "980                    13  \n",
       "981                    14  \n",
       "1004                   14  \n",
       "1011                   14  \n",
       "1018                   14  \n",
       "1032                   18  \n",
       "1052                   14  \n",
       "1059                   14  \n",
       "1067                   14  \n",
       "1074                   14  \n",
       "1117                   14  \n",
       "1131                   13  \n",
       "1132                   14  \n",
       "1176                   14  \n",
       "1189                   13  \n",
       "1231                   14  \n",
       "1246                   14  \n",
       "1268                   13  \n",
       "1275                   13  \n",
       "1276                   14  \n",
       "1298                   14  \n",
       "1313                   14  \n",
       "1319                   13  \n",
       "1334                   14  \n",
       "1358                   14  \n",
       "1373                   15  \n",
       "1435                   13  \n",
       "1443                   14  \n",
       "1520                   14  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check discrepency for the cases with the correct newscast time\n",
    "df_for_check = df[(df['diff']>600)&(df['hour']==df['newscast title time'])]\n",
    "df_for_check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd674d5",
   "metadata": {},
   "source": [
    "The dates 2022-11-04, 2022-10-07, and 2023-01-23 were manually checked from the list above. This discrepancy is, once again, the fault of Channel One. They did not supply all of the individual news stories for these newscasts. In order to evaluate the potential impact on our project, we need to assess the share of such newscasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "834e0e17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.03398058252427184"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SShare of newscasts for which not all individual news stories were provided\n",
    "len(df_for_check) / len(newscasts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1dc2e31",
   "metadata": {},
   "source": [
    "### Manual check for one row\n",
    "Information from one randomly selected row was compared to information from the Channel One website. There was no discrepancy discovered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9111092a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>11224</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>10549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <td>2022-07-28 12:04:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tags_top</th>\n",
       "      <td>В мире</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>Бойцы 2-го армейского корпуса народной милиции ЛНР прикрывают колонны во время эвакуации людей</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>url</th>\n",
       "      <td>https://www.1tv.ru/news/2022-07-28/434433-boytsy_2_go_armeyskogo_korpusa_narodnoy_militsii_lnr_prikryvayut_kolonny_vo_vremya_evakuatsii_lyudey</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>Особая роль в защите мирного населения Донбасса у бойцов второго армейского корпуса народной милиции ЛНР. Они не только прикрывают людей от ударов националистов, ищут и уничтожают диверсантов, отбивают атаки и проводят успешные наступательные операции. На них еще возложена и огромная гуманитарная миссия, доставка топлива и продовольствия, эвакуация из опасных районов, разминирование территории и восстановление нормальных условий жизни. Евгений Лямин с подробностями.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tags_bottom</th>\n",
       "      <td>В мире,Специальная военная операция на Украине</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>video_duration_seconds</th>\n",
       "      <td>179.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datetime</th>\n",
       "      <td>2022-07-28 12:04:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dat</th>\n",
       "      <td>2022-07-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>2022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year_month</th>\n",
       "      <td>2022-07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hour</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weekday</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>whole newscast</th>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>newscast title time</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>newscast title time1</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         11224\n",
       "id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       10549\n",
       "date                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       2022-07-28 12:04:00\n",
       "tags_top                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                В мире\n",
       "title                                                                                                                                                                                                                                                                                                                                                                                                           Бойцы 2-го армейского корпуса народной милиции ЛНР прикрывают колонны во время эвакуации людей\n",
       "url                                                                                                                                                                                                                                                                                                                                                             https://www.1tv.ru/news/2022-07-28/434433-boytsy_2_go_armeyskogo_korpusa_narodnoy_militsii_lnr_prikryvayut_kolonny_vo_vremya_evakuatsii_lyudey\n",
       "body                    Особая роль в защите мирного населения Донбасса у бойцов второго армейского корпуса народной милиции ЛНР. Они не только прикрывают людей от ударов националистов, ищут и уничтожают диверсантов, отбивают атаки и проводят успешные наступательные операции. На них еще возложена и огромная гуманитарная миссия, доставка топлива и продовольствия, эвакуация из опасных районов, разминирование территории и восстановление нормальных условий жизни. Евгений Лямин с подробностями.\n",
       "tags_bottom                                                                                                                                                                                                                                                                                                                                                                                                                                                     В мире,Специальная военная операция на Украине\n",
       "video_duration_seconds                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   179.0\n",
       "datetime                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   2022-07-28 12:04:00\n",
       "dat                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 2022-07-28\n",
       "year                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      2022\n",
       "year_month                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             2022-07\n",
       "hour                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        12\n",
       "weekday                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      3\n",
       "whole newscast                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           False\n",
       "newscast title time                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       <NA>\n",
       "newscast title time1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       NaN"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', None)\n",
    "df_scraped.sample(1).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "cb2de5b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.reset_option('display.max_colwidth')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}