Topic modeling on text data-2_Topic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Health news recommendation system using multi-level topic modeling.\n",
    "\n",
    "- Here we use the news dataset for development of recommendation system.\n",
    "- We apply topic modeling at multiple level for this.\n",
    "- Topic modelling help to build differet topic and among them we choose the health topic.\n",
    "- But the problem is to choose the correct number of topic which give more syncronised results of health data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the 'News dataset' for topic modelling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data=pd.read_csv('abcnews-date-text.csv',error_bad_lines=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_text = data[['headline_text']]\n",
    "data_text['index'] = data_text.index\n",
    "documents = data_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1103665\n",
      "                                       headline_text  index\n",
      "0  aba decides against community broadcasting lic...      0\n",
      "1     act fire witnesses must be aware of defamation      1\n",
      "2     a g calls for infrastructure protection summit      2\n",
      "3           air nz staff in aust strike for pay rise      3\n",
      "4      air nz strike to affect australian travellers      4\n"
     ]
    }
   ],
   "source": [
    "print(len(documents))\n",
    "print(documents[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preprocessing \n",
    "## Different process we do here like as Tokenization ,lemetization and stemming the data\n",
    "- we wanted to convert the data to their normal form for example 'stolen' would converted to steal\n",
    "### Here nltk is used for removing different language rather than english (Hindi and urdu like that word would be removed)\n",
    "- In preprocessing actually we remove all the punctuation marks , exclamatory marks and commas "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\SAMEER\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:1212: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\SAMEER\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "import gensim\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
    "from nltk.stem.porter import *\n",
    "import numpy as np\n",
    "np.random.seed(2018)\n",
    "import nltk\n",
    "nltk.download('wordnet')\n",
    "               \n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lemmatization \n",
    "## Lemmatization is used for gouping of word that's contains the same meaning(synonyms,antonyms)\n",
    "# Tokenization\n",
    "## Tokenization is used for keeps the word having meaningfull meaning\n",
    "- This is used for removal of word like if,the ,a,an that word doesn't make any sense in Topic\n",
    "# Stemming\n",
    "## Stemming is used for convert the word into their root form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lemmatize_stemming(text):\n",
    "    return WordNetLemmatizer().lemmatize(text, pos='v')\n",
    "def preprocess(text):\n",
    "    result = []\n",
    "    for token in gensim.utils.simple_preprocess(text):\n",
    "        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:\n",
    "            result.append(lemmatize_stemming(token))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# preview data after preprocessing \n",
    "- How the data will look like "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original document: \n",
      "['rain', 'helps', 'dampen', 'bushfires']\n",
      "\n",
      "\n",
      " tokenized and lemmatized document: \n",
      "['rain', 'help', 'dampen', 'bushfires']\n"
     ]
    }
   ],
   "source": [
    "# Select a document to preview after preprocessing\n",
    "doc_sample = documents[documents['index'] == 4310].values[0][0]\n",
    "print('original document: ')\n",
    "words = []\n",
    "for word in doc_sample.split(' '):\n",
    "    words.append(word)\n",
    "print(words)\n",
    "print('\\n\\n tokenized and lemmatized document: ')\n",
    "print(preprocess(doc_sample))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0              [decide, community, broadcast, licence]\n",
       "1                         [witness, aware, defamation]\n",
       "2           [call, infrastructure, protection, summit]\n",
       "3                          [staff, aust, strike, rise]\n",
       "4             [strike, affect, australian, travellers]\n",
       "5               [ambitious, olsson, win, triple, jump]\n",
       "6               [antic, delight, record, break, barca]\n",
       "7    [aussie, qualifier, stosur, waste, memphis, ma...\n",
       "8             [aust, address, security, council, iraq]\n",
       "9                         [australia, lock, timetable]\n",
       "Name: headline_text, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_docs = documents['headline_text'].map(preprocess)\n",
    "processed_docs[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionary is formed for "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 broadcast\n",
      "1 community\n",
      "2 decide\n",
      "3 licence\n",
      "4 aware\n",
      "5 defamation\n",
      "6 witness\n",
      "7 call\n",
      "8 infrastructure\n",
      "9 protection\n",
      "10 summit\n"
     ]
    }
   ],
   "source": [
    "dictionary = gensim.corpora.Dictionary(processed_docs)\n",
    "count = 0\n",
    "for k, v in dictionary.iteritems():\n",
    "    print(k, v)\n",
    "    count += 1\n",
    "    if count > 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "## filtering the number of word taken for building the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(112, 1), (489, 1), (970, 1), (4427, 1)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]\n",
    "bow_corpus[4310]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the model by using the 'pickle'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "pickle.dump(bow_corpus, open('bow_corpus.pkl', 'wb'))\n",
    "dictionary.save('dictionary.gensim')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word 112 (\"help\") appears 1 time.\n",
      "Word 489 (\"rain\") appears 1 time.\n",
      "Word 970 (\"bushfires\") appears 1 time.\n",
      "Word 4427 (\"dampen\") appears 1 time.\n"
     ]
    }
   ],
   "source": [
    "bow_doc_4310 = bow_corpus[4310]\n",
    "for i in range(len(bow_doc_4310)):\n",
    "    print(\"Word {} (\\\"{}\\\") appears {} time.\".format(bow_doc_4310[i][0], \n",
    "                                               dictionary[bow_doc_4310[i][0]], \n",
    "bow_doc_4310[i][1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.5955574839932255),\n",
      " (1, 0.394834368882524),\n",
      " (2, 0.4976240783705052),\n",
      " (3, 0.4917188028077664)]\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora, models\n",
    "tfidf = models.TfidfModel(bow_corpus)\n",
    "corpus_tfidf = tfidf[bow_corpus]\n",
    "from pprint import pprint\n",
    "for doc in corpus_tfidf:\n",
    "    pprint(doc)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=2, id2word=dictionary, passes=2, workers=2)\n",
    "lda_model.save('model2.gensim')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic: 0 \n",
      "Words: 0.012*\"police\" + 0.010*\"australia\" + 0.007*\"charge\" + 0.007*\"court\" + 0.006*\"sydney\" + 0.005*\"melbourne\" + 0.005*\"attack\" + 0.005*\"south\" + 0.005*\"murder\" + 0.005*\"kill\"\n",
      "Topic: 1 \n",
      "Words: 0.009*\"say\" + 0.007*\"trump\" + 0.006*\"government\" + 0.005*\"election\" + 0.005*\"plan\" + 0.005*\"australian\" + 0.004*\"world\" + 0.004*\"queensland\" + 0.004*\"change\" + 0.004*\"open\"\n"
     ]
    }
   ],
   "source": [
    "for idx, topic in lda_model.print_topics(-1):\n",
    "    print('Topic: {} \\nWords: {}'.format(idx, topic))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['rain', 'help', 'dampen', 'bushfires']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_docs[4310]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the model on unseen documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.5633942484855652\t Topic: 0.009*\"say\" + 0.007*\"trump\" + 0.006*\"government\" + 0.005*\"election\" + 0.005*\"plan\" + 0.005*\"australian\" + 0.004*\"world\" + 0.004*\"queensland\" + 0.004*\"change\" + 0.004*\"open\"\n",
      "Score: 0.4366057813167572\t Topic: 0.012*\"police\" + 0.010*\"australia\" + 0.007*\"charge\" + 0.007*\"court\" + 0.006*\"sydney\" + 0.005*\"melbourne\" + 0.005*\"attack\" + 0.005*\"south\" + 0.005*\"murder\" + 0.005*\"kill\"\n"
     ]
    }
   ],
   "source": [
    "unseen_document = 'How a Pentagon de to help him identity crisis for Google'\n",
    "bow_vector = dictionary.doc2bow(preprocess(unseen_document))\n",
    "for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):\n",
    "    print(\"Score: {}\\t Topic: {}\".format(score, lda_model.print_topic(index, 10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = gensim.corpora.Dictionary.load('dictionary.gensim')\n",
    "corpus = pickle.load(open('bow_corpus.pkl', 'rb'))\n",
    "lda = gensim.models.ldamodel.LdaModel.load('model2.gensim')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse the result by using the pyLDAvis "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\SAMEER\\Anaconda3\\lib\\site-packages\\pyLDAvis\\_prepare.py:257: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=True'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass sort=False\n",
      "\n",
      "  return pd.concat([default_term_info] + list(topic_dfs))\n"
     ]
    }
   ],
   "source": [
    "import pyLDAvis.gensim\n",
    "lda_display = pyLDAvis.gensim.prepare(lda, corpus, dictionary, sort_topics=False)\n",
    "pyLDAvis.display(lda_display)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## As we see the above results, 2 topic is not give the syncronized result/topic related to health.so, we try it with 5 topic."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}