NLP TooLs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk  # do nltk.download() if not already downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraph = \"\"\"Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen. 'Strong' AI is usually labelled as AGI (Artificial General Intelligence) while attempts to emulate 'natural' intelligence have been called ABI (Artificial Biological Intelligence). Leading AI textbooks define the field as the study of \"intelligent agents\": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term \"artificial intelligence\" is often used to describe machines (or computers) that mimic \"cognitive\" functions that humans associate with the human mind, such as \"learning\" and \"problem solving\".\n",
    "\n",
    "As machines become increasingly capable, tasks considered to require \"intelligence\" are often removed from the definition of AI, a phenomenon known as the AI effect. A quip in Tesler's Theorem says \"AI is whatever hasn't been done yet.\" For instance, optical character recognition is frequently excluded from things considered to be AI, having become a routine technology. Modern machine capabilities generally classified as AI include successfully understanding human speech, competing at the highest level in strategic game systems (such as chess and Go), autonomously operating cars, intelligent routing in content delivery networks, and military simulations.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenizing sentences\n",
    "sentences = nltk.sent_tokenize(paragraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenizing words\n",
    "words = nltk.word_tokenize(paragraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Artificial',\n",
       " 'intelligence',\n",
       " '(',\n",
       " 'AI',\n",
       " ')',\n",
       " 'is',\n",
       " 'intelligence',\n",
       " 'demonstrated',\n",
       " 'by',\n",
       " 'machines',\n",
       " ',',\n",
       " 'unlike',\n",
       " 'the',\n",
       " 'natural',\n",
       " 'intelligence',\n",
       " 'displayed',\n",
       " 'by',\n",
       " 'humans',\n",
       " 'and',\n",
       " 'animals',\n",
       " ',',\n",
       " 'which',\n",
       " 'involves',\n",
       " 'consciousness',\n",
       " 'and',\n",
       " 'emotionality',\n",
       " '.',\n",
       " 'The',\n",
       " 'distinction',\n",
       " 'between',\n",
       " 'the',\n",
       " 'former',\n",
       " 'and',\n",
       " 'the',\n",
       " 'latter',\n",
       " 'categories',\n",
       " 'is',\n",
       " 'often',\n",
       " 'revealed',\n",
       " 'by',\n",
       " 'the',\n",
       " 'acronym',\n",
       " 'chosen',\n",
       " '.',\n",
       " \"'Strong\",\n",
       " \"'\",\n",
       " 'AI',\n",
       " 'is',\n",
       " 'usually',\n",
       " 'labelled',\n",
       " 'as',\n",
       " 'AGI',\n",
       " '(',\n",
       " 'Artificial',\n",
       " 'General',\n",
       " 'Intelligence',\n",
       " ')',\n",
       " 'while',\n",
       " 'attempts',\n",
       " 'to',\n",
       " 'emulate',\n",
       " \"'natural\",\n",
       " \"'\",\n",
       " 'intelligence',\n",
       " 'have',\n",
       " 'been',\n",
       " 'called',\n",
       " 'ABI',\n",
       " '(',\n",
       " 'Artificial',\n",
       " 'Biological',\n",
       " 'Intelligence',\n",
       " ')',\n",
       " '.',\n",
       " 'Leading',\n",
       " 'AI',\n",
       " 'textbooks',\n",
       " 'define',\n",
       " 'the',\n",
       " 'field',\n",
       " 'as',\n",
       " 'the',\n",
       " 'study',\n",
       " 'of',\n",
       " '``',\n",
       " 'intelligent',\n",
       " 'agents',\n",
       " \"''\",\n",
       " ':',\n",
       " 'any',\n",
       " 'device',\n",
       " 'that',\n",
       " 'perceives',\n",
       " 'its',\n",
       " 'environment',\n",
       " 'and',\n",
       " 'takes',\n",
       " 'actions',\n",
       " 'that',\n",
       " 'maximize',\n",
       " 'its',\n",
       " 'chance',\n",
       " 'of',\n",
       " 'successfully',\n",
       " 'achieving',\n",
       " 'its',\n",
       " 'goals',\n",
       " '.',\n",
       " 'Colloquially',\n",
       " ',',\n",
       " 'the',\n",
       " 'term',\n",
       " '``',\n",
       " 'artificial',\n",
       " 'intelligence',\n",
       " \"''\",\n",
       " 'is',\n",
       " 'often',\n",
       " 'used',\n",
       " 'to',\n",
       " 'describe',\n",
       " 'machines',\n",
       " '(',\n",
       " 'or',\n",
       " 'computers',\n",
       " ')',\n",
       " 'that',\n",
       " 'mimic',\n",
       " '``',\n",
       " 'cognitive',\n",
       " \"''\",\n",
       " 'functions',\n",
       " 'that',\n",
       " 'humans',\n",
       " 'associate',\n",
       " 'with',\n",
       " 'the',\n",
       " 'human',\n",
       " 'mind',\n",
       " ',',\n",
       " 'such',\n",
       " 'as',\n",
       " '``',\n",
       " 'learning',\n",
       " \"''\",\n",
       " 'and',\n",
       " '``',\n",
       " 'problem',\n",
       " 'solving',\n",
       " \"''\",\n",
       " '.',\n",
       " 'As',\n",
       " 'machines',\n",
       " 'become',\n",
       " 'increasingly',\n",
       " 'capable',\n",
       " ',',\n",
       " 'tasks',\n",
       " 'considered',\n",
       " 'to',\n",
       " 'require',\n",
       " '``',\n",
       " 'intelligence',\n",
       " \"''\",\n",
       " 'are',\n",
       " 'often',\n",
       " 'removed',\n",
       " 'from',\n",
       " 'the',\n",
       " 'definition',\n",
       " 'of',\n",
       " 'AI',\n",
       " ',',\n",
       " 'a',\n",
       " 'phenomenon',\n",
       " 'known',\n",
       " 'as',\n",
       " 'the',\n",
       " 'AI',\n",
       " 'effect',\n",
       " '.',\n",
       " 'A',\n",
       " 'quip',\n",
       " 'in',\n",
       " 'Tesler',\n",
       " \"'s\",\n",
       " 'Theorem',\n",
       " 'says',\n",
       " '``',\n",
       " 'AI',\n",
       " 'is',\n",
       " 'whatever',\n",
       " 'has',\n",
       " \"n't\",\n",
       " 'been',\n",
       " 'done',\n",
       " 'yet',\n",
       " '.',\n",
       " \"''\",\n",
       " 'For',\n",
       " 'instance',\n",
       " ',',\n",
       " 'optical',\n",
       " 'character',\n",
       " 'recognition',\n",
       " 'is',\n",
       " 'frequently',\n",
       " 'excluded',\n",
       " 'from',\n",
       " 'things',\n",
       " 'considered',\n",
       " 'to',\n",
       " 'be',\n",
       " 'AI',\n",
       " ',',\n",
       " 'having',\n",
       " 'become',\n",
       " 'a',\n",
       " 'routine',\n",
       " 'technology',\n",
       " '.',\n",
       " 'Modern',\n",
       " 'machine',\n",
       " 'capabilities',\n",
       " 'generally',\n",
       " 'classified',\n",
       " 'as',\n",
       " 'AI',\n",
       " 'include',\n",
       " 'successfully',\n",
       " 'understanding',\n",
       " 'human',\n",
       " 'speech',\n",
       " ',',\n",
       " 'competing',\n",
       " 'at',\n",
       " 'the',\n",
       " 'highest',\n",
       " 'level',\n",
       " 'in',\n",
       " 'strategic',\n",
       " 'game',\n",
       " 'systems',\n",
       " '(',\n",
       " 'such',\n",
       " 'as',\n",
       " 'chess',\n",
       " 'and',\n",
       " 'Go',\n",
       " ')',\n",
       " ',',\n",
       " 'autonomously',\n",
       " 'operating',\n",
       " 'cars',\n",
       " ',',\n",
       " 'intelligent',\n",
       " 'routing',\n",
       " 'in',\n",
       " 'content',\n",
       " 'delivery',\n",
       " 'networks',\n",
       " ',',\n",
       " 'and',\n",
       " 'military',\n",
       " 'simulations',\n",
       " '.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stemming--------->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(sentences)):\n",
    "    words = nltk.word_tokenize(sentences[i])\n",
    "    words = [stemmer.stem(word) for word in words if word not in set(stopwords.words('english'))]\n",
    "    sentences[i] = ' '.join(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['artifici intellig ( AI ) intellig demonstr machin , unlik natur intellig display human anim , involv conscious emotion .',\n",
       " 'the distinct former latter categori often reveal acronym chosen .',\n",
       " \"'strong ' AI usual label agi ( artifici gener intellig ) attempt emul 'natur ' intellig call abi ( artifici biolog intellig ) .\",\n",
       " \"lead AI textbook defin field studi `` intellig agent '' : devic perceiv environ take action maxim chanc success achiev goal .\",\n",
       " \"colloqui , term `` artifici intellig '' often use describ machin ( comput ) mimic `` cognit '' function human associ human mind , `` learn '' `` problem solv '' .\",\n",
       " \"As machin becom increasingli capabl , task consid requir `` intellig '' often remov definit AI , phenomenon known AI effect .\",\n",
       " \"A quip tesler 's theorem say `` AI whatev n't done yet . ''\",\n",
       " 'for instanc , optic charact recognit frequent exclud thing consid AI , becom routin technolog .',\n",
       " 'modern machin capabl gener classifi AI includ success understand human speech , compet highest level strateg game system ( chess Go ) , autonom oper car , intellig rout content deliveri network , militari simul .']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lemmetization---------->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(sentences)):\n",
    "    words = nltk.word_tokenize(sentences[i])\n",
    "    words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords.words('english'))]\n",
    "    sentences[i] = ' '.join(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['artifici intellig ( AI ) intellig demonstr machin , unlik natur intellig display human anim , involv conscious emotion .',\n",
       " 'distinct former latter categori often reveal acronym chosen .',\n",
       " \"'strong ' AI usual label agi ( artifici gener intellig ) attempt emul 'natur ' intellig call abi ( artifici biolog intellig ) .\",\n",
       " 'lead AI textbook defin field studi `` intellig agent `` : devic perceiv environ take action maxim chanc success achiev goal .',\n",
       " 'colloqui , term `` artifici intellig `` often use describ machin ( comput ) mimic `` cognit `` function human associ human mind , `` learn `` `` problem solv `` .',\n",
       " 'As machin becom increasingli capabl , task consid requir `` intellig `` often remov definit AI , phenomenon known AI effect .',\n",
       " \"A quip tesler 's theorem say `` AI whatev n't done yet . ``\",\n",
       " 'instanc , optic charact recognit frequent exclud thing consid AI , becom routin technolog .',\n",
       " 'modern machin capabl gener classifi AI includ success understand human speech , compet highest level strateg game system ( chess Go ) , autonom oper car , intellig rout content deliveri network , militari simul .']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bag of words---------------->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = []\n",
    "for i in range(len(sentences)):\n",
    "    review = re.sub('[^a-zA-Z]', ' ', sentences[i])\n",
    "    review = review.lower()\n",
    "    review = review.split()\n",
    "    review = [stemmer.stem(word) for word in review if not word in set(stopwords.words('english'))]\n",
    "    review = ' '.join(review)\n",
    "    corpus.append(review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['artifici intellig ai intellig demonstr machin unlik natur intellig display human anim involv consciou emot',\n",
       " 'distinct former latter categori often reveal acronym chosen',\n",
       " 'strong ai usual label agi artifici gener intellig attempt emul natur intellig call abi artifici biolog intellig',\n",
       " 'lead ai textbook defin field studi intellig agent devic perceiv environ take action maxim chanc success achiev goal',\n",
       " 'colloqui term artifici intellig often use describ machin comput mimic cognit function human associ human mind learn problem solv',\n",
       " 'machin becom increasingli capabl task consid requir intellig often remov definit ai phenomenon known ai effect',\n",
       " 'quip tesler theorem say ai whatev n done yet',\n",
       " 'instanc optic charact recognit frequent exclud thing consid ai becom routin technolog',\n",
       " 'modern machin capabl gener classifi ai includ success understand human speech compet highest level strateg game system chess go autonom oper car intellig rout content deliveri network militari simul']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the Bag of Words model\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "cv = CountVectorizer()\n",
    "X = cv.fit_transform(corpus).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 109)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TFidf ----------------->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the TF-IDF model\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "cv = TfidfVectorizer()\n",
    "X = cv.fit_transform(corpus).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 109)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.13607805, 0.29030706, 0.21319255, 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.29030706, 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.29030706, 0.        ,\n",
       "       0.        , 0.29030706, 0.        , 0.        , 0.        ,\n",
       "       0.29030706, 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.21319255, 0.        ,\n",
       "       0.        , 0.        , 0.45280133, 0.29030706, 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.18836723, 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.24519797, 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.        , 0.29030706,\n",
       "       0.        , 0.        , 0.        , 0.        ])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}