62a_text_feature_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Text Feature Extraction](https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction)\n",
    "\n",
    "### Bag of Words\n",
    "\n",
    "- ext Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.\n",
    "\n",
    "- scikit-learn provides utilities for the most common ways to extract numerical features from text content:\n",
    "\n",
    "    - `tokenizing` strings into integer ids for each possible token. Whitespace characters and punctuation are treated as token separators.\n",
    "\n",
    "    - `counting` the occurrences of tokens in each document.\n",
    "\n",
    "    - `normalizing` and weighting (with diminishing importance) tokens that occur in the majority of samples / documents.\n",
    "\n",
    "- Features and samples are defined as:\n",
    "\n",
    "    - each *individual token occurrence frequency* (normalized or not) is treated as a feature.\n",
    "\n",
    "    - the *vector of all the token frequencies for a given document* is considered a multivariate sample.\n",
    "\n",
    "- A corpus can thus be represented by a matrix with one row/document and one column/token (word).\n",
    "\n",
    "- Vectorization is the process of turning a collection of text documents into numerical feature vectors. The task (tokenization, counting and normalization) is called the *Bag of Words* or “Bag of n-grams” representation. Documents are *described by word occurrences* while ignoring the relative position information of the words in the document.\n",
    "\n",
    "### Sparsity\n",
    "\n",
    "- Most documents use a very small subset of the words used in a corpus. The resulting matrix will typically contain >99% zeroes).\n",
    "\n",
    "- Implementations typically use a sparse representation from scipy.sparse for storage.\n",
    "\n",
    "### [Count Vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer)\n",
    "\n",
    "- Does both tokenization and occurrence counting in a single class, such as this tiny corpus of text documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x9 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 19 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "corpus = [\n",
    "    'This is the first document.',\n",
    "    'This is the second second document.',\n",
    "    'And the third one.',\n",
    "    'Is this the first document?',]\n",
    "\n",
    "X = vectorizer.fit_transform(corpus)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The default configuration extracts words of at least 2 letters. This function can be requested explicitly.\n",
    "\n",
    "- Each term found during the fit is assigned a unique integer index corresponding to a column in the resulting matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "[[0 1 1 1 0 0 1 0 1]\n",
      " [0 1 0 1 0 2 1 0 1]\n",
      " [1 0 0 0 1 0 1 1 0]\n",
      " [0 1 1 1 0 0 1 0 1]]\n"
     ]
    }
   ],
   "source": [
    "analyze = vectorizer.build_analyzer()\n",
    "\n",
    "print(analyze(\"This is a text document to analyze.\") == (\n",
    "    ['this', 'is', 'text', 'document', 'to', 'analyze']))\n",
    "\n",
    "vectorizer.get_feature_names() == (\n",
    "    ['and', 'document', 'first', 'is', 'one',\n",
    "     'second', 'the', 'third', 'this'])\n",
    "\n",
    "print(X.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The map from feature name to column index is stored in `vocabulary_`.\n",
    "- Words not seen in the training corpus will be ignored in future calls to  `transform`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "[[0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.vocabulary_.get('document'))\n",
    "print(vectorizer.transform(['Something completely new.']).toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Previous corpus: the first and last documents have the same words, so are encoded in equal vectors. We lose the knowledge that the last document is a question. To preserve the local order information we can extract 2-grams of words in addition to the 1-grams (individual words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),\n",
    "                                    token_pattern=r'\\b\\w+\\b', min_df=1)\n",
    "analyze = bigram_vectorizer.build_analyzer()\n",
    "analyze('Bi-grams are cool!') == (\n",
    "    ['bi', 'grams', 'are', 'cool', 'bi grams', 'grams are', 'are cool'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This vocabular is much bigger. It can resolve ambiguities in the local position patterns.\n",
    "- For example, it knows \"is this\" is present in the last document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 1 1 1 1 1 0 0 0 0 0 1 1 0 0 0 0 1 1 0]\n",
      " [0 0 1 0 0 1 1 0 0 2 1 1 1 0 1 0 0 0 1 1 0]\n",
      " [1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 1 1 1 0 0 0]\n",
      " [0 0 1 1 1 1 0 1 0 0 0 0 1 1 0 0 0 0 1 0 1]]\n",
      "[0 0 0 1]\n"
     ]
    }
   ],
   "source": [
    "X_2 = bigram_vectorizer.fit_transform(corpus).toarray()\n",
    "print(X_2)\n",
    "\n",
    "feature_index = bigram_vectorizer.vocabulary_.get('is this')\n",
    "print(X_2[:, feature_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Words\n",
    "\n",
    "- Stop words (“and”, “the”, “him”, etc.), are assumed to be uninformative & which may be removed to avoid mistaking them for a signal. Sometimes, however, similar words are useful for prediction, such as in classifying writing style or personality.\n",
    "\n",
    "- There are several known issues in scikit's default ‘english’ stop word list. It does not aim to be a general, ‘one-size-fits-all’ solution as some tasks may require a more custom solution. See [NQY18] for more details.\n",
    "\n",
    "- Please take care in choosing a stop word list. Popular stop word lists may include words that are highly informative to some tasks, such as computer.\n",
    "\n",
    "- Ensure the stop word list has undergone the same preprocessing and tokenization as used in the vectorizer. The word `we’ve` is split into we and ve by CountVectorizer’s default tokenizer, so if `we’ve` is in stop_words, but ve is not, ve will be retained from we’ve in transformed text. Our vectorizers will try to identify and warn about some kinds of inconsistencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tf-Idf [Transformer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html#sklearn.feature_extraction.text.TfidfTransformer) and [Vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer)\n",
    "\n",
    "- In a large text corpus, some words (e.g. “the”, “a”, “is” in English) will convey little meaningful information. These very frequent terms would overshadow the frequencies of rarer yet more interesting terms in a classifier.\n",
    "\n",
    "- In order to re-weight the count features into floating point values suitable for usage by a classifier it is very common to use the tf–idf transform.\n",
    "\n",
    "- $\\text{tf-idf(t,d)}=\\text{tf(t,d)} \\times \\text{idf(t)}$\n",
    "\n",
    "- `TfidfTransformer` default settings: `TfidfTransformer(norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=False)`\n",
    "\n",
    "- *inverse document frequency (IDF)*: $\\text{idf}(t) = \\log{\\frac{1 + n}{1+\\text{df}(t)}} + 1$ where $n$ = the #documents in the corpus; $df(t)$ is the #documents in the corpus containing the term $t$.\n",
    "\n",
    "- The results are Euclidean-normalized: $v_{norm} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v{_1}^2 +\n",
    "v{_2}^2 + \\dots + v{_n}^2}}$\n",
    "\n",
    "- `smooth_idf=False` tells the Transformer & Vectorizer to add the \"1\" count to the idf instead of the idf's denominator: $\\text{idf}(t) = \\log{\\frac{n}{\\text{df}(t)}} + 1$\n",
    "\n",
    "### Example\n",
    "\n",
    "- 1st item is present 100% of the time = not very interesting.\n",
    "- 2nd,3rd items present <50% of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 2)\t0.5732079309279059\n",
      "  (0, 0)\t0.8194099510753754\n",
      "  (1, 0)\t1.0\n",
      "  (2, 0)\t1.0\n",
      "  (3, 0)\t1.0\n",
      "  (4, 1)\t0.8808994832762984\n",
      "  (4, 0)\t0.47330339145578754\n",
      "  (5, 2)\t0.8135516873095774\n",
      "  (5, 0)\t0.5814926070688599 \n",
      "\n",
      " [[0.81940995 0.         0.57320793]\n",
      " [1.         0.         0.        ]\n",
      " [1.         0.         0.        ]\n",
      " [1.         0.         0.        ]\n",
      " [0.47330339 0.88089948 0.        ]\n",
      " [0.58149261 0.         0.81355169]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "transformer = TfidfTransformer(smooth_idf=False)\n",
    "\n",
    "counts = [[3, 0, 1],\n",
    "          [2, 0, 0],\n",
    "          [3, 0, 0],\n",
    "          [4, 0, 0],\n",
    "          [3, 2, 0],\n",
    "          [3, 0, 2]]\n",
    "\n",
    "tfidf = transformer.fit_transform(counts)\n",
    "print(tfidf,\"\\n\\n\",tfidf.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.85151335, 0.        , 0.52433293],\n",
       "       [1.        , 0.        , 0.        ],\n",
       "       [1.        , 0.        , 0.        ],\n",
       "       [1.        , 0.        , 0.        ],\n",
       "       [0.55422893, 0.83236428, 0.        ],\n",
       "       [0.63035731, 0.        , 0.77630514]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer = TfidfTransformer()\n",
    "transformer.fit_transform(counts).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 2.25276297, 1.84729786])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model weights of each feature - from fit method\n",
    "transformer.idf_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Tfidf Vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer)\n",
    "\n",
    "- Combines Count Vectorizer and Tfidf Transformer in a single object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x9 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 19 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer()\n",
    "vectorizer.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binary Occurrences\n",
    "\n",
    "- *Binary occurrence markers* (using the `binary` param) may offer perform better in some case. Some estimators, `Bernoulli Naive Bayes`, in particular, explicitly model discrete boolean random variables. \n",
    "\n",
    "- Also, very short texts are likely to have noisy tf–idf values while the binary occurrence info is more stable.\n",
    "\n",
    "- Use cross validation to find the best feature extraction parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "from time import time\n",
    "import logging\n",
    "\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "857 documents\n",
      "2 categories\n"
     ]
    }
   ],
   "source": [
    "categories = [\n",
    "    'alt.atheism',\n",
    "    'talk.religion.misc',\n",
    "]\n",
    "# Uncomment the following to do the analysis on all the categories\n",
    "#categories = None\n",
    "\n",
    "data = fetch_20newsgroups(subset='train', categories=categories)\n",
    "print(\"%d documents\" % len(data.filenames))\n",
    "print(\"%d categories\" % len(data.target_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier()),\n",
    "])\n",
    "\n",
    "# uncommenting more parameters will give better exploring power but will\n",
    "# increase processing time in a combinatorial way\n",
    "parameters = {\n",
    "    'vect__max_df': (0.5, 0.75, 1.0),\n",
    "    # 'vect__max_features': (None, 5000, 10000, 50000),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
    "    # 'tfidf__use_idf': (True, False),\n",
    "    # 'tfidf__norm': ('l1', 'l2'),\n",
    "    'clf__max_iter': (20,),\n",
    "    'clf__alpha': (0.00001, 0.000001),\n",
    "    'clf__penalty': ('l2', 'elasticnet'),\n",
    "    # 'clf__max_iter': (10, 50, 80),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline: ['vect', 'tfidf', 'clf']\n",
      "parameters:\n",
      "{'clf__alpha': (1e-05, 1e-06),\n",
      " 'clf__max_iter': (20,),\n",
      " 'clf__penalty': ('l2', 'elasticnet'),\n",
      " 'vect__max_df': (0.5, 0.75, 1.0),\n",
      " 'vect__ngram_range': ((1, 1), (1, 2))}\n",
      "Fitting 5 folds for each of 24 candidates, totalling 120 fits\n",
      "done in 12.465s\n",
      "Best score: 0.952\n",
      "Best parameters set:\n",
      "\tclf__alpha: 1e-05\n",
      "\tclf__max_iter: 20\n",
      "\tclf__penalty: 'l2'\n",
      "\tvect__max_df: 1.0\n",
      "\tvect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1)\n",
    "\n",
    "print(\"pipeline:\", [name for name, _ in pipeline.steps])\n",
    "print(\"parameters:\")\n",
    "pprint(parameters)\n",
    "t0 = time()\n",
    "grid_search.fit(data.data, data.target)\n",
    "print(\"done in %0.3fs\" % (time() - t0))\n",
    "\n",
    "print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding Text files\n",
    "\n",
    "- Text is made of characters, but files are made of bytes. These bytes represent characters according to some encoding. To work with text files in Python, their bytes must be decoded to a character set called Unicode. Common encodings are ASCII, Latin-1 (Western Europe), KOI8-R (Russian) and the universal encodings UTF-8 and UTF-16. Many others exist.\n",
    "\n",
    "- An encoding can also be called a ‘character set’, but this term is less accurate: several encodings can exist for a single character set.\n",
    "\n",
    "- The text feature extractors in scikit-learn know how to decode text files, but only if you tell them what encoding the files are in. The CountVectorizer takes an `encoding` parameter for this purpose. For modern text files, the correct encoding is probably UTF-8, which is therefore the default (encoding=\"utf-8\").\n",
    "\n",
    "- If the text you are loading is not encoded with UTF-8, however, you will get a UnicodeDecodeError. The vectorizers can be muted about decoding errors by setting the `decode_error` to \"ignore\" or \"replace\". See the documentation for the Python function bytes.decode for more details (type help(bytes.decode) at the Python prompt).\n",
    "\n",
    "- If you are having trouble decoding text, here are some things to try:\n",
    "\n",
    "    - Find out what the actual encoding of the text is. The file might come with a header or README that tells you the encoding, or there might be some standard encoding you can assume based on where the text comes from.\n",
    "\n",
    "    - You may be able to find out what kind of encoding it is in general using the UNIX command file. The Python `chardet` module comes with a script called `chardetect.py` that will guess the specific encoding, though you cannot rely on its guess being correct.\n",
    "\n",
    "    - You could try UTF-8 and disregard the errors. You can decode byte strings with `bytes.decode(errors='replace')` to replace all decoding errors with a meaningless character, or set `decode_error='replace'` in the vectorizer. This may damage the usefulness of your features.\n",
    "\n",
    "    - Real text may come from a variety of sources that may have used different encodings, or even be sloppily decoded in a different encoding than the one it was encoded with. This is common in text retrieved from the Web. The Python package ftfy can automatically sort out some classes of decoding errors, so you could try decoding the unknown text as `latin-1` and then using `ftfy` to fix errors.\n",
    "\n",
    "    - If the text is in a mish-mash of encodings that is simply too hard to sort out (which is the case for the 20 Newsgroups dataset), you can fall back on a simple single-byte encoding such as `latin-1`. Some text may display incorrectly, but at least the same sequence of bytes will always represent the same feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n",
      "{'sei': 15, 'mir': 13, 'gegrüßt': 6, 'mein': 12, 'sauerkraut': 14, 'holdselig': 10, 'sind': 16, 'deine': 1, 'gerüche': 7, 'auf': 0, 'flügeln': 4, 'des': 2, 'gesanges': 8, 'herzliebchen': 9, 'trag': 17, 'ich': 11, 'dich': 3, 'fort': 5}\n"
     ]
    }
   ],
   "source": [
    "import chardet    \n",
    "text1 = b\"Sei mir gegr\\xc3\\xbc\\xc3\\x9ft mein Sauerkraut\"\n",
    "text2 = b\"holdselig sind deine Ger\\xfcche\"\n",
    "text3 = b\"\\xff\\xfeA\\x00u\\x00f\\x00 \\x00F\\x00l\\x00\\xfc\\x00g\\x00e\\x00l\\x00n\\x00 \\x00d\\x00e\\x00s\\x00 \\x00G\\x00e\\x00s\\x00a\\x00n\\x00g\\x00e\\x00s\\x00,\\x00 \\x00H\\x00e\\x00r\\x00z\\x00l\\x00i\\x00e\\x00b\\x00c\\x00h\\x00e\\x00n\\x00,\\x00 \\x00t\\x00r\\x00a\\x00g\\x00 \\x00i\\x00c\\x00h\\x00 \\x00d\\x00i\\x00c\\x00h\\x00 \\x00f\\x00o\\x00r\\x00t\\x00\"\n",
    "decoded = [x.decode(chardet.detect(x)['encoding'])\n",
    "           for x in (text1, text2, text3)]        \n",
    "v = CountVectorizer().fit(decoded).vocabulary_    \n",
    "for term in v: print(v)                           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of Words Limitations\n",
    "\n",
    "- Unigrams (aka bag of words) cannot capture phrases and multi-word expressions, effectively disregarding any word order dependence. \n",
    "\n",
    "- Bag of words models can't account for misspellings or word derivations.\n",
    "\n",
    "- Instead, consider building a *collection of bigrams (n=2)*, which counts occurrences of consecutive-word pairs.\n",
    "\n",
    "- Or, consider a *collection of character n-grams*, which is more resilient against misspellings and derivations.\n",
    "\n",
    "### Example:\n",
    "    \n",
    "- a corpus of two documents: ['words', 'wprds']. \n",
    "- The 2nd document contains a misspelling of the word ‘words’. \n",
    "- A simple BoW model considers them as very distinct documents, differing in both of the two possible features. \n",
    "- A character 2-gram representation would find the documents *matching in 4 out of 8 features*, which may help a classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1, 0, 1, 1, 1, 0],\n",
       "       [1, 1, 0, 1, 1, 1, 0, 1]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_vectorizer = CountVectorizer(analyzer='char_wb', \n",
    "                                   ngram_range=(2, 2))\n",
    "counts           = ngram_vectorizer.fit_transform(['words', \n",
    "                                                   'wprds'])\n",
    "print(ngram_vectorizer.get_feature_names() == (\n",
    "    [' w', 'ds', 'or', 'pr', 'rd', 's ', 'wo', 'wp']))\n",
    "\n",
    "counts.toarray().astype(int)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Above: `char_wb` analyzer is used. It creates n-grams only from characters inside word boundaries (padded with space on each side). \n",
    "- Below: The `char` analyzer creates n-grams that span across words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True \n",
      "\n",
      "True \n",
      "\n"
     ]
    }
   ],
   "source": [
    "ngram_vectorizer = CountVectorizer(analyzer='char_wb', \n",
    "                                   ngram_range=(5, 5))\n",
    "ngram_vectorizer.fit_transform(['jumpy fox'])\n",
    "print(ngram_vectorizer.get_feature_names() == (\n",
    "    [' fox ', ' jump', 'jumpy', 'umpy ']),\"\\n\")\n",
    "\n",
    "\n",
    "ngram_vectorizer = CountVectorizer(analyzer='char', \n",
    "                                   ngram_range=(5, 5))\n",
    "ngram_vectorizer.fit_transform(['jumpy fox'])\n",
    "print(ngram_vectorizer.get_feature_names() == (\n",
    "    ['jumpy', 'mpy f', 'py fo', 'umpy ', 'y fox']),\"\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `char_wb` is especially interesting for languages that use whitespace for word separation - it generates significantly less noisy features than the raw char variant. \n",
    "\n",
    "- It can increase both predictive accuracy and convergence speed of classifiers while retaining the robustness to misspellings and word derivations.\n",
    "\n",
    "- While local position information can be preserved by extracting n-grams instead of individual words, BoW and bag of n-grams models destroy most of the inner structure of the document - hence most of the meaning.\n",
    "\n",
    "- To address the wider task of Natural Language Understanding, the local structure of sentences and paragraphs should thus be taken into account. Many such models will thus be casted as “Structured output” problems which are currently outside of the scope of scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Hashing Trick\n",
    "\n",
    "- Simple vectorization uses *in-memory mapping* from the string tokens to the integer feature indices (`the vocabulary_`). This causes several problems when dealing with large datasets:\n",
    "\n",
    "- The larger the corpus, the larger the vocabulary  - hence the memory use too.\n",
    "\n",
    "- Fitting requires *intermediate data structures* of size proportional to the original dataset.\n",
    "\n",
    "- Building word maps requires a full pass over the dataset - so it is not possible to fit text classifiers in an online manner.\n",
    "\n",
    "- Pickling/un-pickling vectorizers with a large `vocabulary_` can be very slow.\n",
    "\n",
    "- It's not easy to split vectorization into concurrent subtasks -  `vocabulary_` would have to be a shared state with a fine grained synchronization barrier.\n",
    "\n",
    "- It's possible to overcome these issues by combining the *“hashing trick”* (Feature hashing, by `FeatureHasher`) plus text preprocessing & tokenization (by `CountVectorizer`).\n",
    "\n",
    "- This combination is built into `HashingVectorizer`, a transformer class that is mostly API compatible with `CountVectorizer`. `HashingVectorizer` is stateless, meaning that you don’t have to call fit on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x10 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 16 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "hv = HashingVectorizer(n_features=10)\n",
    "hv.transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 16 non-zero feature tokens were extracted: this is less than the 19 non-zeros extracted by `CountVectorizer` on the same corpus. The discrepancy comes from hash function collisions due to the low `n_features` parameter value.\n",
    "\n",
    "- In a real world setting, `n_features` can be left to its default of 2^20 (roughly 1e6 possible features). If memory or downstream model size is an issue, use a lower value such as 2^18.\n",
    "\n",
    "- Dimensionality does not affect training time of algorithms which operate on *CSR matrices* (LinearSVC(dual=True), Perceptron, SGDClassifier, PassiveAggressive). It *does* for algorithms that work with *CSC matrices* (LinearSVC(dual=False), Lasso(), etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4x1048576 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 19 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hv = HashingVectorizer()\n",
    "hv.transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We no longer get the collisions, but we need a much larger output space dimensionality. Of course, other terms than these 19 might still collide.\n",
    "\n",
    "- `HashingVectorizer` comes with the following limitations:\n",
    "\n",
    "    - It is not possible to invert the model (no `inverse_transform` method), nor to access the original string representation of the features, because of the *one-way nature of the hash function* that performs the mapping.\n",
    "\n",
    "- It does not provide IDF weighting as that would introduce statefulness in the model. A `TfidfTransformer` can be appended in a pipeline if required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Out-of-core Scaling with [Hashing Vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html#sklearn.feature_extraction.text.HashingVectorizer)\n",
    "\n",
    "- This allows learning from data that does not fit into main memory.\n",
    "\n",
    "- The idea is to stream data to the estimator in *mini-batches*. Each mini-batch is vectorized to guarantee the estimator's input space always has the same dimensionality. \n",
    "\n",
    "- The amount of memory used at any time is thus bounded by the size of a mini-batch. Although there is no limit to the amount of data ingested using this approach, learning time is usually limited by CPU runtime budget."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Vectorizer Classes\n",
    "\n",
    "- Customize the behavior by passing a callable to the vectorizer constructor.\n",
    "\n",
    "- `preprocessor`: a callable that ingests an entire document as a single string) & returns a possibly transformed version - still as an entire string. This can be used to remove HTML tags, lowercasing, etc.\n",
    "\n",
    "- `tokenizer`: a callable. Takes the output from the preprocessor and returns a list of tokens.\n",
    "\n",
    "- `analyzer`: a callable that replaces the preprocessor and tokenizer. The default analyzers all call the preprocessor and tokenizer, but custom analyzers will skip this. N-gram extraction and stop word filtering take place at the analyzer level, so a custom analyzer may have to reproduce these steps.\n",
    "\n",
    "- If documents are pre-tokenized by an external package, store them in files (or strings) with the tokens separated by whitespace and pass `analyzer=str.split`.\n",
    "\n",
    "- Token-level analysis such as stemming, lemmatizing, compound splitting, filtering based on part-of-speech, etc. are not included in the scikit-learn but can be added by customizing either the tokenizer or the analyzer. Here’s a CountVectorizer with a tokenizer and lemmatizer using `NLTK`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize          \n",
    "from nltk.stem import WordNetLemmatizer \n",
    "class LemmaTokenizer:\n",
    "    def __init__(self):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "    def __call__(self, doc):\n",
    "        return [self.wnl.lemmatize(t) for t in word_tokenize(doc)]\n",
    "\n",
    "vect = CountVectorizer(tokenizer=LemmaTokenizer()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This example transforms British spelling to American spelling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['color', 'color']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "def to_british(tokens):\n",
    "    for t in tokens:\n",
    "        t = re.sub(r\"(...)our$\", r\"\\1or\", t)\n",
    "        t = re.sub(r\"([bt])re$\", r\"\\1er\", t)\n",
    "        t = re.sub(r\"([iy])s(e$|ing|ation)\", r\"\\1z\\2\", t)\n",
    "        t = re.sub(r\"ogue$\", \"og\", t)\n",
    "        yield t\n",
    "\n",
    "class CustomVectorizer(CountVectorizer):\n",
    "    def build_tokenizer(self):\n",
    "        tokenize = super().build_tokenizer()\n",
    "        return lambda doc: list(to_british(tokenize(doc)))\n",
    "\n",
    "print(CustomVectorizer().build_analyzer()(u\"color colour\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:working] *",
   "language": "python",
   "name": "conda-env-working-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}