Language/Python/Regular Expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Regular Expressions\"\n",
    "\n",
    "- toc: true \n",
    "- badges: true\n",
    "- comments: true\n",
    "- categories: [jupyter]\n",
    "- author: Abhinav Verma\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "\n",
    "Regex or regular expressions is in many ways a language within a language. It has its own syntax. It is used extensively in pattern matching. Many programmers make one of two mistakes regarding __regex__ , either they overuse it on every problem or they completely steer clear of it. To be honest , I'm not a fan of regexes. But as a machine learning engineer, you also need to spend a lot of work analyzing and cleaning the data to find patterns on it. You even need to scrape data from sites sometimes. These are places where knowledge of regular expressions comes in handy. So hopefully, this blog of my notes on learning regex can help a novice to navigate the complex world of regex. Even if your job doesn't involve working with data everyday, you will come across some regex code written by someone else , hence a knowledge of regex is vital. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a ton of libraries that also use regex for extracting patterns like urls from text. I've used some and they also come in handy , so I'll cover a couple of them here as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic regex using inbuilt functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has an inbuilt library __re__ for handling regex operations. But even before that , there are a lot of ways you can match simple patterns using the basic functionality of strings and numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = 'foo123bar'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need to find if the word string foo exists within s. Strings in python have this unique property of being iterable over its individual characters. You can also find a continuous pattern using the __in__ property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"foo\" in s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.find(\"foo\") #This gives the starting index of the pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.find(\"123\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also check for phone numbers now.\n",
    "\n",
    "You have different strings of phone numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "phone1 = \"022-64519300\"\n",
    "\n",
    "phone2 = \"+919820481234\"\n",
    "\n",
    "not_phone1 = \"101 Howard\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0123456789'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_phone(inp):\n",
    "    valid_chars = string.digits + ' -()+'\n",
    "    for char in inp:\n",
    "        if char not in valid_chars:\n",
    "            return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_phone(phone1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_phone(phone2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_phone(not_phone1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we defined a custom function which iterates over all characters in string and checks if it matches the valid characters that are usually present in phone numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enter the Regex\n",
    "\n",
    "Now there's a lot of simple use cases like the one above that can be solved without using regex , however the more complex the problem becomes simple string pattern matching can fail.\n",
    "\n",
    "One really useful to get started in regex is this link\n",
    "https://regexone.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re #import the re module for handling regular expressions in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1,s2,s3,s4,s5,s6 = \"can\",\"fan\",\"man\",\"dan\",\"ran\",\"pan\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'can'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to match the first 3 strings without matching the last 3. So if we look at the strings we can see a pattern. The string 'an' is common across all strings but the letters c,f,m are only present in the first 3 strings.\n",
    "There is a method for matching specific characters using regular expressions, by defining them inside square brackets. For example, the pattern [abc] will only match a single a, b, or c letter and nothing else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the search function in the __re__ module to search for the particular patterns we want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In python if you want to search for patterns either you can write the string or enclose the pattern inside the __r\"\"__ expression. This tells to python to search for this pattern.\n",
    "If the search finds a pattern it returns a Match object which is truthy object and tells if the match is found or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='c'>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"['cfm']\",s1)  # pattern searches if a string contains either c or f or m in the strinv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='f'>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"['cfm']\",s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.search(r\"['cfm']\",s6) # No match found hence nothing returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='can'>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"['cfm']an\",s1) # You can also search for the pattern of either c or f or m along with an"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Excluding specific characters__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases, we might know that there are specific characters that we don't want to match.\n",
    "To represent this, we use a similar expression that excludes specific characters using the square brackets and the ^ (hat). For example, the pattern [^abc] will match any single character except for the letters a, b, or c."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So if we take a look at the previous problem. We can reframe it as finding strings which do not have d,r or p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='fan'>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"[^'drp']an\",s2) # s2 does not have either of the 3 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Multiple matches__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to match a set of characters. Can be any letter but it has to occur atleast 3-5 times in a text. For e.g you want to find patterns where people speak in a casual slang like wasssuppp . One way to check for multiple occurences is using the square brackets method we used earlier. Just keep repeating it as many times as you want so for e.g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"[z][z][z]\",\"Wazzzup\") # search for mutliple occurences of uppercase or lowercase characters "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But honestly , how many times can you keep repeating this ? So regex provides a way out you can enclose a curly bracket where you specify a range which explains how many matches are you looking for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(2, 5), match='zzz'>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"[z]{2,5}\",\"Wazzzup\") # we are searching for multiple occurences of z between 2 and 5 times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regex metacharacters\n",
    "\n",
    "Now Regex has a ton of metacharacters. What are metacharacters? Well, how do I put it in laymans terms. They are characters that are useful in regex for finding patterns. They look similar to characters in python except that their meaning completely changes. Now this isn't the most technical definition that you'll find. For that , there are other blogs that have defined it better. I'll link them below. But more than definitions let's just start looking at some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "match_1 = \"aaaabcc\" \n",
    "match_2 = \"aabbbbc\"\n",
    "skip = \"a\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To match the match_{} variables and skip the skip variables , you would need to match one or more occurences of a,one or more b and one or more c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='aaaabcc'>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.match(r\"a+b+c+\",match_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='aabbbbc'>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.match(r\"a+b+c+\",match_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.match(r\"a+b+c+\",skip) #Nothing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can even wrap this in bool\n",
    "bool(re.match(r\"a+b+c+\",skip)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "re.search and re.match can be wrapped in bool to give boolean values for matches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's revisit the earlier regex. + matches one or more occurences of the preceding character. a+ will match one or more of a, b+ does the same for b and c+ does the same for c. So match variables get matched and the skip variables get skipped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you had to match zero or more b's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='aaaabcc'>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.match(r\"a+b*c+\",match_1) # * matches zero or more occurences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Optional Matches__\n",
    "\n",
    "This example has been ripped of from https://regexone.com/lesson/optional_characters? . I personally found this site to be really good practice on the concept of regex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "match_1,match_2,skip = \"1 file found?\",\"24 files found?\", \"No files found.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "? (question mark) metacharacter which denotes optionality. This metacharacter allows you to match either zero or one of the preceding character or group. For example, the pattern ab?c will match either the strings \"abc\" or \"ac\" because the b is considered optional. (Again ripped off from the site, it's really good)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1 file found?'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what do we need to match. Well you need to match numbers . Ordinarily that should be enough. so a \\d would work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But let's go further. to match file we would add a file. The s is optional so add a ? after s to indicate zero or more. Even a * would work though. Towards the end we have a ?. Now usually the ? mark is used in regex to indicate optionality. It's a metacharacter . (See what I did there). To match the normal ? you would need to escape it with a backslash(\\) character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.match(r\"\\d+files?found\\?\",match_1) #Didn't work. Because we didn't take into consideration space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 13), match='1 file found?'>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.match(r\"\\d+\\sfiles?\\sfound\\?\",match_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.match(r\"\\d+\\sfiles*\\sfound\\?\",skip) #it works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dot (.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The . metacharacter matches any single character except a newline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='fooxbar'>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search('foo.bar', 'fooxbar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='foo.bar'>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'foo\\.bar', 'foo.bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine for example we wanted to match the word \"success\" in a log file. We certainly don't want that pattern to match a line that says \"Error: unsuccessful operation\"! That is why it is often best practice to write as specific regular expressions as possible to ensure that we don't get false positives when matching against real world text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "match = \"Mission: successful\"\n",
    "skip = \"Next Mission: successful upon capture of target\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So over here we want to match string starting with Mission and ending with successful. You don't want any other things"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 19), match='Mission: successful'>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r\"^Mission:\\s*successful$\",match) # The ^ signifies string should start with the proceeding character and the $\n",
    "#specifies string should end with preceding character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.search(r\"^Mission:\\s*successful$\",skip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://realpython.com/regex-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've started on this path of regex . The above link has some really good resources for all the metacharacters. We've only covered a few. The rest would be covered over here and more from practice. Now that we've covered searching and matching and finding some patterns. We can actually go over to the other section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Difference between re.search and re.match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For re.match to work the pattern must match from the start of the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.match(r'foo.bar','afooxbar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(1, 8), match='fooxbar'>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'foo.bar','afooxbar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Search and Group using Regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose you have a large text and you want to extract specific parts from it. Say you're checking a text for extracting urls from it or you have a text and you want to extract only the date from it. This is where regex groups come in handy. Let's look at a simple example.\n",
    "\n",
    "Referenced from - https://www.datacamp.com/community/tutorials/python-regular-expression-tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "statement = 'Please contact us at: support@datacamp.com' # We need to extract email from here along with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "match = re.search(r'([\\w\\.-]+)@([\\w\\.-]+)', statement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paranthesis above group the extracted text in tuples which can then be accessed by indexing them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'support@datacamp.com'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.group() # This is the main group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('support', 'datacamp.com')"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.groups() # These are the sub-groups denoted by the parenthesis inside the main paranthesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you group the data it is grouped into the main group . However you can also divide the group into subgroups by adding parenthesis inside the main one indicating a separate subgroup. This can be accessed by the function groups() as shown above. The groups stores all subgroups in a tuple which can be accessed via index. And just like a tuple we also have a named version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named groups will make your code more readable. The syntax for creating named group is: (?P<name>...). Replace the name part with the name you want to give to your group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Email address: support@averma12.com\n",
      "Username: support\n",
      "Host: averma12.com\n"
     ]
    }
   ],
   "source": [
    "statement = 'Please contact us at: support@averma12.com'\n",
    "match = re.search(r'(?P<email>(?P<username>[\\w\\.-]+)@(?P<host>[\\w\\.-]+))', statement)\n",
    "if statement:\n",
    "    \n",
    "      print(\"Email address:\", match.group('email'))\n",
    "      print(\"Username:\", match.group('username'))\n",
    "      print(\"Host:\", match.group('host'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find all and substitute Regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now imagine a problem where you have a ton of emails or you have text where you want to extract all the urls present in a given text. You want all the information in an iterable like a list or an iterator which then allows you to later store or work on this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "statement = \"\"\"\n",
    "1.8GHz dual-core Intel Core i5, Turbo Boost up to 2.9GHz, with 3MB shared L3 cache, Configurable to 2.2GHz dual-core Intel Core i7, Turbo Boost up to 3.2GHz, with 4MB shared L3 cache\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n1.8GHz dual-core Intel Core i5, Turbo Boost up to 2.9GHz, with 3MB shared L3 cache, Configurable to 2.2GHz dual-core Intel Core i7, Turbo Boost up to 3.2GHz, with 4MB shared L3 cache\\n'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to extract the numbers from this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1. remove the new line \\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "statement = statement.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.8GHz dual-core Intel Core i5, Turbo Boost up to 2.9GHz, with 3MB shared L3 cache, Configurable to 2.2GHz dual-core Intel Core i7, Turbo Boost up to 3.2GHz, with 4MB shared L3 cache'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1.8GHz', '2.9GHz', '3MB', '2.2GHz', '3.2GHz', '4MB']"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r\"(\\d\\.?\\w+)\",statement,re.I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above command re.findall gets all the regex matches from a string in a list. , re.I is the optional third parameter in every regex function . re.I indicates ignore case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find all urls in a string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to find all the urls in a string. Our goal is to filter out all the urls in a string. This will make use of 2 regex functions. Regex findall and also re.sub to substitute the pattern with a string '<-URL->. This tokenization is done a lot to train models in text classification. URLs in many cases are substituted.\n",
    "Let's write a function to this and test it on some text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replace_urls(in_string):\n",
    "    \n",
    "    \"\"\"Replace URLs in strings. See also: ``bit.ly/PyURLre``\n",
    "\n",
    "    Args:\n",
    "        in_string (str): string to filter\n",
    "    \n",
    "\n",
    "    Returns:\n",
    "        str\n",
    "    \"\"\"\n",
    "    \n",
    "    replacement = '<-URL->'\n",
    "    pattern = re.compile('(https?://)?(\\w*[.]\\w+)+([/?=&]+\\w+)*')\n",
    "    return re.sub(pattern, replacement, in_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(https?://)? - You look for http or https pattern with :// pattern that you see with url. The ? at the end marks the pattern as optional. Putting the entire pattern in () groups it.\n",
    "The second group searches for alphanumeric characters with a . and more alphanumeric characters. One or more times\n",
    "The final group searches for literal ? for query params and =& along with alphanumeric characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take some example strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = \"My Profile https://auth.abc.org/user/averma/articles in the portal of http://www.abc.org/. Also you can find me on twitter.com and www.facebook.com .\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'My Profile <-URL-> in the portal of <-URL->/. Also you can find me on <-URL-> and <-URL-> .'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace_urls(string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some patterns that the above regex doesn't cover. Adding all cases would result in a huge regex. \n",
    "There are some libraries that take care of this and you can get away with not using regex. We'll be using the library urlextract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "#! pip install urlextract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urlextract import URLExtract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "extractor = URLExtract()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://auth.abc.org/user/averma/articles',\n",
       " 'http://www.abc.org/.',\n",
       " 'twitter.com',\n",
       " 'www.facebook.com']"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extractor.find_urls(string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replace_urls(in_string):\n",
    "        \n",
    "    replacement = '<-URL->'\n",
    "    extractor = URLExtract()\n",
    "    urls = extractor.find_urls(in_string)\n",
    "    for url in urls:\n",
    "        in_string = in_string.replace(url,replacement)\n",
    "\n",
    "    return in_string\n",
    "\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'My Profile <-URL-> in the portal of <-URL-> Also you can find me on <-URL-> and <-URL-> .'"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace_urls(string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "There are some amazing sources for learning regex\n",
    "\n",
    "\n",
    "https://github.com/fastai/course-nlp/blob/master/4-regex.ipynb Fastai nlp course has an amazing video on regex\n",
    "\n",
    "https://realpython.com/regex-python/ - Realpython is just an amazing resource for anything python\n",
    "\n",
    "\n",
    "https://regexone.com/ - Amazing resource to practice regex for beginners\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is intended to be a public resource. As such, if you see any glaring inaccuracies or if a critical topic is missing, please feel free to point it out or (preferably) submit a pull request to improve the tutorial. Also, we are always looking to improve the scope of this article. For anything feel free to mail us @ colearninglounge@gmail.com\n",
    "\n",
    "Author is __Abhinav Verma__ you can find him on [LinkedIn](\"https://linekdin.com/vermaonline\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}