Chapter 7 - File handling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 7:  File Handling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter you will learn how to read data from and write data to files. This is quite an essential part of programming, as it is the first step for your program to communicate with the outside world. In most cases you will write programs that take data from some source, manipulate it in someway and write it out somewhere. For example if you would write a survey, you could take input from participants and save their answers in some files. When the survey is over you would read these files in and do some analysis on the data you have collected and save your results. In this chapter we will read in text files, analyze them a bit, and save out our analysis to files. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input for your programs often comes from files on your disk, such as texts or some data in csv format. Likewise, you often want output to be written back to files on your disk as well e.g.: you collect tweets about a certain topic and you write it to a file for later analysis. Thus, reading and writing files is often an essential part of programming and, lucky, for us, this is really simple in Python. The following example reads a file from disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_io.TextIOWrapper name='./data/austen-emma-excerpt.txt' mode='r' encoding='utf-8'>\n",
      "Emma by Jane Austen 1816\n",
      "\n",
      "VOLUME I\n",
      "\n",
      "CHAPTER I\n",
      "\n",
      "\n",
      "Emma Woodhouse, handsome, clever, and rich, with a comfortable home\n",
      "and happy disposition, seemed to unite some of the best blessings\n",
      "of existence; and had lived nearly twenty-one years in the world\n",
      "with very little to distress or vex her.\n",
      "\n",
      "She was the youngest of the two daughters of a most affectionate,\n",
      "indulgent father; and had, in consequence of her sister's marriage,\n",
      "been mistress of his house from a very early period.  Her mother\n",
      "had died too long ago for her to have more than an indistinct\n",
      "remembrance of her caresses; and her place had been supplied\n",
      "by an excellent woman as governess, who had fallen little short\n",
      "of a mother in affection.\n"
     ]
    }
   ],
   "source": [
    "f = open('./data/austen-emma-excerpt.txt', 'r', encoding='utf-8') # open the file \n",
    "text = f.read() # read in its content as a string\n",
    "f.close() # close the file\n",
    "print(text) # print the string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `open()` function does not return the actual text that is saved in the text file. It only returns a 'file object' from which we can read the content using the `.read()` function. We passed three arguments to the `open()` function:\n",
    "\n",
    " * the name of the file that you wish to open\n",
    " * the mode, a combination of characters, 'r' represents read-mode, and 't' represent plain text-mode. This indicates we are reading a plain text file.\n",
    " * the last argument, a named argument (encoding), specifies the encoding of the text file.\n",
    " \n",
    "The most important mode arguments the open() function can take are:\n",
    "\n",
    "* r: Opens a file for reading only. The file pointer is placed at the beginning of the file.\n",
    "* w: Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.\n",
    "* a: Opens a file for appending. The file pointer is at the end of the file if the file exists. If the file does not exist, it creates a new file for writing. Use it if you would like to add something to the end of a file\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">UTF-8\n",
    "\n",
    ">You may wonder what an encoding is and what *utf-8* is. For anyone working with texts and computers this is vital to know. Internally, a computer knows no characters whatsoever: every piece of information is represented as numbers (which in turn are represented in a binary format, as zeroes and ones). An encoding specifies which numbers represent which characters. A famous and long-standing encoding scheme is ASCII, in which for example the letter 'A' is encoded using the number 65. ASCII however only has a very limited alphabet and can not encode a lot of writing systems. A modern-day encoding supporting countless writing systems is *unicode* and *utf-8* is a kind of unicode. This the type of encoding that you will want to use for your data whenever possible. Whenever you have a choice, you should use unicode!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading an entire file in one string is not always desirable, especially not with huge files. The following example reads up until a newline everytime, and returns one line at a time. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Emma by Jane Austen 1816\n",
      "\n",
      "\n",
      "\n",
      "VOLUME I\n",
      "\n",
      "\n",
      "\n",
      "CHAPTER I\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Emma Woodhouse, handsome, clever, and rich, with a comfortable home\n",
      "\n",
      "and happy disposition, seemed to unite some of the best blessings\n",
      "\n",
      "of existence; and had lived nearly twenty-one years in the world\n",
      "\n",
      "with very little to distress or vex her.\n",
      "\n",
      "\n",
      "\n",
      "She was the youngest of the two daughters of a most affectionate,\n",
      "\n",
      "indulgent father; and had, in consequence of her sister's marriage,\n",
      "\n",
      "been mistress of his house from a very early period.  Her mother\n",
      "\n",
      "had died too long ago for her to have more than an indistinct\n",
      "\n",
      "remembrance of her caresses; and her place had been supplied\n",
      "\n",
      "by an excellent woman as governess, who had fallen little short\n",
      "\n",
      "of a mother in affection.\n"
     ]
    }
   ],
   "source": [
    "f = open('data/austen-emma-excerpt.txt','rt', encoding='utf-8') # open the file\n",
    "for line in f: # iterate over the file object\n",
    "    print(line)   # the file object yields one line at a time \n",
    "f.close() # close the file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'newline' character is probably something new to you. If you are dealing with plain text files (typically files whose name ends in the '.txt' extension), your machine uses a special character internally to signal that a new line should begin. Internally, such newlines are represented as `\"\\n\"`. Normally, this character is visualized on your screen as if the enter key were pressed. See what happens below: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than just printing, we can of course do whatever we want with this file's content. Let's count the number of lines (but note, that a line does not necessarily correspond to a sentence)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n"
     ]
    }
   ],
   "source": [
    "count = 0\n",
    "f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')\n",
    "for line in f:\n",
    "    count += 1\n",
    "f.close()\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing I would like to show you is to store the contents of a file in a list, which I find useful in some cases. Python provides the fileobject.readlines() function, which creates a list, where each element of the list is one line from the file. As you can see in the example below, this keeps the annoying trainling new line characters \"\\n\" at the end of the lines. So in the second example I read in the file as one string and split it on the newline characters \"\\n\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of lines 19 \n",
      "\n",
      "['Emma by Jane Austen 1816\\n', '\\n', 'VOLUME I\\n', '\\n', 'CHAPTER I\\n', '\\n', '\\n', 'Emma Woodhouse, handsome, clever, and rich, with a comfortable home\\n', 'and happy disposition, seemed to unite some of the best blessings\\n', 'of existence; and had lived nearly twenty-one years in the world\\n', 'with very little to distress or vex her.\\n', '\\n', 'She was the youngest of the two daughters of a most affectionate,\\n', \"indulgent father; and had, in consequence of her sister's marriage,\\n\", 'been mistress of his house from a very early period.  Her mother\\n', 'had died too long ago for her to have more than an indistinct\\n', 'remembrance of her caresses; and her place had been supplied\\n', 'by an excellent woman as governess, who had fallen little short\\n', 'of a mother in affection.']\n",
      "\n",
      "['Emma by Jane Austen 1816', '', 'VOLUME I', '', 'CHAPTER I', '', '', 'Emma Woodhouse, handsome, clever, and rich, with a comfortable home', 'and happy disposition, seemed to unite some of the best blessings', 'of existence; and had lived nearly twenty-one years in the world', 'with very little to distress or vex her.', '', 'She was the youngest of the two daughters of a most affectionate,', \"indulgent father; and had, in consequence of her sister's marriage,\", 'been mistress of his house from a very early period.  Her mother', 'had died too long ago for her to have more than an indistinct', 'remembrance of her caresses; and her place had been supplied', 'by an excellent woman as governess, who had fallen little short', 'of a mother in affection.']\n"
     ]
    }
   ],
   "source": [
    "lines = open('data/austen-emma-excerpt.txt', 'rt').readlines()\n",
    "print(\"Number of lines\", len(lines), '\\n')\n",
    "print(lines)\n",
    "print()\n",
    "print(open('data/austen-emma-excerpt.txt', 'rt').read().split('\\n'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, below I show a more \"pythonic\" way of opening a file. It is preferable to use the `with` syntax, you can read up on why this is the case, but for now just remember that it's safer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Emma by Jane Austen 1816\n",
      "\n",
      "\n",
      "\n",
      "VOLUME I\n",
      "\n",
      "\n",
      "\n",
      "CHAPTER I\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Emma Woodhouse, handsome, clever, and rich, with a comfortable home\n",
      "\n",
      "and happy disposition, seemed to unite some of the best blessings\n",
      "\n",
      "of existence; and had lived nearly twenty-one years in the world\n",
      "\n",
      "with very little to distress or vex her.\n",
      "\n",
      "\n",
      "\n",
      "She was the youngest of the two daughters of a most affectionate,\n",
      "\n",
      "indulgent father; and had, in consequence of her sister's marriage,\n",
      "\n",
      "been mistress of his house from a very early period.  Her mother\n",
      "\n",
      "had died too long ago for her to have more than an indistinct\n",
      "\n",
      "remembrance of her caresses; and her place had been supplied\n",
      "\n",
      "by an excellent woman as governess, who had fallen little short\n",
      "\n",
      "of a mother in affection.\n"
     ]
    }
   ],
   "source": [
    "with open('data/austen-emma-excerpt.txt','rt', encoding='utf-8') as txt:\n",
    "    for line in txt:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the file `data/austen-emma-excerpt.txt` and compute the average length of the lines:\n",
    "* In characters\n",
    "* In words\n",
    "\n",
    "Do not count empty lines!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36.8421052631579\n"
     ]
    }
   ],
   "source": [
    "# insert your code here\n",
    "# important: always remember to properly close your files again!\n",
    "txt = open('data/austen-emma-excerpt.txt', 'r', encoding='utf-8')\n",
    "char_average = 0\n",
    "num_lines = 0\n",
    "for line in txt:\n",
    "    char_average += len(line)\n",
    "    num_lines += 1.0\n",
    "\n",
    "print(char_average/num_lines)\n",
    "txt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Data Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal in the next week is going to be to automatically identify the sentiments in movie reviews. To this end we are going to implement simple, but effecient and effective Sentiment Analysis techniques. Sentiment analysis or opinion mining refers to the use of natural language processing/text mining techniques to identify and extract subjective information in source materials. Sentiment analysis is widely applied to reviews and social media for a variety of applications, ranging from marketing to customer service.\n",
    "\n",
    "Today we are going to learn about how to navigate directory structures with the **os** module and take the opportunity to look at the movie reviews data sets we are going to use for our Sentiment Analysis lecture.\n",
    "http://www.cse.iitb.ac.in/~pb/cs626-449-2009/prev-years-other-things-nlp/sentiment-analysis-opinion-mining-pang-lee-omsa-published.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.profile', '.bash_logout', '.bashrc', '.bash_history', '.ipython', '.ipynb_checkpoints', 'Chapter 1 - Variables.ipynb', 'Untitled Folder 1', 'data', 'Untitled0.ipynb', 'grading', 'Assig6.ipynb', 'stoplist.txt', 'Chapter 7 - File handling.ipynb', 'ngram.ipynb', 'freqdict.pkl']\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "print(os.listdir('.'))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "./data/sentiment/txt_sentoken/pos/cv000_29590.txt\n",
      "films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before . \n",
      "for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen . \n",
      "to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd . \n",
      "the book ( or \" graphic novel , \" if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes . \n",
      "in other words , don't dismiss this film because of its source . \n",
      "if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes . \n",
      "getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ? \n",
      "the ghetto in question is , of course , whitechapel in 1888 london's east end . \n",
      "it's a filthy , sooty place where the whores ( called \" unfortunates \" ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision . \n",
      "when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case . \n",
      "abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium . \n",
      "upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach . \n",
      "i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay . \n",
      "in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end . \n",
      "it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts . \n",
      "and from hell's ending had me whistling the stonecutters song from the simpsons for days ( \" who holds back the electric car/who made steve guttenberg a star ? \" ) . \n",
      "don't worry - it'll all make sense when you see it . \n",
      "now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) . \n",
      "the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic . \n",
      "oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place . \n",
      "even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent . \n",
      "ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham . \n",
      "i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad . \n",
      "the film , however , is all good . \n",
      "2 : 00 - r for strong violence/gore , sexuality , language and drug content \n",
      "\n"
     ]
    }
   ],
   "source": [
    "negatives = os.listdir('./data/sentiment/txt_sentoken/pos/')\n",
    "print(len(negatives))\n",
    "print('./data/sentiment/txt_sentoken/pos/'+negatives[0])\n",
    "file_in = open('./data/sentiment/txt_sentoken/pos/'+negatives[0], 'r', encoding=\"utf-8\")\n",
    "print(file_in.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".ipython\n",
      ".ipynb_checkpoints\n",
      "Untitled Folder 1\n",
      "data\n",
      "grading\n"
     ]
    }
   ],
   "source": [
    "for element in os.listdir('.'):\n",
    "    if os.path.isdir(element):\n",
    "        print(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "1000\n"
     ]
    }
   ],
   "source": [
    "print(len(os.listdir('./data/sentiment/txt_sentoken/neg')))\n",
    "print(len(os.listdir('./data/sentiment/txt_sentoken/pos')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['warning', ':', 'spoilers', 'are', 'included', 'in', 'this', 'review', '.', '.', '.', 'but', 'it', \"doesn't\", 'really', 'make', 'much', 'of', 'a', 'difference', '.', 'deep', 'impact', 'begins', 'the', 'official', 'summer', 'movie', 'season', ',', 'and', 'it', 'also', 'brings', 'back', 'memories', 'of', '1997', '.', 'remember', 'when', \"dante's\", 'peak', 'came', 'out', 'in', 'february', '?', 'a', 'few', 'months', 'later', ',', 'volcano', 'was', 'released', '.', 'the', 'first', 'film', 'was', 'smart', ',', 'exhilirating', ',', 'and', 'one', 'of', 'the', 'best', 'disaster', 'films', 'i', 'had', 'ever', 'seen', '.', 'the', 'latter', 'film', 'was', 'an', 'incohesive', 'mess', 'that', 'defied', 'logic', 'and', 'wasted', 'talent', '.', 'well', ',', \"it's\", 'deja', 'vu', 'all', 'over', 'again', 'as', 'two', 'disaster', 'films', 'go', 'head', 'to', 'head', 'in', 'competition', '.', 'this', 'time', ',', 'unfortunately', ',', 'the', 'first', 'comet', 'flick', 'is', 'so', 'bad', 'that', 'people', 'may', 'shy', 'away', 'from', 'armageddon', ',', 'the', 'upcoming', 'comet-disaster', 'to', 'be', 'released', 'the', 'beginning', 'of', 'july', '.', 'of', 'course', ',', 'the', 'general', 'reaction', 'of', 'the', 'audience', 'was', 'oppposed', 'to', 'mine', ',', 'and', 'so', 'i', 'am', 'in', 'the', 'minority', ',', 'as', 'i', 'was', 'when', 'i', 'stood', 'on', 'the', 'side', 'of', \"dante's\", 'peak', '.', 'but', 'while', 'watching', 'deep', 'impact', ',', 'i', 'began', 'to', 'wonder', 'how', 'anyone', 'in', 'their', 'right', 'mind', 'could', 'actually', 'like', 'this', 'film', '.', 'apparently', 'many', 'did', ',', 'and', 'it', 'utterly', 'baffles', 'me', '.', 'to', 'be', 'completely', 'honest', ',', 'i', \"haven't\", 'had', 'this', 'little', 'fun', 'watching', 'a', 'disaster', 'film', 'in', 'my', 'entire', 'life', '.', 'volcano', 'had', 'implausibilities', 'up', 'the', 'wazoo', ',', 'but', 'it', 'was', 'still', 'rather', 'fun', 'to', 'watch', '.', 'deep', 'impact', \"doesn't\", 'just', 'have', 'implausibilities', ',', 'it', 'also', 'contains', 'cheap', 'human', 'drama', ',', 'incredibly', 'horrible', 'special', 'effects', ',', 'and', 'a', 'poorly', 'constructed', 'plot', '.', 'the', 'only', 'thing', 'that', 'gives', 'it', 'a', 'half', 'star', 'above', 'my', 'bottom', 'ranking', 'is', 'a', 'slightly', 'entertaining', 'final', 'fifteen', 'minutes', 'and', 'some', 'good', 'actors', 'making', 'the', 'most', 'of', 'their', 'characters', '.', 'deep', 'impact', 'begins', 'in', 'an', 'unnamed', 'year', '(', 'the', 'year', 'varies', ';', 'advanced', 'technology', 'sets', 'it', 'in', 'the', 'future', ',', 'but', 'fire', 'in', 'the', 'sky', 'is', 'showing', 'on', 'a', 'local', 'movie', 'theater', ',', 'pushing', 'it', 'back', 'to', '1993', ')', '.', 'a', 'line', 'of', 'students', 'is', 'outside', 'at', 'night', ',', 'peering', 'through', 'telescopes', 'at', 'the', 'dark', 'sky', 'above', '.', 'among', 'these', 'are', 'leo', 'biederman', '(', 'elijah', 'wood', ')', 'and', 'sarah', 'hotchner', '(', 'leelee', 'sobieski', ')', '.', 'leo', 'unknowingly', 'discovers', 'a', 'comet', ',', 'and', 'his', 'teacher', 'sends', 'a', 'photo', 'of', 'the', 'unknown', 'object', 'to', 'an', 'astronomer', ',', 'who', 'then', 'is', 'able', 'to', 'determine', 'the', 'correct', 'path', 'of', 'this', 'distant', 'comet', 'in', 'about', 'a', 'few', 'seconds', '.', 'he', 'races', 'off', 'to', 'mail', 'the', 'information', ',', 'but', 'is', 'killed', 'in', 'a', 'reckless', 'car', 'accident', '.', 'a', 'year', 'passes', ',', 'and', 'nothing', 'is', 'heard', 'about', 'it', 'again', '.', 'we', 'are', 'then', 'introduced', 'to', 'jenny', 'lerner', '(', 't', '?', 'a', 'leoni', ')', ',', 'a', 'reporter', 'for', 'msnbc', '.', 'she', 'gets', 'handed', 'a', 'job', 'to', 'investigate', 'a', 'possible', 'cover-up', 'in', 'the', 'government', 'involving', 'senator', 'alan', 'rittenhouse', '(', 'james', 'cromwell', ')', '.', 'she', 'talks', 'to', 'a', 'woman', 'who', 'mentions', 'that', 'rittenhouse', 'was', 'having', 'an', 'affair', 'with', 'a', 'girl', 'named', 'ellie', '.', 'after', 'talking', 'with', 'rittenhouse', ',', 'and', 'unsatisfied', 'with', 'the', 'information', 'she', 'gets', ',', 'she', 'decides', 'to', 'use', 'the', 'internet', 'for', 'help', '.', 'luckily', ',', 'she', 'knows', 'exactly', 'how', 'to', 'spell', 'the', 'certain', '\"', 'ellie', '\"', 'that', 'she', 'is', 'looking', 'for', '(', 'they', 'spell', 'it', 'ele', 'in', 'the', 'film', '.', '.', '.', 'that', 'girl', 'is', 'pretty', 'darn', 'smart', 'for', 'guessing', 'how', 'it', 'was', 'spelled', ')', '.', 'before', 'she', 'can', 'use', 'the', 'information', ',', 'the', 'government', 'decides', 'to', 'push', 'her', 'car', 'off', 'the', 'road', '.', 'they', 'take', 'her', 'in', 'to', 'meet', 'the', 'president', 'of', 'the', 'united', 'states', ',', 'president', 'beck', '(', 'morgan', 'freeman', ')', '.', 'beck', 'recommends', 'that', 'jenny', 'keep', 'the', 'information', 'secret', 'for', '48', 'hours', 'so', 'they', 'can', 'confirm', 'it', 'and', 'then', 'hold', 'a', 'press', 'conference', '.', 'naturally', ',', 'she', 'wants', 'to', 'be', 'compensated', ',', 'and', 'they', 'offer', 'her', 'a', 'front', 'row', 'seat', 'and', 'the', 'chance', 'for', 'the', 'first', 'question', '.', 'and', 'so', ',', 'yada', 'yada', 'yada', ',', 'they', 'reveal', 'the', 'comet', 'to', 'the', 'public', 'and', 'their', 'plans', ':', 'send', 'a', 'massive', 'spacecraft', 'out', 'to', 'destroy', 'it', 'before', 'it', 'can', 'arrive', '.', 'they', 'announce', 'a', 'plan', 'called', '\"', 'ark', ',', '\"', 'which', 'is', 'their', 'only', 'hope', 'for', 'survival', '.', 'a', 'computer', 'will', 'select', '800', ',', '000', 'people', 'at', 'random', '.', 'these', 'people', 'are', 'the', 'ones', 'who', 'will', 'go', 'into', 'a', 'large', 'cave', 'underground', 'so', 'that', 'the', 'impact', 'of', 'the', 'comet', \"won't\", 'kill', 'off', 'the', 'entire', 'human', 'race', '.', 'after', 'two', 'years', ',', 'the', 'dust', 'will', 'settle', '(', 'actually', ',', 'it', 'would', 'take', 'much', ',', 'much', 'longer--approximately', 'ninety', 'years', ',', 'from', 'what', 'i', 'understand', ')', 'and', 'the', 'humans', 'could', 'come', 'back', 'to', 'the', 'surface', 'and', 'start', 'over', '.', 'the', 'rest', 'of', 'the', 'plot', 'is', 'your', 'standard', 'disaster', 'film', 'procedures', ',', 'but', 'there', 'is', 'one', 'subplot', 'worth', 'mentioning', '.', 'jenny', 'and', 'her', 'father', ',', 'jason', '(', 'maximilian', 'schell', ')', ',', 'have', 'a', 'very', 'touching', 'relationship', 'that', 'forms', 'out', 'of', 'the', 'impending', 'doom', '.', 'the', 'final', 'moment', 'involving', 'the', 'two', 'characters', 'is', 'heartfelt', 'and', 'emotional', '.', \"it's\", 'a', 'shame', 'that', 'nothing', 'else', 'is', 'heartfelt', '.', 'now', ',', 'of', 'course', ',', 'we', 'all', 'know', 'that', 'the', 'comet', 'does', 'impact', 'the', 'surface', '.', 'the', 'title', 'alone', 'suggests', 'it', ',', 'and', 'the', 'previews', 'actually', 'show', 'it', '!', 'by', 'doing', 'this', ',', 'absoltuely', 'no', 'tension', 'can', 'be', 'drawn', 'from', 'any', 'attempt', 'to', 'stop', 'the', 'comet', 'because', 'we', 'all', 'know', 'that', 'it', \"won't\", 'work', '.', 'director', 'mimi', 'leder', 'came', 'from', 'her', 'successful', 'tries', 'at', 'direction', 'with', 'episodes', 'of', 'the', 'hit', 'television', 'show', '\"', 'er', '.', '\"', 'her', 'major', 'film', 'debut', 'was', 'the', 'peacemaker', ',', 'a', 'pathetic', 'and', 'heartless', 'action', 'film', '.', 'well', ',', 'this', 'time', 'leder', 'outdid', 'herself', ',', 'creating', 'a', 'film', 'worse', 'than', 'that', 'one', '.', 'suggestion', 'to', 'ms', '.', 'leder', ':', 'please', ',', 'stay', 'away', 'from', 'the', 'big', 'screen', ',', 'or', 'at', 'least', 'the', 'action', 'genre', '.', 'much', 'of', 'the', 'blame', 'can', 'be', 'placed', 'on', 'leder', 'directly', ',', 'because', 'the', 'pace', 'is', 'disastrously', 'off', '.', 'throughout', 'the', 'film', 'we', 'are', 'given', 'subtitles', 'that', 'tell', 'us', 'how', 'much', 'time', 'has', 'passed', '(', 'it', 'goes', 'from', 'months', 'to', 'weeks', 'to', 'hours', ')', '.', 'it', 'literally', 'feels', 'like', 'this', 'lapsed', 'time', 'is', 'taking', 'place', 'in', \"real-time--it's\", 'that', 'boring', '.', 'of', 'course', ',', 'leder', \"isn't\", 'all', 'to', 'blame', 'for', 'it', '.', 'screenwriters', 'michael', 'tolkin', 'and', 'bruce', 'joel', 'rubin', 'have', 'crafted', 'a', 'simplistic', 'story', 'that', 'only', 'gets', 'worse', 'with', 'time', '.', 'what', 'starts', 'out', 'promising', 'soon', 'turns', 'deadly', '(', 'for', 'the', 'audience', 'anyway', ')', '.', 'chock', 'full', 'of', 'cheesy', 'one', 'liners', '(', '\"', 'you', 'know', ',', 'you', 'are', 'going', 'to', 'have', 'more', 'sex', 'than', 'anyone', 'else', 'in', 'school', '!', '\"', ')', 'and', 'stupid', 'characters', ',', 'you', 'might', 'think', 'we', 'were', 'back', 'in', 'the', '70s', 'again', '.', 'only', 'one', 'of', 'the', 'subplots', 'is', 'remotely', 'interesting', ',', 'while', 'the', 'rest', 'are', 'forgettable', 'and', 'boring', '.', 'and', 'the', 'main', 'plot', 'is', 'so', 'outrageous', 'that', 'you', \"can't\", 'figure', 'out', 'if', 'this', 'film', 'is', 'supposed', 'to', 'be', 'an', 'action', ',', 'drama', ',', 'or', 'sci-fi', '.', 'to', 'put', 'it', 'simply', ':', 'the', 'special', 'effects', 'of', 'this', 'film', 'are', 'a', 'hit', 'and', 'miss', 'situation', '.', \"that's\", 'right', ',', '80%', 'miss', ',', 'and', '20%', 'hit', '.', 'scenes', 'above', 'the', 'earth', 'are', 'well', 'done', ',', 'and', 'the', 'orbiting', 'ship', 'is', 'majestic', '.', 'but', 'the', 'comet', 'is', 'a', 'huge', 'mistake', ',', 'making', 'it', 'more', 'laughable', 'than', 'frightening', '.', 'the', 'concept', 'of', 'even', 'trying', 'to', 'land', 'on', 'a', 'comet', 'is', 'preposterous', 'enough', ',', 'but', \"that's\", 'forgiveable', '.', 'what', \"isn't\", 'forgiveable', 'is', 'actually', 'having', 'humans', 'walk', 'on', 'the', 'surface', '.', 'give', 'me', 'a', 'break', ',', 'will', 'ya', '?', 'and', 'of', 'course', ',', 'the', 'much', 'hyped', 'collision', 'of', 'comet', 'and', 'earth', '.', 'well', ',', 'it', 'is', 'far', 'from', 'spectacular', ',', 'and', 'it', 'makes', 'independence', 'day', 'look', 'brilliant', '.', 'the', 'water', 'rushing', 'towards', 'the', 'land', 'is', 'effective', ',', 'but', 'once', 'it', 'hits', 'the', 'continent', ',', 'the', 'effects', 'turn', 'ridiculous', '.', 'cgi', 'water', 'is', 'used', ',', 'and', 'it', 'looks', 'so', 'bad', 'that', 'i', 'heard', 'more', 'laughs', 'from', 'the', 'audience', 'than', 'shrieks', '.', 'in', 'fact', ',', 'i', 'may', 'even', 'recommend', 'the', 'film', 'for', 'those', 'who', 'want', 'to', 'see', 'how', 'bad', 'effects', 'can', 'actually', 'get', 'these', 'days', '.', 'just', 'when', 'we', 'think', 'visual', 'effects', \"can't\", 'be', 'improved', ',', 'along', 'comes', 'a', 'film', 'to', 'show', 'that', 'they', 'really', 'can', 'and', 'should', 'be', '.', 'one', 'thing', 'has', 'struck', 'a', 'wrong', 'note', 'with', 'me', 'concerning', 'mimi', \"leder's\", 'direction', ',', 'which', 'also', 'influences', 'the', 'actors', '.', 'leder', 'loves', 'to', 'show', 'the', \"actors'\", 'faces', 'before', 'and', 'during', 'moments', 'of', 'terror', '.', 'this', 'reminded', 'me', 'of', 'another', 'action', 'director', ',', 'renny', 'harlin', '(', 'die', 'hard', '2', ',', 'cliffhanger', ')', '.', 'but', 'harlin', 'succeeds', 'because', 'he', 'shows', 'the', 'faces', 'of', 'the', 'victims', 'in', 'realistic', 'situations', '.', 'leder', 'likes', 'to', 'flaunt', \"people's\", 'fears', 'via', 'their', 'faces', ',', 'but', 'instead', 'of', 'coming', 'off', 'as', 'sympathetic', ',', 'leder', 'seems', 'more', 'of', 'a', 'sadist', '.', 'one', 'moment', 'has', 'an', 'astronaut', 'flying', 'off', 'into', 'space', '.', 'this', 'should', 'be', 'enough', 'to', 'warrant', 'a', 'response', 'from', 'the', 'audience', ',', 'but', 'leder', 'wants', 'to', 'go', 'farther', 'and', 'shows', 'us', 'the', \"person's\", 'face', 'while', 'drifting', 'into', 'space', '.', 'this', 'is', 'one', 'of', 'the', 'cheapest', 'ways', 'to', 'ellicit', 'an', 'emotional', 'response', 'from', 'an', 'audience', ',', 'and', 'i', ',', 'for', 'one', ',', 'am', 'not', 'going', 'to', 'be', 'fooled', '.', 'on', 'another', 'note', ',', \"leder's\", 'film', 'only', 'picks', 'up', 'its', 'pace', 'during', 'the', 'last', 'fifteen', 'minutes', 'when', 'the', 'comet', 'actually', 'impacts', '.', 'the', 'pace', 'does', 'pick', 'up', ',', 'and', 'some', 'very', 'emotional', 'moments', 'are', 'shown', '.', 'then', 'again', ',', 'when', 'you', 'watch', 'people', 'cry', 'for', 'their', 'loved', 'ones', ',', \"it's\", 'obvious', 'it', 'will', 'be', 'emotional', '.', \"it's\", 'mostly', 'just', 'a', 'big', 'trick', 'to', 'rope', 'viewers', 'into', '\"', 'feeling', '\"', 'for', 'the', 'characters', ',', 'but', 'it', \"didn't\", 'work', 'for', 'me', '.', 'but', 'anyway', ',', 'the', 'actors', 'do', 'as', 'much', 'as', 'they', 'can', 'with', 'what', 'they', 'are', 'given', '.', 't', '?', 'a', 'leoni', '(', 'bad', 'boys', ',', \"tv's\", '\"', 'the', 'naked', 'truth', '\"', ')', 'is', 'the', 'best', 'of', 'the', 'film', ',', 'and', 'she', 'is', 'given', 'the', 'meatiest', 'role', '.', 'her', 'character', 'is', 'made', 'stronger', 'by', \"leoni's\", 'presence', ',', 'and', 'we', 'grow', 'to', 'care', 'for', 'her', '.', 'robert', 'duvall', 'is', 'energetic', 'and', 'fun', 'to', 'watch', ',', 'but', 'his', 'character', 'is', 'turned', 'into', 'shreds', 'by', 'the', 'plot', '.', 'elijah', 'wood', 'also', 'comes', 'off', 'rather', 'successfully', ',', 'but', 'he', 'still', \"hasn't\", 'had', 'many', 'good', 'roles', '(', 'wood', ',', 'stick', 'to', 'drama', '!', 'you', 'are', 'too', 'talented', 'for', 'this', 'stuff', ')', '.', 'vanessa', 'redgrave', 'is', 'barely', 'acknowledgeable', ',', 'and', 'her', 'performance', 'is', 'only', 'enhanced', 'by', 'her', 'strong', 'presence', 'on', 'screen', '.', 'maximilian', 'schell', 'is', 'distracting', ',', 'but', 'he', 'does', 'provide', 'some', 'nice', 'humor', '.', 'morgan', 'freeman', 'has', 'been', 'infinitely', 'better', 'than', 'this', ',', 'and', 'gives', 'one', 'of', 'his', 'most', 'shallow', 'performances', 'to', 'date', '(', 'which', 'is', 'quite', 'remarkable', 'for', 'him', ')', '.', 'leelee', 'sobieski', 'could', 'have', 'been', 'better', ',', 'but', 'i', 'think', 'she', 'just', 'suffered', 'from', 'a', 'poorly', 'written', 'character', '.', 'a', 'special', 'note', 'should', 'go', 'to', 'ron', 'eldard', 'and', 'denise', 'crosby', '.', 'eldard', 'is', 'good', 'in', 'his', 'role', ',', 'but', 'is', 'limited', 'by', 'the', 'plot', '.', 'crosby', 'is', 'special', 'to', 'me', 'personally', 'because', 'she', 'was', 'tasha', 'yar', 'from', \"tv's\", '\"', 'star', 'trek', ':', 'the', 'next', 'generation', '.', '\"', 'seeing', 'her', 'was', 'one', 'of', 'the', 'highlights', 'of', 'the', 'film', '.', 'overall', ',', 'a', 'very', 'talented', 'cast', 'virtually', 'wasted', '.', 'deep', 'impact', 'is', 'rated', 'pg-13', 'for', 'disaster', 'related', 'elements', 'and', 'brief', 'language', '.', 'this', 'is', 'one', 'of', 'the', 'worst', 'films', 'of', 'the', 'year', ',', 'and', 'if', 'it', 'is', 'any', 'omen', 'of', 'things', 'to', 'come', ',', 'this', 'summer', 'could', 'be', 'one', 'of', 'the', 'worst', 'ever', '.', 'luckily', ',', 'the', 'x-files', 'movie', 'is', 'coming', 'up', ',', 'and', 'hopefully', 'armageddon', 'will', 'be', 'more', 'successful', '.', \"it's\", 'a', 'shame', 'that', 'this', 'film', 'will', 'do', 'so', 'successfully', 'because', 'it', 'just', \"isn't\", 'worth', 'much', '.', 'costing', 'nearly', '$75', 'million', ',', 'with', 'special', 'effects', 'done', 'by', 'the', 'illustrious', 'ilm', '(', 'which', 'is', 'a', 'huge', 'shocker', ')', ',', 'and', 'with', 'a', 'score', 'composed', 'by', 'oscar-winner', 'james', 'horner', '(', 'titanic', ')', ',', 'one', 'might', 'have', 'expected', 'this', 'to', 'be', 'more', 'fun', 'to', 'witness', 'and', 'experience', '.', 'well', ',', \"it's\", 'not', '.', 'when', 'the', 'comet', 'does', 'hit', 'the', 'earth', ',', 'you', 'almost', 'wish', 'it', 'could', 'just', 'take', 'this', 'film', 'along', 'with', 'it', '.']\n"
     ]
    }
   ],
   "source": [
    "neg_data_path = './data/sentiment/txt_sentoken/neg/'\n",
    "negatives = os.listdir(neg_data_path)\n",
    "print(open(neg_data_path+negatives[100], 'r').read().split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now we mastered the art of reading files, let's move on to writing files, which follows a similar logic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open('data/testoutput.txt', 'wt', encoding='utf-8')\n",
    "f.write(\"Hello world!\")\n",
    "f.write(\"Hello world!\")\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open('data/testoutput.txt', 'a', encoding='utf-8')\n",
    "f.write(\"Hey!\")\n",
    "f.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this code block, we have created a new file called `testoutput.txt` in the `data` directory. We then wrote a single line to this file and then we closed it. Note that the `w` in `wt` is a crucial addition: if you would have left this out, Python would have opened the file in 'readonly' mode and you wouldn't have been able to write to it! The 't' in the argument, again, signifies that we will be writing to this file in plain text mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want your data to be written on multiple lines, you need to take care to explicitly encode the newlines. Instead of:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open('data/testouput.txt','wt', encoding='utf-8')\n",
    "f.write(\"Hello world on the first line!\")\n",
    "f.write(\"Hello world on the second line!\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open('data/testoutput.txt','wt', encoding='utf-8')\n",
    "f.write(\"Hello world on the first line!\\n\")\n",
    "f.write(\"Hello world on the second line!\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise your file would have `Hello world!Hello world!` in it, i.e. without the newlines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides 'read-mode' and 'write-mode' when dealing with text files, there is also the 'append-mode' in Python. Watch out: in 'write-mode', you will always *overwrite* the existing content of the file. However, if you've open a file in 'append-mode', everything you write to the file will be added at the end of the file, without deleting anything of the existing content in the file. In order to enable the append mode, you need to specify `'at'` as your second parameter when you open files ('a' for append mode; 't' for text mode)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in 'data/austen-emma.txt' and create a word count dictionary. Write out the results in a text file name \"emma_freqs.txt\" in the following format:\n",
    "\n",
    "word1,count  \n",
    "word2,count\n",
    "    \n",
    "    .\n",
    "    .\n",
    "    .\n",
    "word3, count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the file `data/austen-emma-excerpt-tokenised.txt`, and write to a file `words.txt` all words occuring in this text (without duplicates!!), alphabetically ordered, one word per line. That way, you are really creating a lexicon or word list of the text. (Tip: you should use sets in this exercise!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your output by viewing the `words.txt` file in a text editor such as Sublime Text 2. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another very common way of saving data to disk in Python is to just simply \"dump\" it in a pickle file. This section is going to walk you through this idea. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say you have read in some document and created a frequency dictionary from your text file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'word2': 50, 'word1': 210}\n"
     ]
    }
   ],
   "source": [
    "print(freq_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You would like to remember this for later use. This is where you can use the pickle module. This module let's you write out arbitrary Python objects to disk and read them back later. pickle has two main methods: The first one is dump, which dumps an object to a file object and the second one is load, which loads an object from a file object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "freq_dict = {'word1': 210, 'word2': 50}\n",
    "out = open('freqdict.pkl', 'wb')\n",
    "pickle.dump(freq_dict, out) # passing the thing that i want to right out and a file object to pickle\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'word2': 50, 'word1': 210}\n",
      "{'word2': 50, 'word1': 210, 'word3': 340}\n"
     ]
    }
   ],
   "source": [
    "a = pickle.load(open('freqdict.pkl', \"rb\"))\n",
    "print(a)\n",
    "a['word3'] = 340\n",
    "print(a)\n",
    "out = open('freqdict.pkl', 'wb')\n",
    "pickle.dump(a, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'word1': 210, 'word2': 50}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.load(open('freqdict.pkl', \"rb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we started to work with files we have to gain some insight into how to navigate the folder/directory structure. Most people use some sort of graphical user interface GUI to navigate to files such as the Finder in Mac OS or you click on the My Computer icon on Windows. Now we are going to interact with these folder structures programmatically. The workhorse of this section is going to be Python's os module. The GUI you are using translates the commands of your operating system to clicking on icons for easier use. Python's os modules is very similar to the GUI in that it provides an interface that let's you navigate between folders, create new folders, rename files etc.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started by checking out which is the current directory are we in actually right now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/akadar\n"
     ]
    }
   ],
   "source": [
    "print(os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "getcwd refers to \"get current working directory\". As you can see the name of the current directory is XXXXXXXXXXX. The directories on the left are the names higher level directories.  On Linux and Mac these are delimited by \"/\", while on Windows by \"\\\". This distinctions extremely unnecessary I know, but what can you do. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, now lets check out what files and folders do we have in this directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.profile', '.bash_logout', '.bashrc', '.bash_history', '.ipython', '.ipynb_checkpoints', 'Chapter 1 - Variables.ipynb', 'Untitled Folder 1', 'data', 'Untitled0.ipynb', 'grading', 'Assig6.ipynb', 'stoplist.txt', 'Chapter 7 - File handling.ipynb', 'ngram.ipynb', 'freqdict.pkl']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('.')) # The '.' refers to 'current directory'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see which of these are files and which of these are directories. Whe are going to use os.path.isdir, which returns True if the string in question refers to a directory otherwise it returns False. Since we can have either a directory or a file and there are no other options, we only ask if the current element is a directory and if not, we infer that it is a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".profile  \t --> is a file\n",
      ".bash_logout  \t --> is a file\n",
      ".bashrc  \t --> is a file\n",
      ".bash_history  \t --> is a file\n",
      ".ipython  \t --> is a directory\n",
      ".ipynb_checkpoints  \t --> is a directory\n",
      "Chapter 1 - Variables.ipynb  \t --> is a file\n",
      "Untitled Folder 1  \t --> is a directory\n",
      "data  \t --> is a directory\n",
      "Untitled0.ipynb  \t --> is a file\n",
      "grading  \t --> is a directory\n",
      "Assig6.ipynb  \t --> is a file\n",
      "stoplist.txt  \t --> is a file\n",
      "Chapter 7 - File handling.ipynb  \t --> is a file\n",
      "ngram.ipynb  \t --> is a file\n",
      "freqdict.pkl  \t --> is a file\n"
     ]
    }
   ],
   "source": [
    "file_list = os.listdir('.') # list current working directory\n",
    "files = [] # collect the filenames here\n",
    "directories = [] # collect the directory names here\n",
    "for element in file_list:\n",
    "    if os.path.isdir(element):\n",
    "        print(element, \" \\t --> is a directory\")\n",
    "        directories.append(element)\n",
    "    else:\n",
    "        print(element, \" \\t --> is a file\")\n",
    "        files.append(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The os module also allows us to change to different directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directories: ['.ipython', '.ipynb_checkpoints', 'Untitled Folder 1', 'data', 'grading']\n"
     ]
    }
   ],
   "source": [
    "print(\"Directories:\", directories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/akadar/data\n",
      "['sentiment', 'austen-emma.txt', 'austen-emma-excerpt.txt', 'testoutput.txt', 'testouput.txt']\n",
      "/home/akadar\n"
     ]
    }
   ],
   "source": [
    "os.chdir('data') # descending to the folder \"data\"\n",
    "print(os.getcwd()) # where are we now?\n",
    "print(os.listdir('.')) # what do we have here?\n",
    "os.chdir('..') # going back up\n",
    "print(os.getcwd()) # are we back?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code snippet:\n",
    " + goes to the data directory\n",
    " + creates a new directory inside it \"test\"\n",
    " + creates a new file \"test.txt\"\n",
    " + removes the file \"test.txt\"\n",
    " + removes the directory \"test\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We are here: /home/akadar\n",
      "We are here: /home/akadar/data\n",
      "['sentiment', 'austen-emma.txt', 'austen-emma-excerpt.txt', 'testoutput.txt', 'testouput.txt']\n",
      "['sentiment', 'austen-emma.txt', 'austen-emma-excerpt.txt', 'testoutput.txt', 'testouput.txt', 'test']\n",
      "[]\n",
      "['test.txt']\n",
      "Testing\n",
      "We are here /home/akadar/data\n",
      "['sentiment', 'austen-emma.txt', 'austen-emma-excerpt.txt', 'testoutput.txt', 'testouput.txt']\n",
      "And we're back to: /home/akadar\n"
     ]
    }
   ],
   "source": [
    "print(\"We are here:\", os.getcwd())\n",
    "os.chdir('data') # chdir --> change directory\n",
    "print(\"We are here:\", os.getcwd())\n",
    "print(os.listdir('.'))\n",
    "os.mkdir('test') # mkdir --> make directory\n",
    "print(os.listdir('.'))\n",
    "os.chdir('test') # chdir --> change directory\n",
    "print(os.listdir('.'))\n",
    "open(\"test.txt\", 'wt').write('Testing')\n",
    "print(os.listdir('.'))\n",
    "print(open(r\"test.txt\").read())\n",
    "os.remove(\"test.txt\")\n",
    "os.chdir('..')\n",
    "print(\"We are here\", os.getcwd())\n",
    "os.rmdir('test')\n",
    "print(os.listdir('.'))\n",
    "os.chdir('..')\n",
    "print(\"And we're back to:\", os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Exercise 1**: Go to Project Gutenberg (http://www.gutenberg.org) and download your favorite out- of-copyright book in plain text format, then upload it to the Jupyter server. Make a frequency dictionary of the words in the novel. Sort the words in the dictionary by frequency and write it to a text file called `frequencies.txt`. Make sure your program ignores capitalization as well as punctuation (hint: check out `string.punctuation` online!). Search the web in order to find out how you can sort a dictionary -- this is not easy, because you might have to import another module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Exercise 2**: Rewrite the novel in the previous exercise, by replacing the name of the principal character in the novel by your own name. (Use the `replace()` function for this.) Write the new version of novel to a file called `starring_me.txt`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Exercise 3:** A *hapax legomenon* (often abbreviated to hapax) is a word which occurs only once in either the written record of a language, the works of an author, or in a single text. Define a function that given the file name of a text will return all its hapaxes. Make sure your program ignores capitalization as well as punctuation (hint: check out `string.punctuation` online!). Try out the function on your Gutenberg book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Exercise 4:** Write a program that given a text file will create a new text file in which all the lines from the original file are numbered from 1 to n (where n is the number of lines in the file)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}