beautiful_soup_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping with Beautiful Soup Demo\n",
    "\n",
    "This demo will walk through an example of scraping data from a website programatically using the Beautiful Soup package in Python. Complete code is available on GitHub (see Resources).\n",
    "\n",
    "\n",
    "## Why scrape data from the web?\n",
    "\n",
    "+ No API available\n",
    "+ Build a dataset for analysis\n",
    "+ Keep track of data over time\n",
    "+ Use data in another program\n",
    "+ The Internet is a Big Data Lake\n",
    "\n",
    "## DEMO: Magic The Gathering (MTG)\n",
    "\n",
    "MTG is a card-dueling game. Players build a deck prior to game play with certain rules of deck construction based on the format of play. For this use-case, I was interested in the Commander format of game play. Commander decks consist of 100 cards: 1 commander, 99 spells & mana (currency to cast spells). \n",
    "\n",
    "There are around 20,000 unique MTG cards in existence. They are not all 'legal', but if you assume that 10,000 are legal, you have an astronomical number of combinations of decks possible in nearly every format of game play. Certain deck combinations float to the top and become widely replicated and played in the tournament circuits. These comprise the 'metagame'.\n",
    "\n",
    "I was interested in looking at the deck characteristics of the current 'metagame'. There are several websites that allow users to save deck configurations and provide stats on the decks. After exhaustive searching, I found that there were no websites that exposed API endpoints to pull this data. \n",
    "\n",
    "Example Sites:\n",
    "+ [EDH Rec](https://edhrec.com/)\n",
    "+ [MTG Top 8](https://mtgtop8.com/)\n",
    "+ [MTG Goldfish](https://www.mtggoldfish.com/metagame/commander#paper)\n",
    "\n",
    "**Enter: Beautiful Soup and Web Scraping!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports\n",
    "\n",
    "+ URLlib: (Standard Library)\n",
    "    + urllib is a package that collects several modules for working with URLs:\n",
    "        + **urllib.request** for opening and reading URLs\n",
    "        + **urllib.error** containing the exceptions raised by urllib.request\n",
    "        + urllib.parse for parsing URLs\n",
    "        + urllib.robotparser for parsing robots.txt files\n",
    "+ JSON: (Standard Library)\n",
    "    + JSON encoder and decoder\n",
    "+ RE: Regular Expressions (Standard Library)\n",
    "    + Regular Expression Operations\n",
    "+ BS4: Beautiful Soup\n",
    "    + \"Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "URLError",
     "evalue": "<urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mdo_open\u001b[1;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[0;32m   1318\u001b[0m                 h.request(req.get_method(), req.selector, req.data, headers,\n\u001b[1;32m-> 1319\u001b[1;33m                           encode_chunked=req.has_header('Transfer-encoding'))\n\u001b[0m\u001b[0;32m   1320\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;31m# timeout error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1251\u001b[0m         \u001b[1;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1252\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1253\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_request\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1297\u001b[0m             \u001b[0mbody\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'body'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1298\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1299\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mendheaders\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1246\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1247\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1248\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_output\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1025\u001b[0m         \u001b[1;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1026\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1027\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, data)\u001b[0m\n\u001b[0;32m    965\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 966\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    967\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1413\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1414\u001b[1;33m             \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1415\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    937\u001b[0m         self.sock = self._create_connection(\n\u001b[1;32m--> 938\u001b[1;33m             (self.host,self.port), self.timeout, self.source_address)\n\u001b[0m\u001b[0;32m    939\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msetsockopt\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msocket\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mIPPROTO_TCP\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msocket\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTCP_NODELAY\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address)\u001b[0m\n\u001b[0;32m    727\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 728\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    729\u001b[0m         \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address)\u001b[0m\n\u001b[0;32m    715\u001b[0m                 \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msource_address\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 716\u001b[1;33m             \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msa\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    717\u001b[0m             \u001b[1;31m# Break explicitly a reference cycle\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTimeoutError\u001b[0m: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mURLError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-6-1b6a17dfa302>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmtgsdk\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mCard\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mcards\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mCard\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m#https://mtgjson.com/\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\site-packages\\mtgsdk\\card.py\u001b[0m in \u001b[0;36mall\u001b[1;34m()\u001b[0m\n\u001b[0;32m     66\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     67\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 68\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mQueryBuilder\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mCard\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\site-packages\\mtgsdk\\querybuilder.py\u001b[0m in \u001b[0;36mall\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     66\u001b[0m             \u001b[0mlist\u001b[0m \u001b[0mof\u001b[0m \u001b[0mobject\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mList\u001b[0m \u001b[0mof\u001b[0m \u001b[0mresource\u001b[0m \u001b[0mobjects\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     67\u001b[0m         \"\"\"\n\u001b[1;32m---> 68\u001b[1;33m         \u001b[0mlist\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     69\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     70\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\site-packages\\mtgsdk\\querybuilder.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m     66\u001b[0m             \u001b[0mlist\u001b[0m \u001b[0mof\u001b[0m \u001b[0mobject\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mList\u001b[0m \u001b[0mof\u001b[0m \u001b[0mresource\u001b[0m \u001b[0mobjects\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     67\u001b[0m         \"\"\"\n\u001b[1;32m---> 68\u001b[1;33m         \u001b[0mlist\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0miter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     69\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     70\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\site-packages\\mtgsdk\\querybuilder.py\u001b[0m in \u001b[0;36miter\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     86\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     87\u001b[0m         \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 88\u001b[1;33m             \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mRestClient\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mparams\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mRESOURCE\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     89\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     90\u001b[0m                 \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\site-packages\\mtgsdk\\restclient.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(url, params)\u001b[0m\n\u001b[0;32m     33\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     34\u001b[0m             \u001b[0mreq\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mRequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrequest_url\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m{\u001b[0m\u001b[1;34m'User-Agent'\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;34m'Mozilla/5.0'\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 35\u001b[1;33m             \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murlopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"utf-8\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     36\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     37\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(url, data, timeout, cafile, capath, cadefault, context)\u001b[0m\n\u001b[0;32m    220\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    221\u001b[0m         \u001b[0mopener\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_opener\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 222\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mopener\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    223\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    224\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0minstall_opener\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mopener\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mopen\u001b[1;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[0;32m    523\u001b[0m             \u001b[0mreq\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    524\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 525\u001b[1;33m         \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    526\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    527\u001b[0m         \u001b[1;31m# post-process response\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36m_open\u001b[1;34m(self, req, data)\u001b[0m\n\u001b[0;32m    541\u001b[0m         \u001b[0mprotocol\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mreq\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    542\u001b[0m         result = self._call_chain(self.handle_open, protocol, protocol +\n\u001b[1;32m--> 543\u001b[1;33m                                   '_open', req)\n\u001b[0m\u001b[0;32m    544\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    545\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[1;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[0;32m    501\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    502\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 503\u001b[1;33m             \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    504\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    505\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mhttps_open\u001b[1;34m(self, req)\u001b[0m\n\u001b[0;32m   1360\u001b[0m         \u001b[1;32mdef\u001b[0m \u001b[0mhttps_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1361\u001b[0m             return self.do_open(http.client.HTTPSConnection, req,\n\u001b[1;32m-> 1362\u001b[1;33m                 context=self._context, check_hostname=self._check_hostname)\n\u001b[0m\u001b[0;32m   1363\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1364\u001b[0m         \u001b[0mhttps_request\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mAbstractHTTPHandler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdo_request_\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mdo_open\u001b[1;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[0;32m   1319\u001b[0m                           encode_chunked=req.has_header('Transfer-encoding'))\n\u001b[0;32m   1320\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;31m# timeout error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1321\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0mURLError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1322\u001b[0m             \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mh\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1323\u001b[0m         \u001b[1;32mexcept\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mURLError\u001b[0m: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>"
     ]
    }
   ],
   "source": [
    "from urllib.request import urlopen\n",
    "from urllib.error import HTTPError\n",
    "from urllib.error import URLError\n",
    "import json\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "###THen get deck info using MTG SDK API\n",
    "\n",
    "from mtgsdk import Card\n",
    "cards = Card.all() #https://mtgjson.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine the structure of the site\n",
    "\n",
    "+ Robots.txt\n",
    "    + Example: https://www.mtggoldfish.com/robots.txt\n",
    "    + More Info: http://www.robotstxt.org/robotstxt.html\n",
    "    + BLUF: Respect the robots.txt file...it's just the right thing to do. \n",
    "+ Look at the site and figure out exactly what data you want\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full#paper\n",
    "+ What is the structure of the URLs?\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full?page=2#paper\n",
    "    + notice the page number, how many pages?\n",
    "+ Is there a way to programmatically loop through URLs?\n",
    "+ F12 (developer mode) to 'inspect elements'\n",
    "\n",
    "## Scraping Plan: Two loops\n",
    "\n",
    "+ Loop through the ~12 pages that have **urls** for each deck\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full?page=1#paper\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full?page=2#paper\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full?page=3#paper\n",
    "    + ...\n",
    "    + https://www.mtggoldfish.com/metagame/commander/full?page=12#paper\n",
    "+ Loop through deck URLs on each of the above pages, eg.\n",
    "    + zur-the-enchanter\n",
    "    + kozilek-the-great-distortion\n",
    "    + sen-triplets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "URLError",
     "evalue": "<urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mdo_open\u001b[1;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[0;32m   1318\u001b[0m                 h.request(req.get_method(), req.selector, req.data, headers,\n\u001b[1;32m-> 1319\u001b[1;33m                           encode_chunked=req.has_header('Transfer-encoding'))\n\u001b[0m\u001b[0;32m   1320\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;31m# timeout error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1251\u001b[0m         \u001b[1;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1252\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1253\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_request\u001b[1;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[0;32m   1297\u001b[0m             \u001b[0mbody\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'body'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1298\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1299\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mendheaders\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1246\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1247\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1248\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_send_output\u001b[1;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[0;32m   1025\u001b[0m         \u001b[1;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1026\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1027\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, data)\u001b[0m\n\u001b[0;32m    965\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 966\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    967\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1413\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1414\u001b[1;33m             \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1415\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\http\\client.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    937\u001b[0m         self.sock = self._create_connection(\n\u001b[1;32m--> 938\u001b[1;33m             (self.host,self.port), self.timeout, self.source_address)\n\u001b[0m\u001b[0;32m    939\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msetsockopt\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msocket\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mIPPROTO_TCP\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msocket\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTCP_NODELAY\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address)\u001b[0m\n\u001b[0;32m    727\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 728\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    729\u001b[0m         \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[1;34m(address, timeout, source_address)\u001b[0m\n\u001b[0;32m    715\u001b[0m                 \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msource_address\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 716\u001b[1;33m             \u001b[0msock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msa\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    717\u001b[0m             \u001b[1;31m# Break explicitly a reference cycle\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTimeoutError\u001b[0m: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mURLError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-bd5311d3a6d3>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mmyurl\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m'https://www.mtggoldfish.com/metagame/commander/full#paper'\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;31m#pull the html text of myurl into python\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mhtml\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0murlopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmyurl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      8\u001b[0m \u001b[1;31m#create beautiful soup object from the html\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[0mbs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mBeautifulSoup\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhtml\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'html.parser'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(url, data, timeout, cafile, capath, cadefault, context)\u001b[0m\n\u001b[0;32m    220\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    221\u001b[0m         \u001b[0mopener\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_opener\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 222\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mopener\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    223\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    224\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0minstall_opener\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mopener\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mopen\u001b[1;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[0;32m    523\u001b[0m             \u001b[0mreq\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    524\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 525\u001b[1;33m         \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    526\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    527\u001b[0m         \u001b[1;31m# post-process response\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36m_open\u001b[1;34m(self, req, data)\u001b[0m\n\u001b[0;32m    541\u001b[0m         \u001b[0mprotocol\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mreq\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    542\u001b[0m         result = self._call_chain(self.handle_open, protocol, protocol +\n\u001b[1;32m--> 543\u001b[1;33m                                   '_open', req)\n\u001b[0m\u001b[0;32m    544\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    545\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[1;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[0;32m    501\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    502\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 503\u001b[1;33m             \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    504\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    505\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mhttps_open\u001b[1;34m(self, req)\u001b[0m\n\u001b[0;32m   1360\u001b[0m         \u001b[1;32mdef\u001b[0m \u001b[0mhttps_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1361\u001b[0m             return self.do_open(http.client.HTTPSConnection, req,\n\u001b[1;32m-> 1362\u001b[1;33m                 context=self._context, check_hostname=self._check_hostname)\n\u001b[0m\u001b[0;32m   1363\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1364\u001b[0m         \u001b[0mhttps_request\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mAbstractHTTPHandler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdo_request_\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\anaconda3\\envs\\web-scraper\\lib\\urllib\\request.py\u001b[0m in \u001b[0;36mdo_open\u001b[1;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[0;32m   1319\u001b[0m                           encode_chunked=req.has_header('Transfer-encoding'))\n\u001b[0;32m   1320\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;31m# timeout error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1321\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0mURLError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1322\u001b[0m             \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mh\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1323\u001b[0m         \u001b[1;32mexcept\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mURLError\u001b[0m: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>"
     ]
    }
   ],
   "source": [
    "# Target Domain\n",
    "DOMAIN = 'https://www.mtggoldfish.com'\n",
    "\n",
    "# Use the following URL to make a list of links to all the commander decks in current metagame\n",
    "myurl = 'https://www.mtggoldfish.com/metagame/commander/full?page=1#paper'\n",
    "#pull the html text of myurl into python \n",
    "html = urlopen(myurl)\n",
    "#create beautiful soup object from the html\n",
    "bs = BeautifulSoup(html, 'html.parser')\n",
    "#close the connection to the page\n",
    "html.close()\n",
    "#place to store scraped data\n",
    "decks = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[<title>Magic: the Gathering Commander Decks and Metagame Breakdown (All</title>,\n <meta content=\"Popular Commander Magic: the Gathering decks with prices from tournament results.\" name=\"description\"/>,\n <meta content=\"decks, decklists, tournament results, archetypes, metagame, magic, mtg, magic the gathering, magic the gathering online, mtgo, daily events\" name=\"keywords\"/>,\n <meta content=\"Magic: the Gathering Commander Decks and Metagame Breakdown (All Decks)\" property=\"og:title\"/>,\n <meta content=\"website\" property=\"og:type\"/>,\n <meta content=\"https://www.mtggoldfish.com/metagame/commander/full\" property=\"og:url\"/>,\n <meta content=\"Popular Commander Magic: the Gathering decks with prices from tournament results.\" property=\"og:description\"/>,\n <meta content=\"summary\" name=\"twitter:card\"/>,\n <meta content=\"@mtggoldfish\" name=\"twitter:site\"/>,\n <link href=\"https://www.mtggoldfish.com/feed\" rel=\"alternate\" title=\"ATOM\" type=\"application/atom+xml\"/>,\n <link href=\"https://www.mtggoldfish.com/feed.rss\" rel=\"alternate\" title=\"RSS\" type=\"application/rss+xml\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-32-22599602a047942143b34b034de85b0968d377caf56ff4a12c06146a4c13b989.png\" rel=\"apple-touch-icon\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-72-f4155e01f71cfcaaa34af38dea8296e912c56eb5474a0ecc60064b4601ffebd6.png\" rel=\"apple-touch-icon\" sizes=\"72x72\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-114-1d9d14e0a25de8fa2df41aed9ef917d10c27f55d07a2ca60c34331c56c6b8030.png\" rel=\"apple-touch-icon\" sizes=\"114x114\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-144-fbf44d4bca87904f865574e1cc5ae42ac3789e284b765bc16eb920373b14b906.png\" rel=\"apple-touch-icon\" sizes=\"144x144\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-64-f5b29fefb4ab86245883253afa5faffc2ffae42f26cd0291e620c074f354b3a6.png\" rel=\"apple-touch-icon-precomposed\"/>,\n <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-32-22599602a047942143b34b034de85b0968d377caf56ff4a12c06146a4c13b989.png\" rel=\"shortcut icon\" type=\"image/png\"/>,\n <link href=\"https://fonts.googleapis.com/css?family=Montserrat:400,700,900|Open+Sans\" rel=\"stylesheet\"/>,\n <meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>,\n <meta content=\"width=device-width, initial-scale=1\" name=\"viewport\"/>,\n <meta content=\"authenticity_token\" name=\"csrf-param\">\n <meta content=\"2iAcE9kT4TzLRzqBlHL5Nr7haAF9MxSo5Fmt9YqalkbfL8cpDQVb+4WUhpMZXOSD46jAYO7GcD4KNBX6xnDcUg==\" name=\"csrf-token\">\n <link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n <link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n <script>(function() {\n var _fbq = window._fbq || (window._fbq = []);\n if (!_fbq.loaded) {\n var fbds = document.createElement('script');\n fbds.async = true;\n fbds.src = '//connect.facebook.net/en_US/fbds.js';\n var s = document.getElementsByTagName('script')[0];\n s.parentNode.insertBefore(fbds, s);\n _fbq.loaded = true;\n }\n _fbq.push(['addPixelId', '928681210525176']);\n })();\n window._fbq = window._fbq || [];\n window._fbq.push(['track', 'PixelInitialized', {}]);\n </script>\n <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n <script type=\"text/javascript\">\n   window[\"nitroAds\"] = window[\"nitroAds\"] || {\n     createAd: function() {\n       window.nitroAds.queue.push([\"createAd\", arguments]);\n     },\n     queue: []\n   };\n </script>\n <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n </link></link></meta></meta>,\n <meta content=\"2iAcE9kT4TzLRzqBlHL5Nr7haAF9MxSo5Fmt9YqalkbfL8cpDQVb+4WUhpMZXOSD46jAYO7GcD4KNBX6xnDcUg==\" name=\"csrf-token\">\n <link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n <link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n <script>(function() {\n var _fbq = window._fbq || (window._fbq = []);\n if (!_fbq.loaded) {\n var fbds = document.createElement('script');\n fbds.async = true;\n fbds.src = '//connect.facebook.net/en_US/fbds.js';\n var s = document.getElementsByTagName('script')[0];\n s.parentNode.insertBefore(fbds, s);\n _fbq.loaded = true;\n }\n _fbq.push(['addPixelId', '928681210525176']);\n })();\n window._fbq = window._fbq || [];\n window._fbq.push(['track', 'PixelInitialized', {}]);\n </script>\n <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n <script type=\"text/javascript\">\n   window[\"nitroAds\"] = window[\"nitroAds\"] || {\n     createAd: function() {\n       window.nitroAds.queue.push([\"createAd\", arguments]);\n     },\n     queue: []\n   };\n </script>\n <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n </link></link></meta>,\n <link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n <link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n <script>(function() {\n var _fbq = window._fbq || (window._fbq = []);\n if (!_fbq.loaded) {\n var fbds = document.createElement('script');\n fbds.async = true;\n fbds.src = '//connect.facebook.net/en_US/fbds.js';\n var s = document.getElementsByTagName('script')[0];\n s.parentNode.insertBefore(fbds, s);\n _fbq.loaded = true;\n }\n _fbq.push(['addPixelId', '928681210525176']);\n })();\n window._fbq = window._fbq || [];\n window._fbq.push(['track', 'PixelInitialized', {}]);\n </script>\n <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n <script type=\"text/javascript\">\n   window[\"nitroAds\"] = window[\"nitroAds\"] || {\n     createAd: function() {\n       window.nitroAds.queue.push([\"createAd\", arguments]);\n     },\n     queue: []\n   };\n </script>\n <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n </link></link>,\n <link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n <script>(function() {\n var _fbq = window._fbq || (window._fbq = []);\n if (!_fbq.loaded) {\n var fbds = document.createElement('script');\n fbds.async = true;\n fbds.src = '//connect.facebook.net/en_US/fbds.js';\n var s = document.getElementsByTagName('script')[0];\n s.parentNode.insertBefore(fbds, s);\n _fbq.loaded = true;\n }\n _fbq.push(['addPixelId', '928681210525176']);\n })();\n window._fbq = window._fbq || [];\n window._fbq.push(['track', 'PixelInitialized', {}]);\n </script>\n <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n <script type=\"text/javascript\">\n   window[\"nitroAds\"] = window[\"nitroAds\"] || {\n     createAd: function() {\n       window.nitroAds.queue.push([\"createAd\", arguments]);\n     },\n     queue: []\n   };\n </script>\n <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n </link>,\n <script>(function() {\n var _fbq = window._fbq || (window._fbq = []);\n if (!_fbq.loaded) {\n var fbds = document.createElement('script');\n fbds.async = true;\n fbds.src = '//connect.facebook.net/en_US/fbds.js';\n var s = document.getElementsByTagName('script')[0];\n s.parentNode.insertBefore(fbds, s);\n _fbq.loaded = true;\n }\n _fbq.push(['addPixelId', '928681210525176']);\n })();\n window._fbq = window._fbq || [];\n window._fbq.push(['track', 'PixelInitialized', {}]);\n </script>,\n <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>,\n <img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/>,\n <script type=\"text/javascript\">\n   window[\"nitroAds\"] = window[\"nitroAds\"] || {\n     createAd: function() {\n       window.nitroAds.queue.push([\"createAd\", arguments]);\n     },\n     queue: []\n   };\n </script>,\n <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>]"
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "source": [
    "bs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the deck URL in the HTML\n",
    "+ 'Inspect Element'\n",
    "+ Use Beautiful Soup findAll() method to search the bs object for that html Element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "<title>Magic: the Gathering Commander Decks and Metagame Breakdown (All</title>"
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "source": [
    "bs.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "'Magic: the Gathering Commander Decks and Metagame Breakdown (All'"
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "source": [
    "bs.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[<meta content=\"Popular Commander Magic: the Gathering decks with prices from tournament results.\" name=\"description\"/>]"
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "bs.findAll('meta',{'name':\"description\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "<div class=\"card-image-tile\" style=\"background-image: url('https://cdn1.mtggoldfish.com/images/gf/Sisay%252C%2BWeatherlight%2BCaptain%2B%255BMH1%255D.jpg');\"></div>"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "bs.find('div',{'class':\"card-image-tile\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\"background-image: url('https://cdn1.mtggoldfish.com/images/gf/Sisay%252C%2BWeatherlight%2BCaptain%2B%255BMH1%255D.jpg');\""
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "source": [
    "#chain together\n",
    "bs.find('div',{'class':\"card-image-tile\"}).get('style')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "https://cdn1.mtggoldfish.com/images/gf/Sisay%252C%2BWeatherlight%2BCaptain%2B%255BMH1%255D.jpg\n"
    }
   ],
   "source": [
    "image_style = bs.find('div',{'class':\"card-image-tile\"}).get('style')\n",
    "image = image_style.split(\"('\", 1)[1].split(\"')\")[0]\n",
    "print(image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": "<img src=\"https://cdn1.mtggoldfish.com/images/gf/Sisay%252C%2BWeatherlight%2BCaptain%2B%255BMH1%255D.jpg\"/>",
      "text/plain": "<IPython.core.display.Image object>"
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "source": [
    "#Display image\n",
    "from IPython.display import Image\n",
    "from IPython.core.display import HTML \n",
    "Image(url= image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE: We can use a Magic the Gathering API to get some of the card data. It's useful to evaluate if there is a request limit when scraping a site. You might need to split the data collection tasks between multiple data sources. In this use case, I am pulling one page at a time and parsing the html locally. This also limits the number of requests being made to the site.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[<span class=\"deck-price-paper\">\n <!-- items [ [url, label], [url, label] ] -->\n <div class=\"subNav-menu-mobile\">\n <label class=\"sr-only\" for=\"menu-format-selection-6021cbeb-0a31-4917-8557-0625dbf8394a\">Format Selection</label>\n <select class=\"form-control subNav-menu-select\" id=\"menu-format-selection-6021cbeb-0a31-4917-8557-0625dbf8394a\">\n <option selected=\"selected\" value=\"\">Select a Format ...</option>\n <option value=\"/metagame/standard/full#paper\">Standard</option>\n <option value=\"/metagame/modern/full#paper\">Modern</option>\n <option value=\"/metagame/pioneer/full#paper\">Pioneer</option>\n <option value=\"/metagame/pauper/full#paper\">Pauper</option>\n <option value=\"/metagame/legacy/full#paper\">Legacy</option>\n <option value=\"/metagame/vintage/full#paper\">Vintage</option>\n <option value=\"/metagame/historic/full#paper\">Historic</option>\n <option value=\"/metagame/penny_dreadful/full#paper\">Penny Dreadful</option>\n <option value=\"/metagame/commander_1v1/full#paper\">Commander 1v1</option>\n <option value=\"/metagame/commander/full#paper\">Commander</option>\n <option value=\"/metagame/brawl/full#paper\">Brawl</option>\n </select>\n </div>\n <div class=\"subNav-menu-desktop\">\n <a href=\"/metagame/standard/full#paper\">Standard</a>\n |\n <a href=\"/metagame/modern/full#paper\">Modern</a>\n |\n <a href=\"/metagame/pioneer/full#paper\">Pioneer</a>\n |\n <a href=\"/metagame/pauper/full#paper\">Pauper</a>\n |\n <a href=\"/metagame/legacy/full#paper\">Legacy</a>\n |\n <a href=\"/metagame/vintage/full#paper\">Vintage</a>\n |\n <a href=\"/metagame/historic/full#paper\">Historic</a>\n |\n <a href=\"/metagame/penny_dreadful/full#paper\">Penny Dreadful</a>\n |\n <a href=\"/metagame/commander_1v1/full#paper\">Commander 1v1</a>\n |\n <a href=\"/metagame/commander/full#paper\">Commander</a>\n |\n <a href=\"/metagame/brawl/full#paper\">Brawl</a>\n </div>\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/sisay-weatherlight-captain#paper\">Sisay, Weatherlight Captain</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,046\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/golos-tireless-pilgrim#paper\">Golos, Tireless Pilgrim</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,089\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kenrith-the-returned-king#paper\">Kenrith, the Returned King</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,009\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/commander-rin-and-seri-inseparable#paper\">Rin and Seri, Inseparable</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 469\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/breya-etherium-shaper#paper\">Breya, Etherium Shaper</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,173\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/chulane-teller-of-tales#paper\">Chulane, Teller of Tales</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 772\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/alela-artful-provocateur#paper\">Alela, Artful Provocateur</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 629\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/atraxa-praetors-voice#paper\">Atraxa, Praetors' Voice</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 911\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/muldrotha-the-gravetide#paper\">Muldrotha, the Gravetide</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 644\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/yuriko-the-tiger-s-shadow#paper\">Yuriko, the Tiger's Shadow</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 613\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/korvold-fae-cursed-king#paper\">Korvold, Fae-Cursed King</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 665\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/krenko-mob-boss#paper\">Krenko, Mob Boss</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 256\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/gishath-sun-s-avatar#paper\">Gishath, Sun's Avatar</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 299\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/gavi-nest-warden#paper\">Gavi, Nest Warden</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 465\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kalamax-the-stormsire#paper\">Kalamax, the Stormsire</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 620\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kess-dissident-mage#paper\">Kess, Dissident Mage</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,403\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/the-ur-dragon#paper\">The Ur-Dragon</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 648\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/arcades-the-strategist#paper\">Arcades, the Strategist</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 288\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/nethroi-apex-of-death#paper\">Nethroi, Apex of Death</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 602\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/niv-mizzet-parun#paper\">Niv-Mizzet, Parun</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 400\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/meren-of-clan-nel-toth#paper\">Meren of Clan Nel Toth</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 511\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/syr-gwyn-hero-of-ashvale#paper\">Syr Gwyn, Hero of Ashvale</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 450\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kaalia-of-the-vast#paper\">Kaalia of the Vast</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 596\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kinnan-bonder-prodigy#paper\">Kinnan, Bonder Prodigy</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,534\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/jhoira-weatherlight-captain#paper\">Jhoira, Weatherlight Captain</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 649\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/queen-marchesa#paper\">Queen Marchesa</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 609\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/tatyova-benthic-druid#paper\">Tatyova, Benthic Druid</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 571\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/urza-lord-high-artificer#paper\">Urza, Lord High Artificer</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,718\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/najeela-the-blade-blossom#paper\">Najeela, the Blade-Blossom</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 1,327\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/lord-windgrace#paper\">Lord Windgrace</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 761\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/zaxara-the-exemplary#paper\">Zaxara, the Exemplary</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 860\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/omnath-locus-of-the-roil#paper\">Omnath, Locus of the Roil</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 514\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/damia-sage-of-stone#paper\">Damia, Sage of Stone</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 749\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/feather-the-redeemed#paper\">Feather, the Redeemed</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 221\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/animar-soul-of-elements#paper\">Animar, Soul of Elements</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 718\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/commander-radha-heart-of-keld#paper\">Radha, Heart of Keld</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 522\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/commander-tinybones-trinket-thief#paper\">Tinybones, Trinket Thief</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 807\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/aminatou-the-fateshifter#paper\">Aminatou, the Fateshifter</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 688\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/elsha-of-the-infinite#paper\">Elsha of the Infinite</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 935\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/alesha-who-smiles-at-death#paper\">Alesha, Who Smiles at Death</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 403\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/yarok-the-desecrated#paper\">Yarok, the Desecrated</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 698\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/edgar-markov#paper\">Edgar Markov</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 413\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/oloro-ageless-ascetic#paper\">Oloro, Ageless Ascetic</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 761\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/xyris-the-writhing-storm#paper\">Xyris, the Writhing Storm</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 953\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/commander-inniaz-the-gale-force#paper\">Inniaz, the Gale Force</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 373\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/teysa-karlov#paper\">Teysa Karlov</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 331\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kynaios-and-tiro-of-meletis#paper\">Kynaios and Tiro of Meletis</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 549\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/k-rrik-son-of-yawgmoth#paper\">K'rrik, Son of Yawgmoth</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 590\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/the-scarab-god#paper\">The Scarab God</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 506\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/silvar-devourer-of-the-free-trynn-champion-of-freedom#paper\">Silvar, Devourer of the Free // Trynn, Champion of Freedom</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 376\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/zedruu-the-greathearted#paper\">Zedruu the Greathearted</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 378\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/atla-palani-nest-tender#paper\">Atla Palani, Nest Tender</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 597\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/yidris-maelstrom-wielder#paper\">Yidris, Maelstrom Wielder</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 940\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/shirei-shizo-s-caretaker#paper\">Shirei, Shizo's Caretaker</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 211\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/hope-of-ghirapur#paper\">Hope of Ghirapur</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 427\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/anje-falkenrath#paper\">Anje Falkenrath</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 454\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/zur-the-enchanter#paper\">Zur the Enchanter</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 967\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/riku-of-two-reflections#paper\">Riku of Two Reflections</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 612\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/winota-joiner-of-forces#paper\">Winota, Joiner of Forces</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 458\n </span>,\n <span class=\"deck-price-paper\">\n <a href=\"/archetype/kozilek-the-great-distortion#paper\">Kozilek, the Great Distortion</a>\n </span>,\n <span class=\"deck-price-paper\">\n $ 555\n </span>]"
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "source": [
    "bs.findAll('span', {'class': \"deck-price-paper\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "deck_urls = []\n",
    "try:\n",
    "    deck_url = bs.findAll('span', {'class': \"deck-price-paper\"})\n",
    "    for i in deck_url:\n",
    "        for link in i.findAll('a'):\n",
    "            deck_urls.append(link.attrs['href'])\n",
    "except AttributeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "['/metagame/standard/full#paper',\n '/metagame/modern/full#paper',\n '/metagame/pioneer/full#paper',\n '/metagame/pauper/full#paper',\n '/metagame/legacy/full#paper',\n '/metagame/vintage/full#paper',\n '/metagame/historic/full#paper',\n '/metagame/penny_dreadful/full#paper',\n '/metagame/commander_1v1/full#paper',\n '/metagame/commander/full#paper',\n '/metagame/brawl/full#paper',\n '/archetype/sisay-weatherlight-captain#paper',\n '/archetype/golos-tireless-pilgrim#paper',\n '/archetype/kenrith-the-returned-king#paper',\n '/archetype/commander-rin-and-seri-inseparable#paper',\n '/archetype/breya-etherium-shaper#paper',\n '/archetype/chulane-teller-of-tales#paper',\n '/archetype/alela-artful-provocateur#paper',\n '/archetype/atraxa-praetors-voice#paper',\n '/archetype/muldrotha-the-gravetide#paper',\n '/archetype/yuriko-the-tiger-s-shadow#paper',\n '/archetype/korvold-fae-cursed-king#paper',\n '/archetype/krenko-mob-boss#paper',\n '/archetype/gishath-sun-s-avatar#paper',\n '/archetype/gavi-nest-warden#paper',\n '/archetype/kalamax-the-stormsire#paper',\n '/archetype/kess-dissident-mage#paper',\n '/archetype/the-ur-dragon#paper',\n '/archetype/arcades-the-strategist#paper',\n '/archetype/nethroi-apex-of-death#paper',\n '/archetype/niv-mizzet-parun#paper',\n '/archetype/meren-of-clan-nel-toth#paper',\n '/archetype/syr-gwyn-hero-of-ashvale#paper',\n '/archetype/kaalia-of-the-vast#paper',\n '/archetype/kinnan-bonder-prodigy#paper',\n '/archetype/jhoira-weatherlight-captain#paper',\n '/archetype/queen-marchesa#paper',\n '/archetype/tatyova-benthic-druid#paper',\n '/archetype/urza-lord-high-artificer#paper',\n '/archetype/najeela-the-blade-blossom#paper',\n '/archetype/lord-windgrace#paper',\n '/archetype/zaxara-the-exemplary#paper',\n '/archetype/omnath-locus-of-the-roil#paper',\n '/archetype/damia-sage-of-stone#paper',\n '/archetype/feather-the-redeemed#paper',\n '/archetype/animar-soul-of-elements#paper',\n '/archetype/commander-radha-heart-of-keld#paper',\n '/archetype/commander-tinybones-trinket-thief#paper',\n '/archetype/aminatou-the-fateshifter#paper',\n '/archetype/elsha-of-the-infinite#paper',\n '/archetype/alesha-who-smiles-at-death#paper',\n '/archetype/yarok-the-desecrated#paper',\n '/archetype/edgar-markov#paper',\n '/archetype/oloro-ageless-ascetic#paper',\n '/archetype/xyris-the-writhing-storm#paper',\n '/archetype/commander-inniaz-the-gale-force#paper',\n '/archetype/teysa-karlov#paper',\n '/archetype/kynaios-and-tiro-of-meletis#paper',\n '/archetype/k-rrik-son-of-yawgmoth#paper',\n '/archetype/the-scarab-god#paper',\n '/archetype/silvar-devourer-of-the-free-trynn-champion-of-freedom#paper',\n '/archetype/zedruu-the-greathearted#paper',\n '/archetype/atla-palani-nest-tender#paper',\n '/archetype/yidris-maelstrom-wielder#paper',\n '/archetype/shirei-shizo-s-caretaker#paper',\n '/archetype/hope-of-ghirapur#paper',\n '/archetype/anje-falkenrath#paper',\n '/archetype/zur-the-enchanter#paper',\n '/archetype/riku-of-two-reflections#paper',\n '/archetype/winota-joiner-of-forces#paper',\n '/archetype/kozilek-the-great-distortion#paper']"
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "source": [
    "deck_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting all the links at the top of the page....looks like we only want the ones starting with /archetype...Identify the pattern and use regular expressions to narrow down the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "['/archetype/sisay-weatherlight-captain#paper', '/archetype/golos-tireless-pilgrim#paper', '/archetype/kenrith-the-returned-king#paper', '/archetype/commander-rin-and-seri-inseparable#paper', '/archetype/breya-etherium-shaper#paper', '/archetype/chulane-teller-of-tales#paper', '/archetype/alela-artful-provocateur#paper', '/archetype/atraxa-praetors-voice#paper', '/archetype/muldrotha-the-gravetide#paper', '/archetype/yuriko-the-tiger-s-shadow#paper', '/archetype/korvold-fae-cursed-king#paper', '/archetype/krenko-mob-boss#paper', '/archetype/gishath-sun-s-avatar#paper', '/archetype/gavi-nest-warden#paper', '/archetype/kalamax-the-stormsire#paper', '/archetype/kess-dissident-mage#paper', '/archetype/the-ur-dragon#paper', '/archetype/arcades-the-strategist#paper', '/archetype/nethroi-apex-of-death#paper', '/archetype/niv-mizzet-parun#paper', '/archetype/meren-of-clan-nel-toth#paper', '/archetype/syr-gwyn-hero-of-ashvale#paper', '/archetype/kaalia-of-the-vast#paper', '/archetype/kinnan-bonder-prodigy#paper', '/archetype/jhoira-weatherlight-captain#paper', '/archetype/queen-marchesa#paper', '/archetype/tatyova-benthic-druid#paper', '/archetype/urza-lord-high-artificer#paper', '/archetype/najeela-the-blade-blossom#paper', '/archetype/lord-windgrace#paper', '/archetype/zaxara-the-exemplary#paper', '/archetype/omnath-locus-of-the-roil#paper', '/archetype/damia-sage-of-stone#paper', '/archetype/feather-the-redeemed#paper', '/archetype/animar-soul-of-elements#paper', '/archetype/commander-radha-heart-of-keld#paper', '/archetype/commander-tinybones-trinket-thief#paper', '/archetype/aminatou-the-fateshifter#paper', '/archetype/elsha-of-the-infinite#paper', '/archetype/alesha-who-smiles-at-death#paper', '/archetype/yarok-the-desecrated#paper', '/archetype/edgar-markov#paper', '/archetype/oloro-ageless-ascetic#paper', '/archetype/xyris-the-writhing-storm#paper', '/archetype/commander-inniaz-the-gale-force#paper', '/archetype/teysa-karlov#paper', '/archetype/kynaios-and-tiro-of-meletis#paper', '/archetype/k-rrik-son-of-yawgmoth#paper', '/archetype/the-scarab-god#paper', '/archetype/silvar-devourer-of-the-free-trynn-champion-of-freedom#paper', '/archetype/zedruu-the-greathearted#paper', '/archetype/atla-palani-nest-tender#paper', '/archetype/yidris-maelstrom-wielder#paper', '/archetype/shirei-shizo-s-caretaker#paper', '/archetype/hope-of-ghirapur#paper', '/archetype/anje-falkenrath#paper', '/archetype/zur-the-enchanter#paper', '/archetype/riku-of-two-reflections#paper', '/archetype/winota-joiner-of-forces#paper', '/archetype/kozilek-the-great-distortion#paper']\n"
    }
   ],
   "source": [
    "deck_urls = []\n",
    "try:\n",
    "    deck_url = bs.findAll('span', {'class': \"deck-price-paper\"})\n",
    "    for i in deck_url:\n",
    "        for link in i.findAll('a', href=re.compile(r'/archetype*')):\n",
    "            deck_urls.append(link.attrs['href'])\n",
    "except AttributeError as e:\n",
    "    print(e)\n",
    "\n",
    "print(deck_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we have a deck list we can loop through! Look at first deck."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "https://www.mtggoldfish.com/archetype/golos-tireless-pilgrim#paper\n"
    }
   ],
   "source": [
    "deck_url = DOMAIN + deck_urls[1]\n",
    "print(deck_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "[<title>Golos, Tireless Pilgrim Deck for Magic: the Gathering</title>, <meta content=\"Golos, Tireless Pilgrim deck list with prices for Magic: the Gathering (MTG).\" name=\"description\"/>, <meta content=\"golos, tireless pilgrim, golos, tireless pilgrim, sol ring, cultivate, commander, deck, decklist, price, archetype, magic, mtg, magic the gathering, magic the gathering online, mtgo\" name=\"keywords\"/>, <meta content=\"https://cdn1.mtggoldfish.com/images/gf/Golos%252C%2BTireless%2BPilgrim%2B%255BM20%255D.jpg\" property=\"og:image\"/>, <meta content=\"Golos, Tireless Pilgrim Deck for Magic: the Gathering\" property=\"og:title\"/>, <meta content=\"website\" property=\"og:type\"/>, <meta content=\"https://www.mtggoldfish.com/archetype/golos-tireless-pilgrim\" property=\"og:url\"/>, <meta content=\"Golos, Tireless Pilgrim deck list with prices for Magic: the Gathering (MTG).\" property=\"og:description\"/>, <meta content=\"summary\" name=\"twitter:card\"/>, <meta content=\"@mtggoldfish\" name=\"twitter:site\"/>, <link href=\"https://www.mtggoldfish.com/feed\" rel=\"alternate\" title=\"ATOM\" type=\"application/atom+xml\"/>, <link href=\"https://www.mtggoldfish.com/feed.rss\" rel=\"alternate\" title=\"RSS\" type=\"application/rss+xml\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-32-22599602a047942143b34b034de85b0968d377caf56ff4a12c06146a4c13b989.png\" rel=\"apple-touch-icon\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-72-f4155e01f71cfcaaa34af38dea8296e912c56eb5474a0ecc60064b4601ffebd6.png\" rel=\"apple-touch-icon\" sizes=\"72x72\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-114-1d9d14e0a25de8fa2df41aed9ef917d10c27f55d07a2ca60c34331c56c6b8030.png\" rel=\"apple-touch-icon\" sizes=\"114x114\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-144-fbf44d4bca87904f865574e1cc5ae42ac3789e284b765bc16eb920373b14b906.png\" rel=\"apple-touch-icon\" sizes=\"144x144\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-64-f5b29fefb4ab86245883253afa5faffc2ffae42f26cd0291e620c074f354b3a6.png\" rel=\"apple-touch-icon-precomposed\"/>, <link href=\"https://assets1.mtggoldfish.com/assets/goldfish-32-22599602a047942143b34b034de85b0968d377caf56ff4a12c06146a4c13b989.png\" rel=\"shortcut icon\" type=\"image/png\"/>, <link href=\"https://fonts.googleapis.com/css?family=Montserrat:400,700,900|Open+Sans\" rel=\"stylesheet\"/>, <meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>, <meta content=\"width=device-width, initial-scale=1\" name=\"viewport\"/>, <meta content=\"authenticity_token\" name=\"csrf-param\">\n<meta content=\"GFyw0Ixg+xRgvKI8XeiRA7SqKqp3QSfHslvY6VQB7ahn+PuF6SUzefGX4oyTCwWoniLXYXjDQqMw1duhI7yLqQ==\" name=\"csrf-token\">\n<link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n<link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n<script>(function() {\nvar _fbq = window._fbq || (window._fbq = []);\nif (!_fbq.loaded) {\nvar fbds = document.createElement('script');\nfbds.async = true;\nfbds.src = '//connect.facebook.net/en_US/fbds.js';\nvar s = document.getElementsByTagName('script')[0];\ns.parentNode.insertBefore(fbds, s);\n_fbq.loaded = true;\n}\n_fbq.push(['addPixelId', '928681210525176']);\n})();\nwindow._fbq = window._fbq || [];\nwindow._fbq.push(['track', 'PixelInitialized', {}]);\n</script>\n<noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n<script type=\"text/javascript\">\n  window[\"nitroAds\"] = window[\"nitroAds\"] || {\n    createAd: function() {\n      window.nitroAds.queue.push([\"createAd\", arguments]);\n    },\n    queue: []\n  };\n</script>\n<script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n</link></link></meta></meta>, <meta content=\"GFyw0Ixg+xRgvKI8XeiRA7SqKqp3QSfHslvY6VQB7ahn+PuF6SUzefGX4oyTCwWoniLXYXjDQqMw1duhI7yLqQ==\" name=\"csrf-token\">\n<link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n<link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n<script>(function() {\nvar _fbq = window._fbq || (window._fbq = []);\nif (!_fbq.loaded) {\nvar fbds = document.createElement('script');\nfbds.async = true;\nfbds.src = '//connect.facebook.net/en_US/fbds.js';\nvar s = document.getElementsByTagName('script')[0];\ns.parentNode.insertBefore(fbds, s);\n_fbq.loaded = true;\n}\n_fbq.push(['addPixelId', '928681210525176']);\n})();\nwindow._fbq = window._fbq || [];\nwindow._fbq.push(['track', 'PixelInitialized', {}]);\n</script>\n<noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n<script type=\"text/javascript\">\n  window[\"nitroAds\"] = window[\"nitroAds\"] || {\n    createAd: function() {\n      window.nitroAds.queue.push([\"createAd\", arguments]);\n    },\n    queue: []\n  };\n</script>\n<script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n</link></link></meta>, <link href=\"https://assets1.mtggoldfish.com/assets/application-7cc5263e55d07cb6f32f8be85c9cc859792f7c7789b82f250c162f58ac5d7ca9.css\" media=\"all\" rel=\"stylesheet\">\n<link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n<script>(function() {\nvar _fbq = window._fbq || (window._fbq = []);\nif (!_fbq.loaded) {\nvar fbds = document.createElement('script');\nfbds.async = true;\nfbds.src = '//connect.facebook.net/en_US/fbds.js';\nvar s = document.getElementsByTagName('script')[0];\ns.parentNode.insertBefore(fbds, s);\n_fbq.loaded = true;\n}\n_fbq.push(['addPixelId', '928681210525176']);\n})();\nwindow._fbq = window._fbq || [];\nwindow._fbq.push(['track', 'PixelInitialized', {}]);\n</script>\n<noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n<script type=\"text/javascript\">\n  window[\"nitroAds\"] = window[\"nitroAds\"] || {\n    createAd: function() {\n      window.nitroAds.queue.push([\"createAd\", arguments]);\n    },\n    queue: []\n  };\n</script>\n<script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n</link></link>, <link href=\"https://assets1.mtggoldfish.com/assets/print-6ea5076dba0b9bf0a07bd1b6ba2c5f860ceb3acc4e7adac560bbaf6f8f89b26a.css\" media=\"print\" rel=\"stylesheet\">\n<script>(function() {\nvar _fbq = window._fbq || (window._fbq = []);\nif (!_fbq.loaded) {\nvar fbds = document.createElement('script');\nfbds.async = true;\nfbds.src = '//connect.facebook.net/en_US/fbds.js';\nvar s = document.getElementsByTagName('script')[0];\ns.parentNode.insertBefore(fbds, s);\n_fbq.loaded = true;\n}\n_fbq.push(['addPixelId', '928681210525176']);\n})();\nwindow._fbq = window._fbq || [];\nwindow._fbq.push(['track', 'PixelInitialized', {}]);\n</script>\n<noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>\n<script type=\"text/javascript\">\n  window[\"nitroAds\"] = window[\"nitroAds\"] || {\n    createAd: function() {\n      window.nitroAds.queue.push([\"createAd\", arguments]);\n    },\n    queue: []\n  };\n</script>\n<script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>\n</link>, <script>(function() {\nvar _fbq = window._fbq || (window._fbq = []);\nif (!_fbq.loaded) {\nvar fbds = document.createElement('script');\nfbds.async = true;\nfbds.src = '//connect.facebook.net/en_US/fbds.js';\nvar s = document.getElementsByTagName('script')[0];\ns.parentNode.insertBefore(fbds, s);\n_fbq.loaded = true;\n}\n_fbq.push(['addPixelId', '928681210525176']);\n})();\nwindow._fbq = window._fbq || [];\nwindow._fbq.push(['track', 'PixelInitialized', {}]);\n</script>, <noscript><img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/></noscript>, <img alt=\"\" height=\"1\" src=\"https://www.facebook.com/tr?id=928681210525176&amp;ev=PixelInitialized\" style=\"display:none\" width=\"1\"/>, <script type=\"text/javascript\">\n  window[\"nitroAds\"] = window[\"nitroAds\"] || {\n    createAd: function() {\n      window.nitroAds.queue.push([\"createAd\", arguments]);\n    },\n    queue: []\n  };\n</script>, <script async=\"\" src=\"https://s.nitropay.com/ads-243.js\"></script>]\n"
    }
   ],
   "source": [
    "#Get the deck page\n",
    "try:\n",
    "    html = urlopen(deck_url)\n",
    "except HTTPError as e:\n",
    "    print(e)\n",
    "except URLError as e:\n",
    "    print('The server could not be found!')\n",
    "try:\n",
    "    page = BeautifulSoup(html.read(), 'html.parser')\n",
    "    html.close()\n",
    "except AttributeError as e:\n",
    "    print(e)\n",
    "print(page.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Data!\n",
    "\n",
    "+ Deck Name\n",
    "+ NumberOfDecks\n",
    "+ % of Metagame\n",
    "+ Online Price\n",
    "+ Paper Price\n",
    "+ Cards!!!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deck Name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[<h1 class=\"deck-view-title\">\n Golos, Tireless Pilgrim\n <a class=\"deck-report-issue\" href=\"/contact?title=Deck+name+issue+with+Golos%2C+Tireless+Pilgrim&amp;topic=Incorrect+or+inappropriate+deck+name.&amp;url=https%3A%2F%2Fwww.mtggoldfish.com%2Farchetype%2Fgolos-tireless-pilgrim\"><span class=\"glyphicon glyphicon-warning-sign\"></span>\n Report Deck Name\n </a></h1>]"
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "source": [
    "page.findAll('h1', {'class': 'deck-view-title'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Deck name: Golos, Tireless Pilgrim\n"
    }
   ],
   "source": [
    "try:\n",
    "    for i in page.findAll('h1', {'class': 'deck-view-title'}):\n",
    "        name =i.get_text()\n",
    "        name=name.split('\\n\\n')[0].strip('\\n')\n",
    "        print(f'Deck name: {name}')\n",
    "except AttributeError as e:\n",
    "    print(e)\n",
    "#page.findAll('h1', {'class': 'deck-view-title'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of Decks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "['48', 'Decks', '(1.73%', 'of', 'meta)']\nNumber of Decks: 48\n% of Metagame: 1.73\n"
    }
   ],
   "source": [
    "deckStats = page.find('p').get_text().strip().split()\n",
    "# Number of Decks of same kind on MTG Goldfish  #\n",
    "numDecks = deckStats[0]\n",
    "# % of Meta #\n",
    "pctMeta = re.sub('[^0-9.]+', '', deckStats[2])\n",
    "\n",
    "print(deckStats)\n",
    "print(f'Number of Decks: {numDecks}')\n",
    "print(f'% of Metagame: {pctMeta}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Online Price/Paper Price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Paper Price: 1,349.48\nOnline Price: 322.07\n"
    }
   ],
   "source": [
    "# $ Deck #\n",
    "deckPrice_paper = list(page.find('div', {'class': 'price-box paper'}).div.next_siblings)[1].get_text()\n",
    "deckPrice_online = list(page.find('div', {'class': 'price-box online'}).div.next_siblings)[1].get_text()\n",
    "print(f'Paper Price: {deckPrice_paper}')\n",
    "print(f'Online Price: {deckPrice_online}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Cards!!\n",
    "\n",
    "This is the real impetus behind this use case: what cards are in the most popular decks being played right now?\n",
    "\n",
    "A few applications in mind for using this data paired with the data collected above:\n",
    "\n",
    "+ Network analysis of decks with prices as edge weights\n",
    "+ Text analysis of entire decks\n",
    "+ EDA on deck stats with price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "['Golos, Tireless Pilgrim', 'Bloom Tender', 'Faeburrow Elder', 'Arena Rector', \"Atraxa, Praetors' Voice\", 'Felidar Guardian', 'Nicol Bolas, the Ravager', 'Spark Double', 'Deepglow Skate', 'Seedborn Muse', 'Aminatou, the Fateshifter', 'Narset, Parter of Veils', 'Oko, Thief of Crowns', 'Saheeli Rai', 'Teferi, Time Raveler', 'Chandra, Torch of Defiance', 'Narset Transcendent', 'Ral Zarek', 'Tamiyo, Field Researcher', 'Teferi, Master of Time', 'Nicol Bolas, Dragon-God', 'Teferi, Hero of Dominaria', 'Tezzeret the Seeker', 'Tezzeret, Artifice Master', 'Chandra, Awakened Inferno', 'Liliana, Dreadhorde General', 'Teferi, Temporal Archmage', 'Nicol Bolas, God-Pharaoh', 'Nicol Bolas, Planeswalker', 'Ugin, the Spirit Dragon', 'Swords to Plowshares', \"Assassin's Trophy\", 'Cyclonic Rift', 'Heroic Intervention', 'Generous Gift', \"Teferi's Protection\", 'Supreme Verdict', \"Urza's Ruinous Blast\", 'Merciless Eviction', \"Primevals' Glorious Rebirth\", 'Sol Ring', 'Arcane Signet', 'Talisman of Creativity', 'Talisman of Curiosity', 'Talisman of Dominance', 'Talisman of Indulgence', 'Talisman of Progress', 'Chromatic Lantern', \"Commander's Sphere\", 'The Chain Veil', 'Vedalken Orrery', \"Norn's Annex\", 'Fires of Invention', 'Leyline of Anticipation', 'Smothering Tithe', 'Wilderness Reclamation', 'Doubling Season', 'Maelstrom Nexus', \"Mirari's Wake\", 'Oath of Teferi', 'Sphere of Safety', 'Omniscience', 'Ancient Tomb', 'Arid Mesa', 'Blood Crypt', 'Bloodstained Mire', 'Breeding Pool', 'Cascading Cataracts', 'City of Brass', 'Command Tower', 'Exotic Orchard', 'Fabled Passage', 'Field of the Dead', 'Flooded Strand', 'Forbidden Orchard', 'Forest', 'Godless Shrine', 'Hallowed Fountain', 'Interplanar Beacon', 'Island', \"Karn's Bastion\", 'Mana Confluence', 'Marsh Flats', 'Misty Rainforest', 'Mountain', 'Overgrown Tomb', 'Plains', 'Polluted Delta', 'Prismatic Vista', 'Reflecting Pool', 'Sacred Foundry', 'Scalding Tarn', 'Steam Vents', 'Stomping Ground', 'Swamp', 'Temple Garden', 'Verdant Catacombs', 'Watery Grave', 'Windswept Heath', 'Wooded Foothills']\n"
    }
   ],
   "source": [
    "###First get the cards\n",
    "\n",
    "card_list = []\n",
    "\n",
    "try:\n",
    "    deck_page = page.find('table', {'class': \"deck-view-deck-table\"})\n",
    "\n",
    "    link_finder = re.compile('^(.*(\\/price).*)', re.IGNORECASE)\n",
    "\n",
    "    for i in deck_page.findAll('a', {'href': link_finder}):\n",
    "        card = i.get_text()\n",
    "        if '//' in card.split():\n",
    "            card = card.split(' // ')\n",
    "            for c in card:\n",
    "                card_list.append(c)\n",
    "        else:\n",
    "            card_list.append(card)\n",
    "except AttributeError as e:\n",
    "    print(e)\n",
    "\n",
    "print(card_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "deck_list = {}\n",
    "try:\n",
    "    for j in range(0, len(card_list)):\n",
    "        one_card = [(card.name, card.text, card.colors, card.mana_cost) for card in cards if card.name == card_list[j]]\n",
    "        if not one_card:\n",
    "            deck_list[j] = {'name': card_list[j], 'text': 'NA',\n",
    "                                'colors': 'NA', 'mana_cost': 'NA'}\n",
    "        else:\n",
    "            deck_list[j] = {'name': one_card[0][0], 'text': one_card[0][1],\n",
    "                                'colors': one_card[0][2], 'mana_cost': one_card[0][3]}\n",
    "except AttributeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "tap another target permanent.\\n[−2]: Ral Zarek deals 3 damage to any target.\\n[−7]: Flip five coins. Take an extra turn after this one for each coin that comes up heads.', 'colors': ['Red', 'Blue'], 'mana_cost': '{2}{U}{R}'}, 18: {'name': 'Tamiyo, Field Researcher', 'text': '[+1]: Choose up to two target creatures. Until your next turn, whenever either of those creatures deals combat damage, you draw a card.\\n[−2]: Tap up to two target nonland permanents. They don\\'t untap during their controller\\'s next untap step.\\n[−7]: Draw three cards. You get an emblem with \"You may cast spells from your hand without paying their mana costs.\"', 'colors': ['Green', 'Blue', 'White'], 'mana_cost': '{1}{G}{W}{U}'}, 19: {'name': 'Teferi, Master of Time', 'text': \"You may activate loyalty abilities of Teferi, Master of Time on any player's turn any time you could cast an instant.\\n[+1]: Draw a card, then discard a card.\\n[−3]: Target creature you don't control phases out. (Treat it and anything attached to it as though they don't exist until its controller's next turn.)\\n[−10]: Take two extra turns after this one.\", 'colors': ['Blue'], 'mana_cost': '{2}{U}{U}'}, 20: {'name': 'Nicol Bolas, Dragon-God', 'text': \"Nicol Bolas, Dragon-God has all loyalty abilities of all other planeswalkers on the battlefield.\\n[+1]: You draw a card. Each opponent exiles a card from their hand or a permanent they control.\\n[−3]: Destroy target creature or planeswalker.\\n[−8]: Each opponent who doesn't control a legendary creature or planeswalker loses the game.\", 'colors': ['Black', 'Red', 'Blue'], 'mana_cost': '{U}{B}{B}{B}{R}'}, 21: {'name': 'Teferi, Hero of Dominaria', 'text': '[+1]: Draw a card. At the beginning of the next end step, untap up to two lands.\\n[−3]: Put target nonland permanent into its owner\\'s library third from the top.\\n[−8]: You get an emblem with \"Whenever you draw a card, exile target permanent an opponent controls.\"', 'colors': ['Blue', 'White'], 'mana_cost': '{3}{W}{U}'}, 22: {'name': 'Tezzeret the Seeker', 'text': '[+1]: Untap up to two target artifacts.\\n[−X]: Search your library for an artifact card with converted mana cost X or less and put it onto the battlefield. Then shuffle your library.\\n[−5]: Artifacts you control become artifact creatures with base power and toughness 5/5 until end of turn.', 'colors': ['Blue'], 'mana_cost': '{3}{U}{U}'}, 23: {'name': 'Tezzeret, Artifice Master', 'text': '[+1]: Create a 1/1 colorless Thopter artifact creature token with flying.\\n[0]: Draw a card. If you control three or more artifacts, draw two cards instead.\\n[−9]: You get an emblem with \"At the beginning of your end step, search your library for a permanent card, put it onto the battlefield, then shuffle your library.\"', 'colors': ['Blue'], 'mana_cost': '{3}{U}{U}'}, 24: {'name': 'Chandra, Awakened Inferno', 'text': 'This spell can\\'t be countered.\\n[+2]: Each opponent gets an emblem with \"At the beginning of your upkeep, this emblem deals 1 damage to you.\"\\n[−3]: Chandra, Awakened Inferno deals 3 damage to each non-Elemental creature.\\n[−X]: Chandra, Awakened Inferno deals X damage to target creature or planeswalker. If a permanent dealt damage this way would die this turn, exile it instead.', 'colors': ['Red'], 'mana_cost': '{4}{R}{R}'}, 25: {'name': 'Liliana, Dreadhorde General', 'text': 'Whenever a creature you control dies, draw a card.\\n[+1]: Create a 2/2 black Zombie creature token.\\n[−4]: Each player sacrifices two creatures.\\n[−9]: Each opponent chooses a permanent they control of each permanent type and sacrifices the rest.', 'colors': ['Black'], 'mana_cost': '{4}{B}{B}'}, 26: {'name': 'Teferi, Temporal Archmage', 'text': '[+1]: Look at the top two cards of your library. Put one of them into your hand and the other on the bottom of your library.\\n[−1]: Untap up to four target permanents.\\n[−10]: You get an emblem with \"You may activate loyalty abilities of planeswalkers you control on any player\\'s turn any time you could cast an instant.\"\\nTeferi, Temporal Archmage can be your commander.', 'colors': ['Blue'], 'mana_cost': '{4}{U}{U}'}, 27: {'name': 'Nicol Bolas, God-Pharaoh', 'text': '[+2]: Target opponent exiles cards from the top of their library until they exile a nonland card. Until end of turn, you may cast that card without paying its mana cost.\\n[+1]: Each opponent exiles two cards from their hand.\\n[−4]: Nicol Bolas, God-Pharaoh deals 7 damage to target opponent, creature an opponent controls, or planeswalker an opponent controls.\\n[−12]: Exile each nonland permanent your opponents control.', 'colors': ['Black', 'Red', 'Blue'], 'mana_cost': '{4}{U}{B}{R}'}, 28: {'name': 'Nicol Bolas, Planeswalker', 'text': \"[+3]: Destroy target noncreature permanent.\\n[−2]: Gain control of target creature.\\n[−9]: Nicol Bolas, Planeswalker deals 7 damage to target player or planeswalker. That player or that planeswalker's controller discards seven cards, then sacrifices seven permanents.\", 'colors': ['Black', 'Red', 'Blue'], 'mana_cost': '{4}{U}{B}{B}{R}'}, 29: {'name': 'Ugin, the Spirit Dragon', 'text': \"[+2]: Ugin, the Spirit Dragon deals 3 damage to any target.\\n[−X]: Exile each permanent with converted mana cost X or less that's one or more colors.\\n[−10]: You gain 7 life, draw seven cards, then put up to seven permanent cards from your hand onto the battlefield.\", 'colors': [], 'mana_cost': '{8}'}, 30: {'name': 'Swords to Plowshares', 'text': 'Exile target creature. Its controller gains life equal to its power.', 'colors': ['White'], 'mana_cost': '{W}'}, 31: {'name': \"Assassin's Trophy\", 'text': 'Destroy target permanent an opponent controls. Its controller may search their library for a basic land card, put it onto the battlefield, then shuffle their library.', 'colors': ['Black', 'Green'], 'mana_cost': '{B}{G}'}, 32: {'name': 'Cyclonic Rift', 'text': 'Return target nonland permanent you don\\'t control to its owner\\'s hand.\\nOverload {6}{U} (You may cast this spell for its overload cost. If you do, change its text by replacing all instances of \"target\" with \"each.\")', 'colors': ['Blue'], 'mana_cost': '{1}{U}'}, 33: {'name': 'Heroic Intervention', 'text': 'Permanents you control gain hexproof and indestructible until end of turn.', 'colors': ['Green'], 'mana_cost': '{1}{G}'}, 34: {'name': 'Generous Gift', 'text': 'Destroy target permanent. Its controller creates a 3/3 green Elephant creature token.', 'colors': ['White'], 'mana_cost': '{2}{W}'}, 35: {'name': \"Teferi's Protection\", 'text': \"Until your next turn, your life total can't change and you gain protection from everything. All permanents you control phase out. (While they're phased out, they're treated as though they don't exist. They phase in before you untap during your untap step.)\\nExile Teferi's Protection.\", 'colors': ['White'], 'mana_cost': '{2}{W}'}, 36: {'name': 'Supreme Verdict', 'text': \"This spell can't be countered.\\nDestroy all creatures.\", 'colors': ['Blue', 'White'], 'mana_cost': '{1}{W}{W}{U}'}, 37: {'name': \"Urza's Ruinous Blast\", 'text': \"(You may cast a legendary sorcery only if you control a legendary creature or planeswalker.)\\nExile all nonland permanents that aren't legendary.\", 'colors': ['White'], 'mana_cost': '{4}{W}'}, 38: {'name': 'Merciless Eviction', 'text': 'Choose one —\\n• Exile all artifacts.\\n• Exile all creatures.\\n• Exile all enchantments.\\n• Exile all planeswalkers.', 'colors': ['Black', 'White'], 'mana_cost': '{4}{W}{B}'}, 39: {'name': \"Primevals' Glorious Rebirth\", 'text': '(You may cast a legendary sorcery only if you control a legendary creature or planeswalker.)\\nReturn all legendary permanent cards from your graveyard to the battlefield.', 'colors': ['Black', 'White'], 'mana_cost': '{5}{W}{B}'}, 40: {'name': 'Sol Ring', 'text': '{T}: Add {C}{C}.', 'colors': [], 'mana_cost': '{1}'}, 41: {'name': 'Arcane Signet', 'text': \"{T}: Add one mana of any color in your commander's color identity.\", 'colors': [], 'mana_cost': '{2}'}, 42: {'name': 'Talisman of Creativity', 'text': '{T}: Add {C}.\\n{T}: Add {U} or {R}. Talisman of Creativity deals 1 damage to you.', 'colors': [], 'mana_cost': '{2}'}, 43: {'name': 'Talisman of Curiosity', 'text': '{T}: Add {C}.\\n{T}: Add {G} or {U}. Talisman of Curiosity deals 1 damage to you.', 'colors': [], 'mana_cost': '{2}'}, 44: {'name': 'Talisman of Dominance', 'text': '{T}: Add {C}.\\n{T}: Add {U} or {B}. Talisman of Dominance deals 1 damage to you.', 'colors': [], 'mana_cost': '{2}'}, 45: {'name': 'Talisman of Indulgence', 'text': '{T}: Add {C}.\\n{T}: Add {B} or {R}. Talisman of Indulgence deals 1 damage to you.', 'colors': [], 'mana_cost': '{2}'}, 46: {'name': 'Talisman of Progress', 'text': '{T}: Add {C}.\\n{T}: Add {W} or {U}. Talisman of Progress deals 1 damage to you.', 'colors': [], 'mana_cost': '{2}'}, 47: {'name': 'Chromatic Lantern', 'text': 'Lands you control have \"{T}: Add one mana of any color.\"\\n{T}: Add one mana of any color.', 'colors': [], 'mana_cost': '{3}'}, 48: {'name': \"Commander's Sphere\", 'text': \"{T}: Add one mana of any color in your commander's color identity.\\nSacrifice Commander's Sphere: Draw a card.\", 'colors': [], 'mana_cost': '{3}'}, 49: {'name': 'The Chain Veil', 'text': \"At the beginning of your end step, if you didn't activate a loyalty ability of a planeswalker this turn, you lose 2 life.\\n{4}, {T}: For each planeswalker you control, you may activate one of its loyalty abilities once this turn as though none of its loyalty abilities have been activated this turn.\", 'colors': [], 'mana_cost': '{4}'}, 50: {'name': 'Vedalken Orrery', 'text': 'You may cast spells as though they had flash.', 'colors': [], 'mana_cost': '{4}'}, 51: {'name': \"Norn's Annex\", 'text': \"({W/P} can be paid with either {W} or 2 life.)\\nCreatures can't attack you or a planeswalker you control unless their controller pays {W/P} for each of those creatures.\", 'colors': ['White'], 'mana_cost': '{3}{W/P}{W/P}'}, 52: {'name': 'Fires of Invention', 'text': 'You can cast spells only during your turn and you can cast no more than two spells each turn.\\nYou may cast spells with converted mana cost less than or equal to the number of lands you control without paying their mana costs.', 'colors': ['Red'], 'mana_cost': '{3}{R}'}, 53: {'name': 'Leyline of Anticipation', 'text': 'If Leyline of Anticipation is in your opening hand, you may begin the game with it on the battlefield.\\nYou may cast spells as though they had flash.', 'colors': ['Blue'], 'mana_cost': '{2}{U}{U}'}, 54: {'name': 'Smothering Tithe', 'text': 'Whenever an opponent draws a card, that player may pay {2}. If the player doesn\\'t, you create a Treasure token. (It\\'s an artifact with \"{T}, Sacrifice this artifact: Add one mana of any color.\")', 'colors': ['White'], 'mana_cost': '{3}{W}'}, 55: {'name': 'Wilderness Reclamation', 'text': 'At the beginning of your end step, untap all lands you control.', 'colors': ['Green'], 'mana_cost': '{3}{G}'}, 56: {'name': 'Doubling Season', 'text': 'If an effect would create one or more tokens under your control, it creates twice that many of those tokens instead.\\nIf an effect would put one or more counters on a permanent you control, it puts twice that many of those counters on that permanent instead.', 'colors': ['Green'], 'mana_cost': '{4}{G}'}, 57: {'name': 'Maelstrom Nexus', 'text': 'The first spell you cast each turn has cascade. (When you cast your first spell, exile cards from the top of your library until you exile a nonland card that costs less. You may cast it without paying its mana cost. Put the exiled cards on the bottom of your library in a random order.)', 'colors': ['Black', 'Green', 'Red', 'Blue', 'White'], 'mana_cost': '{W}{U}{B}{R}{G}'}, 58: {'name': \"Mirari's Wake\", 'text': 'Creatures you control get +1/+1.\\nWhenever you tap a land for mana, add one mana of any type that land produced.', 'colors': ['Green', 'White'], 'mana_cost': '{3}{G}{W}'}, 59: {'name': 'Oath of Teferi', 'text': \"When Oath of Teferi enters the battlefield, exile another target permanent you control. Return it to the battlefield under its owner's control at the beginning of the next end step.\\nYou may activate the loyalty abilities of planeswalkers you control twice each turn rather than only once.\", 'colors': ['Blue', 'White'], 'mana_cost': '{3}{W}{U}'}, 60: {'name': 'Sphere of Safety', 'text': \"Creatures can't attack you or a planeswalker you control unless their controller pays {X} for each of those creatures, where X is the number of enchantments you control.\", 'colors': ['White'], 'mana_cost': '{4}{W}'}, 61: {'name': 'Omniscience', 'text': 'You may cast spells from your hand without paying their mana costs.', 'colors': ['Blue'], 'mana_cost': '{7}{U}{U}{U}'}, 62: {'name': 'Ancient Tomb', 'text': '{T}: Add {C}{C}. Ancient Tomb deals 2 damage to you.', 'colors': [], 'mana_cost': None}, 63: {'name': 'Arid Mesa', 'text': '{T}, Pay 1 life, Sacrifice Arid Mesa: Search your library for a Mountain or Plains card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 64: {'name': 'Blood Crypt', 'text': \"({T}: Add {B} or {R}.)\\nAs Blood Crypt enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 65: {'name': 'Bloodstained Mire', 'text': '{T}, Pay 1 life, Sacrifice Bloodstained Mire: Search your library for a Swamp or Mountain card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 66: {'name': 'Breeding Pool', 'text': \"({T}: Add {G} or {U}.)\\nAs Breeding Pool enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 67: {'name': 'Cascading Cataracts', 'text': 'Indestructible\\n{T}: Add {C}.\\n{5}, {T}: Add five mana in any combination of colors.', 'colors': [], 'mana_cost': None}, 68: {'name': 'City of Brass', 'text': 'Whenever City of Brass becomes tapped, it deals 1 damage to you.\\n{T}: Add one mana of any color.', 'colors': [], 'mana_cost': None}, 69: {'name': 'Command Tower', 'text': \"{T}: Add one mana of any color in your commander's color identity.\", 'colors': [], 'mana_cost': None}, 70: {'name': 'Exotic Orchard', 'text': '{T}: Add one mana of any color that a land an opponent controls could produce.', 'colors': [], 'mana_cost': None}, 71: {'name': 'Fabled Passage', 'text': '{T}, Sacrifice Fabled Passage: Search your library for a basic land card, put it onto the battlefield tapped, then shuffle your library. Then if you control four or more lands, untap that land.', 'colors': [], 'mana_cost': None}, 72: {'name': 'Field of the Dead', 'text': 'Field of the Dead enters the battlefield tapped.\\n{T}: Add {C}.\\nWhenever Field of the Dead or another land enters the battlefield under your control, if you control seven or more lands with different names, create a 2/2 black Zombie creature token.', 'colors': [], 'mana_cost': None}, 73: {'name': 'Flooded Strand', 'text': '{T}, Pay 1 life, Sacrifice Flooded Strand: Search your library for a Plains or Island card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 74: {'name': 'Forbidden Orchard', 'text': '{T}: Add one mana of any color.\\nWhenever you tap Forbidden Orchard for mana, target opponent creates a 1/1 colorless Spirit creature token.', 'colors': [], 'mana_cost': None}, 75: {'name': 'Forest', 'text': '({T}: Add {G}.)', 'colors': [], 'mana_cost': None}, 76: {'name': 'Godless Shrine', 'text': \"({T}: Add {W} or {B}.)\\nAs Godless Shrine enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 77: {'name': 'Hallowed Fountain', 'text': \"({T}: Add {W} or {U}.)\\nAs Hallowed Fountain enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 78: {'name': 'Interplanar Beacon', 'text': 'Whenever you cast a planeswalker spell, you gain 1 life.\\n{T}: Add {C}.\\n{1}, {T}: Add two mana of different colors. Spend this mana only to cast planeswalker spells.', 'colors': [], 'mana_cost': None}, 79: {'name': 'Island', 'text': '({T}: Add {U}.)', 'colors': [], 'mana_cost': None}, 80: {'name': \"Karn's Bastion\", 'text': '{T}: Add {C}.\\n{4}, {T}: Proliferate. (Choose any number of permanents and/or players, then give each another counter of each kind already there.)', 'colors': [], 'mana_cost': None}, 81: {'name': 'Mana Confluence', 'text': '{T}, Pay 1 life: Add one mana of any color.', 'colors': [], 'mana_cost': None}, 82: {'name': 'Marsh Flats', 'text': '{T}, Pay 1 life, Sacrifice Marsh Flats: Search your library for a Plains or Swamp card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 83: {'name': 'Misty Rainforest', 'text': '{T}, Pay 1 life, Sacrifice Misty Rainforest: Search your library for a Forest or Island card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 84: {'name': 'Mountain', 'text': '({T}: Add {R}.)', 'colors': [], 'mana_cost': None}, 85: {'name': 'Overgrown Tomb', 'text': \"({T}: Add {B} or {G}.)\\nAs Overgrown Tomb enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 86: {'name': 'Plains', 'text': '({T}: Add {W}.)', 'colors': [], 'mana_cost': None}, 87: {'name': 'Polluted Delta', 'text': '{T}, Pay 1 life, Sacrifice Polluted Delta: Search your library for an Island or Swamp card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 88: {'name': 'Prismatic Vista', 'text': '{T}, Pay 1 life, Sacrifice Prismatic Vista: Search your library for a basic land card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 89: {'name': 'Reflecting Pool', 'text': '{T}: Add one mana of any type that a land you control could produce.', 'colors': [], 'mana_cost': None}, 90: {'name': 'Sacred Foundry', 'text': \"({T}: Add {R} or {W}.)\\nAs Sacred Foundry enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 91: {'name': 'Scalding Tarn', 'text': '{T}, Pay 1 life, Sacrifice Scalding Tarn: Search your library for an Island or Mountain card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 92: {'name': 'Steam Vents', 'text': \"({T}: Add {U} or {R}.)\\nAs Steam Vents enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 93: {'name': 'Stomping Ground', 'text': \"({T}: Add {R} or {G}.)\\nAs Stomping Ground enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 94: {'name': 'Swamp', 'text': '({T}: Add {B}.)', 'colors': [], 'mana_cost': None}, 95: {'name': 'Temple Garden', 'text': \"({T}: Add {G} or {W}.)\\nAs Temple Garden enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 96: {'name': 'Verdant Catacombs', 'text': '{T}, Pay 1 life, Sacrifice Verdant Catacombs: Search your library for a Swamp or Forest card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 97: {'name': 'Watery Grave', 'text': \"({T}: Add {U} or {B}.)\\nAs Watery Grave enters the battlefield, you may pay 2 life. If you don't, it enters the battlefield tapped.\", 'colors': [], 'mana_cost': None}, 98: {'name': 'Windswept Heath', 'text': '{T}, Pay 1 life, Sacrifice Windswept Heath: Search your library for a Forest or Plains card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}, 99: {'name': 'Wooded Foothills', 'text': '{T}, Pay 1 life, Sacrifice Wooded Foothills: Search your library for a Mountain or Forest card, put it onto the battlefield, then shuffle your library.', 'colors': [], 'mana_cost': None}}\n"
    }
   ],
   "source": [
    "print(deck_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store the information\n",
    "\n",
    "Storing the data as a JSON object to be used for analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "decks[0] = {'DeckName': name, 'NumberOfDecks': numDecks,\n",
    "            'PercentOfDecks': pctMeta, 'PaperPrice': deckPrice_paper,\n",
    "            'OnlinePrice': deckPrice_online, 'Cards': deck_list}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "'Golos, Tireless Pilgrim'"
     },
     "metadata": {},
     "execution_count": 32
    }
   ],
   "source": [
    "decks[0].get('DeckName')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "{'name': 'Tezzeret the Seeker',\n 'text': '[+1]: Untap up to two target artifacts.\\n[−X]: Search your library for an artifact card with converted mana cost X or less and put it onto the battlefield. Then shuffle your library.\\n[−5]: Artifacts you control become artifact creatures with base power and toughness 5/5 until end of turn.',\n 'colors': ['Blue'],\n 'mana_cost': '{3}{U}{U}'}"
     },
     "metadata": {},
     "execution_count": 40
    }
   ],
   "source": [
    "decks[0].get('Cards')[22]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "100"
     },
     "metadata": {},
     "execution_count": 41
    }
   ],
   "source": [
    "len(decks[0].get('Cards'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data_demo.json', 'w') as fp:\n",
    "    json.dump(decks, fp, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beautiful Soup Cheatsheet\n",
    "\n",
    "+ [Beautiful Soup Cheatsheet on GitHub](https://gist.github.com/yoki/b7f2fcef64c893e307c4c59303ead19a)\n",
    "\n",
    "Beautiful Soup features not shown here:\n",
    "\n",
    "+ css selectors\n",
    "+ parent/child/sibling selectors (up/down/sideways naviagtion through html elements)\n",
    "+ modifying tags\n",
    "+ deleting/replacing/encoding\n",
    "+ soup strainer: parse document on import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "+ Official Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "+ Maggie's GitHub Repo: https://github.com/MKS310/MTG-Web-Scraping\n",
    "+ MTG: https://en.wikipedia.org/wiki/Magic:_The_Gathering\n",
    "+ MTG Goldfish: https://www.mtggoldfish.com\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.7 64-bit ('web-scraper': conda)",
   "language": "python",
   "name": "python37764bitwebscrapercondab008bf96f9b840eba2d1cb83292381da"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}