traffic-sign-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Traffic Sign Classification\n",
    "***\n",
    "\n",
    "In this script, I developed a convolutional neural network (CNN) to classify German traffic signs (43 categories). The emphasis of this script is to develop a simple, yet robust model, rather then *very clever* implementation. It can be divided into two parts:\n",
    "\n",
    "1. **Data perparation:** The images have various sizes, with 53x53 being the average. Therefore, we rescale all images to 52x52 to ensure equal images sizes throughout training. We further rescale the pixel values to be between -1.0 and 1.0 with a mean of 0.0 to improve the learning results. Finally, the images and their one-hot labels are saved and serialised appropriately.\n",
    "\n",
    "2. **Learning:** The data set is first divided into a training and validation set, and then learning is performed. The network architecture is the following:\n",
    "\n",
    "    Input(52,52,3)  \n",
    "    &darr;  \n",
    "    ConvLayer+ReLU and MaxPooling (26,26,32)  \n",
    "    &darr;  \n",
    "    ConvLayer+ReLU and MaxPooling (13,13,64)  \n",
    "    &darr;  \n",
    "    ConvLayer+ReLU and MaxPooling (7,7,128)  \n",
    "    &darr;  \n",
    "    DenseLayer+ReLU+Dropout(2048)  \n",
    "    &darr;  \n",
    "    DenseLayer+ReLU+Dropout(1024)  \n",
    "    &darr;  \n",
    "    DenseLayer+SoftMax(43)  \n",
    "    &darr;  \n",
    "    Cross-Entropy Loss\n",
    "    \n",
    "    We use the [Adam optimizer](https://arxiv.org/abs/1412.6980) which is a variant of the (stochastic) gradient descent algorithm. We train the neural network with a batch size of 128 examples. The classification accuracy on the current batch as well as the held-out validation set is calculated every 50 iterations. After 5000 iterations, which is equivalent to 16 epochs, a classification accuracy of 97% is achieved (compared to 2% at random). \n",
    "\n",
    "***\n",
    "## Additional Remarks\n",
    "\n",
    "The training data that was used can be downloaded from http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import cv2 as cv\n",
    "import pickle\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import figure\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import time\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Data and Rescaling\n",
    "\n",
    "In the first step, we have a quick look at the data and rescale it accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions of the image: (30, 29, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f09bd2678d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPcAAAD8CAYAAACrSzKQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHHdJREFUeJztnW2MnfV55q/7vM677fE7tvFgg6EpBGgMlC0iVNmkNKpKsquiRmpFV1GpVo3USP3QKPuh+bAfolWTKh9W0ToNKlll01abZINWkIaw6UI2W8C82BhsbPAbY3vG9ow97+f93g9z3B3Mue4zeIY5wz/XT7I8c67zPP//eea5ztv13Pff3B1CiPTIdHoCQogPBplbiESRuYVIFJlbiESRuYVIFJlbiESRuYVIFJlbiESRuYVIlNxSNjazBwF8A0AWwN+4+1ej+xcKRe/p6WmpOeIr5bIZ/jyUC56iunIFvs82F+cVM0a1WjCfgfXrqZbJ8/kAQKPBtXow38sT0+F+y9Ua1bLBlHqL/HH2FLrDMXPZPNVq1SrV6rOz4X4r05NUm5mbo5pl+eleD+YKAGWv823rXGtE53X0xw4olUqoVCr85GxyzeY2syyA/wzgkwCGAbxoZk+4+xtsm56eHtz38Qdaat7mgfb18jNwU5E/zj2bdlBtTSUcEkO9Rapd7uMn9m/+wb+jWv/W7eGYU4FHp7k/8cOn/k+43+Oj56m2dhs38D039lLtzutvC8dc17uZauMjI1SbPPBSuN/hZ39KtRfeOEC13JqNVJtYy+cKAEfL/A8zOX2JauVa8CRWKYdjGvHEiy+8GG53haW8Lb8bwFvuftzdKwD+DsBDS9ifEGIZWYq5twF4Z8Hvw83bhBCrgA/8CzUze9TM9pvZ/kqbtyFCiOVjKeY+A2DhB9rtzdvehbvvc/e97r63UOCfYYUQy8tSzP0igJvM7AYzKwD4fQBPLM+0hBBL5Zq/LXf3mpl9AcA/Yj4Ke8zdX4+2sYyhq9D6W+96LfgqGHHctb2/dbwGALeu499qHztEv9if18/xbT/2b/6IatE34u3Cj+Iarj3/7ATVRk/xeAgAbty8m2pbe9+iWj9eodpUbW04Zr62gWq9g1uo1rP7vnC/428d5fs9w+dbD765zpa7wjEHjP/lqpkZqjUyfLuu7vidbC6TbXl7Nru41+Ql5dzu/iSAJ5eyDyHEB4OuUBMiUWRuIRJF5hYiUWRuIRJF5hYiUWRuIRJlSVHY+8UbjmqldSlWVzEuhcwaL8nb3s8vac+cvUC18oX3XFD3Ln739/491XY++Em+YfCUWWlTZnp5luei09M8w71tz0C439/9+Eepduogr3h6+rn9VLv5/vi1oTDUOqcFgO6gYHFwdxD2A1i/61aq9R97m2p94KWZGVwOx7xc4mWocxm+3+k6Lz2sB/MBgEyu9Tm/2IVE9MotRKLI3EIkiswtRKLI3EIkiswtRKLI3EIkyspGYXDUSKfIfCOOBdZleNQzUO+jWuksL+vcuXldOObOG2/kIu+JBwSVkEF1IAAgN3uWamsyx6k2tOeucL9e4tpN9/wrPmZQKnrkFG9yCACW5Q92cAdvvIiJuPvprjt5SWj1HC/rfOHJfVSzAm8gCQB7d3+EaocneDQ1NsNPlEbQXRcAqkReXBCmV24hkkXmFiJRZG4hEkXmFiJRZG4hEkXmFiJRVjQKgzvq3rpKJt/gEQYAbCr0U21Ng3cFzZR4BrTlhiDqAjA1xTuK9gcpWuUiX4yuUQ8yKQAzIzySWZflcWBmmh8DAJhp8A6no8fHqTY3yaul1k3zxwkAoy++SbWz/5vP9+wFHgcCwMDgJqpdeu0w1Tbm+KIYM+dHwzEzBV6pNrR9D9WOXOTHb9rjXrjFgcHWcwkWNHzX/RZ1LyHEhw6ZW4hEkbmFSBSZW4hEkbmFSBSZW4hEWfGqsAZaR15bNrb+2v8Kt2zfSLXJE6eo1hVEYdlzvDEgANRP8CqsQwf+I9WGgwhtdjaueJqd5ZVU5aBJXzl4nEC80GItWCCvEmmkwu8K1uBRTzZo8jfVpgHgdFAX1WuRxpsVbtjBm2wCwEiZH79yiR+HRo4vMGiV+G+2lURh+UVGYUsyt5mdBDAFoA6g5u57l7I/IcTysRyv3L/p7heXYT9CiGVEn7mFSJSlmtsB/MTMXjKzR1vdwcweNbP9ZrafLUgghFh+lvq2/D53P2NmmwA8bWZH3P3ZhXdw930A9gHAwJqBxXaIEUIskSW9crv7meb/5wH8EMDdyzEpIcTSuWZzm1mvmfVf+RnApwAcWq6JCSGWxlLelm8G8EMzu7Kf/+buP442KHZ1Y2h36wXpesBzRADoz/FumSdP8Tx6N4pUmzsXLwR44AleeugZ/rxYzPAF8LqyfEFDAFgXZLhRUazzIQHwReUAINfNy2mrBZ7TNrr4dgBQDPTJyXNUy2wMOqMCGAPvGloyfiBmPTiCHp9/Y0dPU61S5nn1TJBl59qM6ayMt02n4H/Z/6Lu1Wpg9+MAbr/W7YUQHyyKwoRIFJlbiESRuYVIFJlbiESRuYVIlBUt+ewqduNmEoUNxo0gMVPjtSlb7uXXzpQnefxRz8dRTiPPY7RcoZtqhWIP1dbkebQ0rxeo1hdEbP3ZOFZBPehUWufP8fVu3nG10b0+HHJ0nMdAdol3Rn3z8rFwv4fO88X15qo8Spyr8QgpH5SnAkAhyBovjfAFEesNfsl1JhuPOTJ2oeXt1aAM9137X9S9hBAfOmRuIRJF5hYiUWRuIRJF5hYiUWRuIRJlRaOwYqGIXdfvbKn1Wfw8Ux7nC7EN7PoY1Z577hdUKwWLCwLAaHmKapVp3jk1m+MRx7Z83K9iE/hCgFvAx8xdjNvYTQd6t/HTYDbDte6Nm8MxewP97GUeZ400ePwGANM9/BiWy3zhvUaDx4wNj8+/qRKPtOZqXMsF8WWjTeuSConn2m13Bb1yC5EoMrcQiSJzC5EoMrcQiSJzC5EoMrcQibKiUZhlMujqad38zoKF6gCgf/2NVHv9JGkkB+BC5nqqjY3zqAsAxmfLVNue5fHH+mDRuN64PyJmZnmzx7fn3qaazcUVRpkij5cyRf4cX6nw+K16gldDAUD3aX4cenJbuNbTOi69wr27t1Pt+ASvRDs1ybVKOW46WGnwx5LJ87jLg0UNG/U408pmW/9djPeHfPe8Fnc3IcSHDZlbiESRuYVIFJlbiESRuYVIFJlbiESRuYVIlLY5t5k9BuB3AJx391ubtw0C+HsAQwBOAnjY3XkNXxP3Giqki2mtOhNue/4izyH7rhui2oVhnlVfKMVdJC3Ly0w3Gl+sbk/lFNUmxw+HY16/k+fy05VbqHb73nj15F1776LaxaCGsDfoCnr8wOvhmOcCPT/OF2E8PnYg3G+xi+fga/t4fn4yM061Sr3N6ZvlWXYmWJiwFnYqja9NaNRJtr6MJZ9/C+DBq277EoBn3P0mAM80fxdCrCLamtvdnwVw9VPeQwAeb/78OIDPLPO8hBBL5Fo/c2929ysLLI9gfq3ulpjZo2a238z2T07wy0SFEMvLkr9Q8/mLZ+mnAHff5+573X3vwBr+GVYIsbxcq7lHzWwrADT/542/hBAd4VrN/QSAR5o/PwLgR8szHSHEcrGYKOx7AB4AsMHMhgH8JYCvAvgHM/s8gFMAHl7MYOWZSZzY/3RLras/rmN77Qh/c7Dnnt/mGwalmUXweAMA1uV43FWY5uWDhaBMcsP6uMy0OMg7st52D//e8oZ7eAdYAEAP7/y5gz9MzEzwuGbd1jvDIdfs5HGXH3mCarU3/ke43+FTw1SbHRik2sCatVSrZ+O/SxRo5TL83PU6P//yhfj86yPl0awU9D3zancHd/8ckT6xqBGEEB1BV6gJkSgytxCJInMLkSgytxCJInMLkSgr2v20tyuLu3e3vkrtqX9+Jdx2bJJXhV2a4hU9FnSm7AKPKQBgWzcvv7muMUa1/pkgflu/JxyzZ/N6qo1lR6n20vP/PdzvbTfzSqrB3nVUe+4d3uH0tem+cMzKL45R7b4cP7bXD90f7tcuvkq1GfBjVOvmMVmpxKNCAOjq6aZavcIrD+sVfi4UcnEUZkHn1MWgV24hEkXmFiJRZG4hEkXmFiJRZG4hEkXmFiJRVjQKyxez2Hhj68qc/KGecNupE+9QrVrlFVr5PC956g4WwAOAbd18vzPDvIlfxXjEMXTnvw7HfN1bN5AEgIkj/4tqk+P8+ABA38QOqvXc8mtUy4/ymDF7lsdDANCX4805Dl3g8dGNd90b7rd+5CTVGuNvUu3mbh751ctBaRyAmTV8IcWpCX6MSllusXrQWBEACnmyauQiVwLUK7cQiSJzC5EoMrcQiSJzC5EoMrcQiSJzC5EoMrcQibKiOXepWsfRkcmWWr4vzrn7enlX0FqN5379A7yEslCMV0AZ9MtUO3WZZ5sbbr6NalvvjDuGvjLJS0mHNvGyzeLpg+F+Tx/m+tMHfka1iWoX1T5+/2fDMa+/79eptu/x/0K1nh08jwaADbuHqPbm2SNUyw7zctCtm7aGY75W4tc8VGu8HNmCnLtdRWeu2NoTZot7TdYrtxCJInMLkSgytxCJInMLkSgytxCJInMLkSiLWQjwMQC/A+C8u9/avO0rAP4YwIXm3b7s7k+221el2sBpEoXVnccJADDQx6MwK/PF6rZv4VFYo5tvBwBdzqOyKvi2XdddR7W+fl46CAB3bRqiWmlsjmt9G8L9Tk7yxQnvvuMBqj314+eptm4zj+YAIMdTNOSL/O89UR4P97vpxl1UK/88GDRDSigBWCa2QnV2hmre4OdCNojCGvX4/KtUaq3HW2RX1MW8cv8tgAdb3P7X7n5H819bYwshVpa25nb3ZwHET6VCiFXHUj5zf8HMDprZY2YWX1IkhFhxrtXc3wSwG8AdAM4B+Bq7o5k9amb7zWz/9PTsNQ4nhHi/XJO53X3U3evu3gDwLQB3B/fd5+573X1vX5vrx4UQy8c1mdvMFl5l/1kAh5ZnOkKI5WIxUdj3ADwAYIOZDQP4SwAPmNkdABzASQB/spjBqjXg7FjrCq6c88gKAAbX8E6aBeeRwtTZ16m2s00U1l/h8VHrkGKezHoeSxX6+eMAgL65KtUmgqfiHx/jFU8A0LX2dqrNDvDobjrLH+npkQtUA4BNO3+Fah5UNk1WeDUeAHhpimqVoDGoZfmYmUzcUbQaLPbnHp0N/ByzTBxpWYZsu7jmp+3N7e6fa3Hztxe3eyFEp9AVakIkiswtRKLI3EIkiswtRKLI3EIkiswtRKKsaPfTWh24PN46pKuV4pLPWplniZnZc1zL8a6Vt7bpstkV5Jdrc/zQTbz1Nt/peb6KJwD09PI5Hfj5c1S7eDzOnG//KF85syfIubs38BLVU1Px5cQzr56i2nSQc/d0xedCo8xz7kKDb5sL/ma1Kr++AAAaVZ5zZ3M8r3bjOXc+z1eDBYBstvW2huUr+RRCfAiRuYVIFJlbiESRuYVIFJlbiESRuYVIlBWNwgr5IrZfd0NLLdOm+2RE3ngNXC3opDnQJnKZOnOaauuC1p6NkTNcu8DjIQB4+iffp9rBV/ZTbfOuj4b7zdT5nE6f4yWW3Rt7qXZi8kQ45utnuD60Zw/Verq2hfs9cfQXVCs6j+emp/nijY2uoGsqgHyWx1YelHVG8Vs2KEGdh53Xi6v51Cu3EIkicwuRKDK3EIkicwuRKDK3EIkicwuRKCsahXV1dePmm1t34Ww04kqXRrBQYDmo2KnU1lLt2Mk3wjHXX3c91fJ8TT6MHD9KtZ899YNwzNv23kI1y26l2qGROGIbv8CjnCp45dcdt/BjgEwxHPO6br5447Zuvojga0+9HO732NF3qJYPorv8ntYxLACcHR4Lx8wFUVjNgyi2FsStbc75bKb1a+9yLgQohPgQInMLkSgytxCJInMLkSgytxCJInMLkSiLWQhwB4DvANiM+YX/9rn7N8xsEMDfAxjC/GKAD7s7L7tpkiGLmzXCxdQAIIgUCjyKmA3ihrdLcYO/o2XeNO/+7YNUWzfLo6WjB/9vOGb/Rh65fOoT91Ht3oEbw/12rbmJahXjxyFnPPNr1LrDMbfU+ZLN7zzzFNVGDnINAHZev51qJ/I87npxrEK1sTkepwJAvlCgWqHAH2dlboZqjVqbpoykEeJyRmE1AH/u7h8B8OsA/tTMPgLgSwCecfebADzT/F0IsUpoa253P+fuLzd/ngJwGMA2AA8BeLx5t8cBfOaDmqQQ4v3zvj5zm9kQgDsBPA9gs7tfaRg+gvm37UKIVcKizW1mfQC+D+CL7j65UPP5DwEtPwiY2aNmtt/M9k9OxIuqCyGWj0WZ28zymDf2d939ysXRo2a2talvBXC+1bbuvs/d97r73oE1/DpvIcTy0tbcZmYAvg3gsLt/fYH0BIBHmj8/AuBHyz89IcS1spiqsN8A8IcAXjOzV5u3fRnAVwH8g5l9HsApAA9/MFMUQlwLbc3t7j8Hb7f4ifczWBaOAW+d7Y3V+IJ9ADAelHUGFXfIklwdABrxOmwYmeKL9h2Z5YN+dAvPuXszfBE7AHjhp09SzerTVLvtnvi51ed28Dlt4KWb3Zkgi83EH7NGXhqm2ttBWWwlx7cDgHOVLVQ7O8MXUhyr8b9ntRF3FM0H3W5nS/wajUaw31qNn5sAkMuRks9wq/+PrlATIlFkbiESReYWIlFkbiESReYWIlFkbiESZUW7n5ZmLuLwP/9NS+2c8a6VAHDGeW51bniEap/6xG9RrZHNh2POZXiM8fKZoNvoeh4Rfex6HkkBwGbjZZRHf/IK1ab385gHAC4N8i6m6++6lWrDB1+g2u3bbw7H3Lidd3LN5XkJZTU/FO53rMLLfyero3y/NR4l1ixeFLIWBFBzZR7jNho8Sqw14iisQsZsqPupEL/cyNxCJIrMLUSiyNxCJIrMLUSiyNxCJMqKRmHr1xbxyL/d01L77j/Fi/I9+088krlpzx1Uq5TGqZbJxA/fsvy5r9rHK6km1/GqsDcLcfy27VfvotpEjsddU+Nnw/1OjR6m2qv/8xDVsjkeB568EI+5YYRXfm28jsdvU71xxFbK8OizVH6batUS70SaLfKqLwCYb2vQGg8irWIQ+TUqcaSVy7WOf80W95qsV24hEkXmFiJRZG4hEkXmFiJRZG4hEkXmFiJRVjQKq1fruDzcemGCj+z81XDbrWvOUO3kUa7t3nUv1Syo+gKAbBBblfp5dHKsyquEzsZD4vU6j7tuuoFXm/X0xA0mb9uxjWoXTp2mWn1wPdUuBzEPAJTyPC6MFmEs9/KF9QDgYvBQS1leVefgTTbr9bhCq1zmCyK68wUGo7X+sm3OPzibk6rChPilRuYWIlFkbiESReYWIlFkbiESReYWIlFkbiESpW3ObWY7AHwHwGbMB2z73P0bZvYVAH8M4ELzrl92d76KHYCJ2TJ+fKB1Sd6Zmbj76Zq1PKd158FnV50/xI29feGYl6Z4zr25j2fOjSrPIfMWl3wWgoX3qpkJquW28wXwAOB0g2ex+e3bqTbB17jDZJtsuC/oLrtmgJfFvnOGl+kCAILun5YJcm7jJZ+1Kj8+AGBZfo7lc8F8gn1m28XcaH1822z2LyzmIpYagD9395fNrB/AS2b2dFP7a3f/q0WOJYRYQRazhO85AOeaP0+Z2WEA/GVUCLEqeF+fuc1sCMCdAJ5v3vQFMztoZo+ZWcv3hWb2qJntN7P90zPxWx8hxPKxaHObWR+A7wP4ortPAvgmgN0A7sD8K/vXWm3n7vvcfa+77+3rja9FFkIsH4syt5nlMW/s77r7DwDA3Ufdve7uDQDfAnD3BzdNIcT7pa25bb4z3LcBHHb3ry+4feuCu30WAO+yJ4RYcRbzbflvAPhDAK+Z2avN274M4HNmdgfm47GTAP6k3Y5qVsTF3O6WWqUQP8/kKrzkbv0Az2uqE3y7jds2hWNmhniHTtSmqOQ1HhFZPS7X8xr/XqIeaCNt9htVCXZ3raHa9DR/LJemeBdSAMhPjFGtaBeoBotLPrPBon0DAzzenC1NUq08xWMyAECJR5T5PA+nokX7rE3pZn9/67gwE3RiXchivi3/OVpHa2GmLYToLLpCTYhEkbmFSBSZW4hEkbmFSBSZW4hEWdHup44Cao3WFUiNoDMlAOQLvConm+ER0cUxHn+cOX8wHLMcLPBWa/AYo9bgUU0miEYAwJwfh0yO7/fsaBxLbd3CY79fueUGqs2V+PP/5NSJcMz1PfyxXLeZlydkcv3hfk8P8263pRI/T3qLQTfbmTheMtqJFCgEXWC7e3iVWjYXv7Zm8q077GYyrRcIfM/9FnUvIcSHDplbiESRuYVIFJlbiESRuYVIFJlbiERZ0SisVCrhjTePtBaz8VQuTfFIqxJo3cFia8U2C7H1kCgCALzAI456EHHk28Qf5Tm+QN5koHmbRflGRlsvwAgAu3bzeC6T4/FRm/6IyIBv25Pnf+8NWwbD/Q4Pn6SaNYKV94JV+Qptzr9ancd6XQW+4GF3kf9din38HAKAybnWEa+bFgIU4pcamVuIRJG5hUgUmVuIRJG5hUgUmVuIRJG5hUiUlS35bNRRmZluqfWs2xBuOzXDM96c87y6VOH5ZKGX59gAkA/K9eZq/HkxKjtEMT7kvcECg+Uaz42rjbhkNhM8jdeChQuj8sJaI75OYLbBB50r8ONwudr6HLlCOeh+WqnzLLuri/+9K7VgxUMA5Rl+fKdn+bnZ3cfHnJnh12cAQIX8XTwoN16IXrmFSBSZW4hEkbmFSBSZW4hEkbmFSBSZW4hEMW/TjXNZBzO7AODUgps2ALi4YhNoj+YTs9rmA6y+Oa3EfHa6+8Z2d1pRc79ncLP97r63YxO4Cs0nZrXNB1h9c1pN89HbciESReYWIlE6be59HR7/ajSfmNU2H2D1zWnVzKejn7mFEB8cnX7lFkJ8QHTE3Gb2oJm9aWZvmdmXOjGHq+Zz0sxeM7NXzWx/h+bwmJmdN7NDC24bNLOnzexY8/91HZ7PV8zsTPM4vWpmn17B+ewws5+Z2Rtm9rqZ/Vnz9o4co2A+HTtG75njSr8tN7MsgKMAPglgGMCLAD7n7m+s6ETePaeTAPa6e8fyUjO7H8A0gO+4+63N2/4TgHF3/2rzSXCdu/9FB+fzFQDT7v5XKzGHq+azFcBWd3/ZzPoBvATgMwD+CB04RsF8HkaHjtHVdOKV+24Ab7n7cXevAPg7AA91YB6rCnd/FsD4VTc/BODx5s+PY/7k6eR8Ooa7n3P3l5s/TwE4DGAbOnSMgvmsGjph7m0A3lnw+zA6f1AcwE/M7CUze7TDc1nIZnc/1/x5BMDmTk6myRfM7GDzbfuKfUxYiJkNAbgTwPNYBcfoqvkAq+AYAfpC7Qr3ufuvAfhtAH/afEu6qvD5z0+djja+CWA3gDsAnAPwtZWegJn1Afg+gC+6+7tamXTiGLWYT8eP0RU6Ye4zAHYs+H1787aO4e5nmv+fB/BDzH90WA2MNj/bXfmMd76Tk3H3UXevu3sDwLewwsfJzPKYN9J33f0HzZs7doxazafTx2ghnTD3iwBuMrMbzKwA4PcBPNGBeQAAzKy3+YUIzKwXwKcAHIq3WjGeAPBI8+dHAPyog3O5Yp4rfBYreJzMzAB8G8Bhd//6Aqkjx4jNp5PH6D24+4r/A/BpzH9j/jaA/9CJOSyYyy4AB5r/Xu/UfAB8D/Nv46qY/x7i8wDWA3gGwDEAPwUw2OH5/FcArwE4iHlTbV3B+dyH+bfcBwG82vz36U4do2A+HTtGV//TFWpCJIq+UBMiUWRuIRJF5hYiUWRuIRJF5hYiUWRuIRJF5hYiUWRuIRLl/wGDW+fdISqN/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "img = cv.imread('Final_Training/Images/00000/00000_00000.ppm', cv.IMREAD_COLOR)\n",
    "img = cv.cvtColor(img, cv.COLOR_BGR2RGB)\n",
    "print('Dimensions of the image:', img.shape)\n",
    "#plt.figure(num=None, figsize=(1, 1))\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions of the rescaled image: (53, 53, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f09bd1d81d0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztfWuoZed53vOuy76e61w9mlEspTZJXWicoroJCdR16uK6IfYPE5KGooJAf1pwSEpst1Ca0oLzJxdoSRE4RIU0dtIk2Ji0qeo6lECwLcdOYltxJLuyJc1obue6z957Xd/+OFua/T7fmnO2pDP7jL3eB4Y571prr/Wty7fXft7L84qqwuFwtAvRaQ/A4XAsHz7xHY4Wwie+w9FC+MR3OFoIn/gORwvhE9/haCF84jscLYRPfIejhXhDE19E3iMiXxeR50Tkwyc1KIfDcW8hrzdzT0RiAH8N4N0AXgTwBQA/rapfu9tnOp2uDgaDV23F0ccWEWNHZB8us3ZMdhLFdj3ZEQ3BrgViPiYdUBvGlPZ7xu4OhnYXcWJsiez3L9+SurY2X7WirIw9mWbBmPLCblPW1hZ6BdBlQicRsu0H0jgl254jAERCO6UTqSo7Ji0LY9eFtatsSrY975w+X5RlMCa+9nziNa2vYK9DSTerUnuzNLDt9o0zgB+A1zBHp9Mp8jwPH0pCeHcWxzsAPKeq3wQAEfk4gPcBuOvEHwwG+NG//85X7eBLh+yUHq5uNxzukJ6l1dQuOL+ybuzN3oqxe/ZZwyrNgE3an3TtAz4lGwAe+Jt/y9hv+Tt/1x5zfdPY6XDV2PT8IivIpi+Ca7d2jP2Vv3o+GNNLN24b+/b+yNjJwF77/rp9dr7nXMfaZ/vGfmDjkrHPr54LxtDv2PNEaY853t839vTGdWtffdHYe88/Z+ydbz5r7G9fv2rs6ztbwZgieh6U7Gnf2jv0RXCrzI09yifGzsguK/vlU/K3OoCavqCUvqT5TTB/p77w+S8E+2vCG/mpfxnAC3P2i7NldlAij4vI0yLydJ6HbyKHw7F83HPnnqo+oaqPqOojnU73Xh/O4XAsgDfyU/8lAA/O2Vdmy+4OAeK5n0oB3yGbOX0TcenE9rtrQD/NVzv2FFeJq9bEE3Piywe5/Snfi9eM3d3YCMbUG1h6kfQHxpbEjokZnNBd0Yp+Eh/YMe7vWS4wGRE3AKCFPe9hYv0O3TSn9dYeyNjYMeyvN1X7k7isrQ0ApVp6EIm9tlFqXwxpn6jBiqUP08EtOya6zglRwyQJubKI/dlc0TMotX2+YvICJWSn5Mco6d1aE5WMGli+0DMN0SPXz69mv9jd8Ebe+F8A8FYReVhEOgB+CsCn3sD+HA7HkvC63/iqWorIvwTwRzh0hv+Gqn71xEbmcDjuGd7IT32o6h8C+MMTGovD4VgS3tDEf60QCJL4DgcKOD6FKaKI4/jhPvuJ5VQbXRt2Ot+zPHKDOP7Wjg1rTfdt+KVQ+nzXcvxLaw8EY1rdtKGt3hr5AWKKQXKqAEUItbTXZWfX8u+b162fYve2tQFACwpTDq0fYrVrQ2mrHbuPtdiGDDtq19dqcxemtfUhAEBUWd7fiSynT3r2M8mafTx7pV2f7+wZe/+WDd919q5ZexTy6Zrj7BU9gyXlgST25rC7ugCF68hho8TXlRNJEPqyIuL0ge9rzl4Gx3c4HN+h8InvcLQQPvEdjhZiqRwf0IDHzyOidMg44Pjh91Q/sbHhja6N5Q4oPTvJbDxaR5YnJjS88xcsX7/40MPGvvDWtwZjWr940dhCfoiaeBiF6VHUdsG0sCcxmdrU07raNvZaSK+x0rMx8cs0xgTWN6KUZTnasym/N/YPjJ1T6molTTFzWtCx9zMhdpv07JjSs3b73o5NfR6cuUjrbZy/t2fHDACR2ilQiT1mlNiciQL2AZnWVA8Aa5ecJ0CfzzkdF2FuAV9JpXmwGKu38De+w9FC+MR3OFoIn/gORwuxVI6vamuumdMnMXN8qp3nem6EHH8ttVy2M7VxehxYWw9s/Lo3sJ9/8PIVYz9AnH7zrW8JxhSvE8mmr1euRmaWl5XM8S3fznLKUY9sjP3cZnhbHzhnz+ttD18wdr5nr+3Oy9YX8uy2jYnf2LWjrumQUS9knp2UODzfTtZS6NgNOn17IftnbX7E8JzNqRjetr6P/k6Y35ASR6+VNAGIszMnnypxfKE4PsXpc9IcgIYaATXXCxDLF5oHah6wxWr3/Y3vcLQQPvEdjhbCJ77D0UL4xHc4WoilJ/BUcwk8XHQTKQk60vB6Eurb9SjxpFtbGxPSLxvbIpwhCXesrNtCkjNnz9j15PyLsobvTtbB6JNNSUJCzryosGOMi11jd9Um8Kx1rCNusHI2GNLZFRK9qO217pM2YfrQQ3bIQ3sSm6Rfl8cksmGHfLiMHF8x6c/1B/Z+d9gLSolM/Z4tmDp35XvtmEZ2+2gcOhx3rn3D2Pu71nGqkXWsDrqURETPwwp5ObdIba6qbaLTOA8TeAL3HD1inAL3euRy/Y3vcLQQPvEdjhbCJ77D0UIsN4EHVviANBCC5hYdYi+DhgSeAYkd9irL4+Kp5VRCzSZWelZKYWPFcvzh0CbjdLpWcEI0/O5kcUxQ0Q04Z4OaXYAKiSSzhUQdtUlIcWKdChu2TgkAsMLuEWpOERF37a9Y7nq+JJ+A2Ot0e9cWwGR79roDQF3b86hybpBhH8eMLlRFhUNC9zpKrX+n37X3bqVL4p0ARiRyomOb0AWxdkftMQZD+zx0aAygop8DSsZqEtvkIi4NlFqoActcoltTg5cm+Bvf4WghfOI7HC2ET3yHo4VYbhxfFWqKEoiPU2y5S0KFK0kY8xyI5ZI9EoGMKrs+Ir9CSgIVw76NDdckvlhQHLbTDTmasvjCAeUSHNMcsjiwfLk6sHH9tOKmDnYMcRE21KinlqvmSoUeIxKcKKkB5dj6HTLKh4jHdvtOHhafZKObxt6iOP0OF8Tk9pgHY+r3l1ADDmrIsf+CLSwaX2vo9zK1BU4DLrqZ2nuhFfkZYsvxe0NbOLRJz9N2avffiex1BICC7mdFnD5Ke2TfceAETUDvAn/jOxwthE98h6OF8InvcLQQS8/Vr+dkJ1g8s0sNLld6loeeGYa5+v2IuGROcdjCcn4pOV+cxD+ouWR1QNz29g1rR2FuQUFFCBkLUVYkyEjrxwc23j0e28T3jNYrndMktvnmAJBElhemibVL8gsUmeWyJcXQi4LOifLo66JBYKJg3wZtQ9dhkln+O5pSfgPFzGPqxlxu2+tW7Vo+DwBpRkUFas+zDkRUSFiDrlPVsecoVH8Qkd8qisIpKJTgIsE8Id9G906dRJMgbRP8je9wtBA+8R2OFsInvsPRQiw9V1/nOH6c2O+dlYHl8GfWbI33BaqVB4DuvuV905Hl+NxAI6X4cnRgeWWxY7evtywvHFOu/+i5rwdj2iM+vE92znw6t2PI8ynZxK/JrigPgHMPAECpXoBL3WvKLaioVr7k9TXl0dMx66bGKTQGYVFJ5ZwJal7B62n3BeWpJySMmVah36EDjqvbY3QpFz+mjqY5aUiUlPtfFtRAg7ZX7pAKADTOmPJdBhTH7/fuFGfEHsd3OBx3g098h6OFOHbii8hviMgNEfnK3LIzIvKUiDw7+3/zqH04HI77C4tw/N8E8J8A/Ne5ZR8G8BlV/aiIfHhmf+i4HcVJgrX1c6/aK6R31+va4fQ7trB82LHcBgDKwnLw/V2rBTfMLIeLqVa+pBz0yS0bA98ivqWJHeO0DPPiJ8SHc4pXV8RdlfhxRMdMaX9C3LWumX83cHwWPzhGqY3rurlKoiLeWVN8WpPwnRJxzgONs6K6iooEGgb0vHB7DLa5NL3pLSd0ZnVM14WOWdK9mh6Qv4b8GCXd+yndy6KhaWZN9yomvwA/H9FcXgj7Te6GY9/4qvp/AWzR4vcBeHL295MA3r/Q0RwOx32B1+vVv6iqr5Q+vQzg4t02FJHHATwOAP1BgzSMw+FYOt6wc08Pcxjv+vtCVZ9Q1UdU9ZEuyVw5HI7Twet9418XkUuqek1ELgG4cewnACRJijNzjQ0H9H3RJcXwDtVbdyPSMwOQUVx9f8c2SuyQJl9f7SmXE1tvfTCxTHFyy9aQB8SxQeNMKV86ITvlWCtrqPH+yGa2HrD3hq9zrtPmlO4oTo6068TGm6vY2kr3StLwSz6mbWrKBZiSZoB26MyH9v5PqCZiTH6HkuyCtesAFMzxOU+erm45sj6h8dg+bzlx/Io5PvmE8obcgogad8Z8s6imAfO9HU6K498FnwLw6OzvRwF88nXux+FwnAIWCef9NoA/BfB9IvKiiDwG4KMA3i0izwL4hzPb4XB8h+DYn/qq+tN3WfVjJzwWh8OxJCw1Vz9JUlyY4/gclV8hftKNKc+ZY6wAkjO2T9wqaabFFGYvqcBauR8f506HZNiujsN6fK6xZr6c0PqU9pEGPgKK45IdE3VNuEEBgCiIxB9d8y30Y7BOLL+uicNrYusqqgZ/TEHa/FPS1IsmNmo8qkdk2/r87am9uVuk009p8yga6G+hR9cYJLReC65hoPp84uxZbf0WB6RjUDdwfBG7z5r8BuOpvQ7FnB+hasgLaIKn7DocLYRPfIejhfCJ73C0EEvl+Gmc4sLZN92xaf2AcpK7leUyVWU5IQAkZy4Yu7NiswOrkY3zH2Scm02xZeL4rGnOHD9q4vgxacFRDLxPMfA+cf44Jh114vQ94f4DlP8Q8Hkgqi3/VcqL52QAoXtRcwyeasJBdRWlhHH8cUYx8am9ThHpJ5ZTa++T5t5t0jG4Qf0HCsqrL7mHIcLceub47FaKyA/F7hTOvZ+QHyOjGH3QQBKA0k5ZXmEa6DvMc/wGHYQG+Bvf4WghfOI7HC2ET3yHo4Xwie9wtBBLTuBJcHbzzF0P3qMiiqiwTgylBoYA0Fk9Z+yYkh+uX33Z2Fu5bUZRkXMvo0SWSX206GSdhUIcQs6ZmBKRVshjtEoCj0Mpj7Zh7S41ckymLEkB1Jz0QTY7qRKq9CkpCYmLdOKBFUKNemEJtpBDMCNH2N7EJrvskCjpXmXv1ZjudUY5Q9yopKwamomy+CWdt9J6FgxlEZWCnHc5lVix6625AQYlDfFaWqDz2y9Wo+NvfIejjfCJ73C0ED7xHY4WYqkcP4oirKzc4YKcHBGUdeSW01Vxn7eA0ilMiUTtxpYP36Lkl6wmAUdKqBiVLAhpeSInxgBATBwtoQSbjAh1FlleeKA26aPPdm3tTkHFLuMGjk/NHQsWIQUX/th3AotYcGJT3LX3KumFwqhp394/FiyZ5JT8Qny7EHuvuBHoSp/uZWGLfLIiTGyiQ7L+Z6BrEYiO0vZUh4SKbKXrKE1CLszhaUEgtCJ3+fsI+Bvf4WghfOI7HC2ET3yHo4VYKscXiZCmd5h82DSRCiDEsv40DhsMbu9brnp7z3LZ7cLGl/fVcv4xcd1xMSWbhRksy1tpKIjpUXy6T7kAfeL4KXH8urC5BpPSClRkpRV4jIjjCxNXICSGJEAigdgmCVdy00ziy3Vu+bTsh8Ui1JsCCRVERUI+nHjVfj5ZM/aZNetXOL86NPYOOXx2GnIu9qiJ6jgnnw6dBhfBsPCFUiBdIibdTPrDwDv7ARjsF2ARlUXgb3yHo4Xwie9wtBA+8R2OFmKpHB+oUeNO/nRE8e6K+HdZUt50EYpeFMRv6o61pyRqsU/8aExahxOigXlljzkk+RAWxQCANYr1r1PNQUSikXwduon9PPefrIifJ33LhYeb1q8BAP2VdWP31jaMXVMz0JIbcBDZVWoUMRlZv0Q2spwfACpqXiLErxMKeufkb1HaPk1tPUCnZzl+ofbe5VFYP3AgNp9fyeZnsiJOXvPtJ04vx3B6FtI83IZtiuOzb0wWE9+Yh7/xHY4Wwie+w9FC+MR3OFqIpXJ8RY1qrklCTTFQqS2/KigenR2EfKjuWq6arll+W+5S7j3F3Q8oxh6EwCmXIKZLNmjgV5tq+e45qgvP8x07RhIR3diwfLw/sPHtrLJj6q/Yc750+UowpvNXHjT2mSt2m5xyJA6IR3Y4x4Jq5beuXTf2NtkAsE/Lim2bnxBP7HUbja1dTm7Z7SPr20giO6Zux+by9ygvAABisf6WsPad8hV4BxxD556qtEC54UVTyF6O5vQs0Knzjgavx3c4HHeDT3yHo4Xwie9wtBBL5fhlPsXWi8++akfEj1mbbjSy3Hh728bDAaC/ednYnY03GbukmDkoL56/+VLhBpb2EvWJb6Vl2PQwoaLshDTxotjadcfGq7sD67fob1h7c902EVm/cMnY5x+0fB4A1s5bbcLeuvUjdCn/oUcXRqnjZEXOkP7AjqnaeHMwhu7mbbvNLauHKDvfNnZn29q6bX0h09Jet2KXfCfdM9buhE0+Ymru2SH9QxVK7BBeT6s5jz44IvPzkJRzfn9EORUJ5VzEc+vD2oBm+Bvf4WghfOI7HC3EsRNfRB4Ukc+KyNdE5Ksi8sHZ8jMi8pSIPDv7f/PeD9fhcJwEFuH4JYCfV9U/E5FVAF8UkacA/HMAn1HVj4rIhwF8GMCHjtpRXeYY3/zWnQWBmLvlP1u7lp9fuxnq6p+hmOdG3/K4srZ8mvOaOfs/pu/CIdWpDyuKbzeUvie0TUINKqOe9QtI327fHVI9wLqtQz/74MPG3rxi7bXLlvMDQHfN5rFL1x4j7Vhu2CF7MiHdQLJr4tMYhrXvnc6uXdC35xXTdUBq73cV3TT27pbl+NkWbV/a/AatQj6d9uz97VHD05qapCo9MWFc35pRkKt/tI8ACJukMsfvpFSrMde4lT97Nxz7xlfVa6r6Z7O/9wE8A+AygPcBeHK22ZMA3r/QER0Ox6njNXF8EXkIwA8C+ByAi6p6bbbqZQAX7/KZx0XkaRF5ekRtjB0Ox+lg4YkvIisAfg/Az6ranFQ9zClsTBZU1SdU9RFVfWRlGMpjOxyO5WOhOL6IpDic9L+lqr8/W3xdRC6p6jURuQTgxnH76SYR3nzuzuTfHVtOdu22jcPuEccfT0Pe2KMecN3M2iXXLhOpjylum5DfYZWu0Hpk+fm6hr9iVirLPXtUHxBTHXk8tDnk3RXLfZOh5Z15an0GNzN76a9es9cRAHoH9pibG5b/9nrWN9KlmPe1PXue10fWd3IzsxdqdzfMb6hfsLn5gy2be38ht+fVp1z89Y2HjC253Z+M7HlX1FOhQOgjqhLi9HS/q5IeGNIJ1Jh9APT+C4T6ieNXocZERLH4mOP6C+j2HYdFvPoC4GMAnlHVX55b9SkAj87+fhTAJ1/z0R0Ox6lgkTf+jwD4ZwD+UkS+PFv2rwF8FMDviMhjAL4F4CfvzRAdDsdJ49iJr6p/grs35vmxkx2Ow+FYBpaaq58kES7Ocfz6tuWJL1y39nR6PMcfZJZP56RvV1PclfXiU8oD6BHnX00sT1yh2vkuKDYNICUuGRPH7w1snnxn0+Y+6arltiXFmvcr0rfbs33ld7fCmobBnq1NLzJ7zHXyK6wO7Bh2dqyG3nXi/Nemlitvh24G1LftuFd27T5V7f290LG5BxfWrV2N7XXJulTzQPtLSScBAFYiyiWgGHlNOo8RWOPRbh/o7pd2DDX1J5AoZNtBHJ9q/jkfv6n/3nHwlF2Ho4Xwie9wtBA+8R2OFsInvsPRQizVuRcnEVbO33Hu9SvrIEp61kGkYp1kGTnyAKAgZ15VUQEMBSQSSriQxK4fUi7EWmr316usQ6qcWjEJAChyO85KrEOns25FMVYu2yKb7dg6hPbEnmO+d83Y09w6rcZT6+wDgFUq/OlMrLhHfNZmXPfIxj41Admz5xiNqAHmXtjgNKYGKdxM4taUxD0Sm3R0ae2sPca+vS7asYlMdWY9jFEZPj/rkU1UGqb0DJLzLqbnqaDEp4Kev4wSdip+/hoaXio562rhhB529s3ZJ1Wk43A4vvvgE9/haCF84jscLcRSOX4NxXguwSFnYULiUxGJCrIgQeMxiDdGJKTBIgZxasfAWhBDSujpUNOPbBomhSjxwE7fctXBWcvxVx/4HmNvU5JQVhBnn9jrkFJjx5UsLByKJ5YPH7xs+fAtKrrJSdTi9oE9xpRyhIax5d8DSr4BgO4FmzSU9S3nvn2VfDr0PCTr1i/RGdn9dUhANN6y100OwsSmNLPLOonl7Cv0zFUkzMFNWBukM4+2G5/pQMHTmpF9psX4DZzjOxyOu8AnvsPRQvjEdzhaiKVy/KJS3JiL/+5nJNaQWP6UdG1MlcUhACAivlOTemFCnE0iqwIUp5YP92lIA1BeABVZHExDPh0NiNOv24KX1QvnrU0NLOPMcnyZWu47nFo+3R1Zocu4EzaHnO7ahpX7t619cN3y7atqRS4mpb3OZUQx9jfbhhoXLodKbBtnbKOPW9vWP7K1a3Miyh49D+v23vUm9jyHO5bzpyMrzontsMGpjCiXBNavMKRipYzi8Lv0PJTUYIWLdtgHIFEoxBFuRByfGpzGc/Nm0YIdf+M7HC2ET3yHo4Xwie9wtBBL5fhVXWN7/w6XPBjbnPSKRANj4lNNHD8h9UzqiYk++QmijuWmYldjUFhO1qU8eRbvrDXkjaB8784aNahcsbyxP7DCl2f71pbSnlQyoTz5HjWz6NrPA0BOJ5pl9tp3V+x1WV2xuQYv36R6gAN73mnPxthXVuw5A0B/YMVAOhMSLEnttVUSNs2Vzrtvue7KOev7iK+TcEfD465Kee8UB+fcEV5f072pK26Kyc8Hi2g0vXsDkm/X0jNYzeWuLCq76W98h6OF8InvcLQQPvEdjhZiqRy/rBTbczXUBfHMMsizt/y917UcEQASzlumxog99hMQ/0Zqx9Cn+oAO5QUwY2tg+EFDymTNCjqmA8s9ez0bnz6XWrujdn9F1445Ty2/ztLQF1IWlotO920e++qmzS248MDfMPZW+bwdw9TG+aOuPcdOP/QzxEEtBvHpmPwnYs8zI/2Gftfe28GmvQ4RXdeyieNzhxU5ul6ee1dUxOlZTJP5OKOp/uQ4vwDXo5incMHmGv7GdzhaCJ/4DkcL4RPf4WghlhzHB3bnQrdaUZ59TXn0lJM86FGQHkBKTQ9jbmgwtg0vysTG5QfUdGFI63s1NeGM7PZBk0QAmthxC3H6iOrzE/JdpBRbTolHWkUAYJ/yw1/aHoOxe2B5YtaxufRZz8bA857l7Dld5yk1Cdmf2Ou0Nw717XqrNt8gon2C8tYrujdTakaqZXaknakdY9WQxs76dpzrHua+kx+iLsmmZzTI82C7gZPL0XH8QFfPG2o4HI5F4BPf4WghfOI7HC3EcjX3KmA0zzWJy0Y1aZrXdnhxHHJ81JbfVMfo7IPi+mvEn/qpPUaXtNgzyh+PmugV66KHG9h9kJ1SnDamfPCMNAC296zW/0u3Q1396dRe637H1s8rcXzt2Vz7umtzA0qKwY8K63nYOQg5fj+zV2Ja2n2UlDNREf+t1B6jqKhBKnH8ivh3oyIev/oab+j8LojjU9w+4Phgm8fQMKZjKb2SPXddF6T7/sZ3OFoIn/gORwtx7MQXkZ6IfF5E/lxEvioivzhb/rCIfE5EnhORT4hwgavD4bhfsQjHzwC8S1VHIpIC+BMR+R8Afg7Ar6jqx0XkvwB4DMCvH7WjugYm4zskRInLakkxeLbzkC1HheWSQjp+Qrr4fdLRj4aWu3ZSiqGT5l6HCv57DZxQpnZMk1u3jV3u2J5uOLB16UlMufa5Peb2Nds77/pVq5+3e4P2DyCh/P/+Gcvxez3SqO/avPeUNAPivv2enxBVvdnA8fMbNqfiYGR9ESPyZdCtQZpSj4Oc9O3IH5OQfydu4NMREWi2VY/JxWcfEnF8iY7h9EHMPswd4HT+mMsLzD5OKFdfD/GK9yid/VMA7wLw32fLnwTw/oWO6HA4Th0LcXwRiUXkywBuAHgKwDcA7Ki+2sLlRQCX7/LZx0XkaRF5ejIJM8ocDsfysdDEV9VKVd8O4AqAdwD4/kUPoKpPqOojqvpIn1JVHQ7H6eA1efVVdQfAZwH8MIANEXnFR3AFwEsnPDaHw3GPcKxzT0TOAyhUdUdE+gDeDeCXcPgF8AEAHwfwKIBPHr+vCGlyp2CFRQqUHHFJQuvThlQYysAIiipILLMbk5gmCTyy8w6VLRSJanbuhd+dSU4iojvbxq73do6098nJdXPXJuhce/7/GXvrJu0/Dx08ySo5LWnYSgUtU3KUKT0p8cAWIo2pdKg4sGMCgB1yzpaUbKUd6zBMSZwzTUnco7DUsaREJuT2HGLlhB5AKcmHE75qslkkQ8iZFtqE4xcgiti5d/QzPm8uWq6ziFf/EoAnRSTG4S+E31HVT4vI1wB8XET+A4AvAfjYgsd0OBynjGMnvqr+BYAfbFj+TRzyfYfD8R0Gz9xzOFqIpRbpxHGCjfW5YhCm48d9DzXkJgQJGCyUQAKNHbW8coXyDRPinSU1sKxJtLIn4SVMSRhD923iio4spy/3bhn72nWboPPs1avG/vZLLxh7f2p9Cr11K5wJAL3IZsNEaseU0T648KeEvS7RwJ73AfHtvCGBp8xtE8uUBEtWVq1gyXDdcvpuaguHpplNjBrdtNe1GtsxRTVLmABVYZ+PIrcPREVj5Gc2TADiZ5gLaki8s+GRD/bZ4Ee6+6C8aabD4bgLfOI7HC2ET3yHo4VYKsdPkxTnz186Youj+UmTsCULI2jNTQwth0NleV+W2wKZvcJy3aRnsw3rIRV+TEJxkCqzfHi0bxtO3rxqOXpN1SgsMrnRtce8YHUwkVB+QxlRPBtAXVg+nI2pEUlC50nFlh0SKHnTecu3i8q+Q6qqIb+hpgYYVDi0MbSFQaukjplt2+uye8Peu9vE8fPSbh8Pia8DiNasX6Ee2DFNqJBsWh3d7IL5+bHSmg0dWUiPJHjuF0gFOBb+xnc4Wgif+A5HC+ET3+FoIZbK8ZPUcvwgV58IEHMbFkUAgPKYXOuSmnSUmeWuoy0bbx5RYHVIzTCYp0oZEqzyto3LH+zavPVrL33bHlNsfHntihXJuLRp+XdMOes9ovTX9xvIad/oAAATE0lEQVRi6JXlv1lGag5je51UrN9h84x1LKyt2+YYoHyGNA759FrSJXtItuX4o6s21+Da87YObPu69Z1s3abmKV37vMSrYdPViM6r7Npre7Br/QgTEv8Ie11QXn3wjFOeScMzzWKaTPrZjzA/huOadL66j4W2cjgc31Xwie9wtBA+8R2OFmKpHB84WhhQOagZ8JWGoCfxISXqylH2Ke1jh3wCGeXqJ5Xd/yp9/twqKUIC6JeWL6eF5ZZ5xnF9yoPv2lFvdG2DywfO2eYX5688aOw3a6h0pB0Sz+zbfUQdiuuTTkGvZy9sl7ZXKthP0MDxKTcgHds4e3n9hrEnL1hfyP7Vb9ox5tZvsblJPgI6h1E4JNyu7P0bjUnYNGOdAhKApdp5ocafSWzfrUVQ39+Um0LHCOaJhRy18i7wN77D0UL4xHc4Wgif+A5HC7FUji8A4rkGfzXXzpPuG3P+qoHj18TxeZ8Vrc/JPqBj7pc2pl5S7n5Ogdu1TngJB5QT3sssx58cUCOJHRt/7t+yPLE3tPbFc6vGHp49Z+x6+KZgTNHA5gbEPWtTf1LUQXNQ0h4k74lSw9O4Cq9LPye+m9t8h/FNW69/cP1b1r79nN0h5c33hjYvYBzZMRRCDiAA23R/t6nx5oSamXDcPaZjhJoSfEz7eeXEfBw2nrEb0We4aeb8upNqqOFwOL774BPf4WghfOI7HC3Ekjm+ojfHqQsiMxXl3RfUoHBah7XvzGlqoky1MlflpoXWLIm7HuS2nj+mMdxo1EOzvDFdo1h/ZHlkPLZ+he3btglmllmfgFY2F//CxJ7jxpvCxsWddesXEOK22kuOtBOhngdBs0fSqitDPp3dstdy/5r1bWzdtJx//8Dahdiah4pyB+rKcvw97RxpA8CI7ifLK5DEYpCLH1GNgtIDVZfkcyL+HvB5NDSTDfpP8Jhee0G+v/EdjhbCJ77D0UL4xHc4Woilcnytckx37uRfZ8RlDsgeEQE6aOD4BfWZq8gvMBzavHmJiS8RV+UjZHTMXeb8VdiPjaXVk4Tyt/uWa/Zim1tf7FtOn21bLrz9LVuXHk/sAeOdkF+nG9avUA3sMaINy4/jDarPJz+DlPY6sOZ9XIe17/mO9StkuzbXfjq2te8Z9SfgvPgpcducuPCI8urHDfdqUlOeBtk1xd2j2B5T6WbXNIaSns+aNSEb6ufrmvNbrB2B6wPurF8wVd/f+A5HG+ET3+FoIXziOxwtxFI5fpGNcP35P33VnlIe8z4NZ48ClnsNec2jEfHCqY2JP/Tmh4195oytQw/qBSguXxCvzCj+nY0s/waAKrXnpX3Ldy92bS7/OeoDL2LX677l0xOKf7983fL36fOWOwMAht+w2wysZl7nos3371+2/ff2b9j+ffnulrEvrNrPb66QJh+AXn/T2JxLEORYRNbPUEX28znlFtgnAZiofRamVQZGWduciJo+U7PXh+rtg9oQ8ksVFfkMiOM36UjyPrhmRY4Q0m/yGTTB3/gORwvhE9/haCEWnvgiEovIl0Tk0zP7YRH5nIg8JyKfEJEwH9LhcNyXeC1v/A8CeGbO/iUAv6KqbwGwDeCxkxyYw+G4d1jIuSciVwD8EwD/EcDPyWFVwLsA/NPZJk8C+HcAfv2o/XQ7gu+9fOe7ZntKYovUvODWrm1weWM3bAapyuKGtiCmruxnqtImqgTCBZx9Q0kiJa1moQ8AGFOSx4jEOgbk3OumlNAj1tkXUcPKyba9TmMq0sn27XUDgJq2mabWAShjK3QZ79iinsme3b6c2Ou617cFNWsrVvgSAFbWSeAzteeZkVMrS6xzL+9YB2Km1lmXg2xqmFqgodEI7aOmQjEuiEGDmIfZnp3FTWKacwiaYwCoaVkg1kHr54t0jnL8meMutBXwqwB+AXfkQ84C2FF9tfTtRQCXF9yXw+E4ZRw78UXkxwHcUNUvvp4DiMjjIvK0iDy9S2Eph8NxOljkp/6PAPgJEXkvgB6ANQC/BmBDRJLZW/8KgJeaPqyqTwB4AgC+7+ELi6YSOxyOe4hjJ76qfgTARwBARN4J4F+p6s+IyO8C+ACAjwN4FMAnj9vXsJ/gHT9wR+TxhVu2scT0G1aA4sVblnduvWwbLADA+oZtNrF2/gFjJxElZBDvYyHDoOkhJfAoNUjQbngJi57l7NO+9TvsppSgk1h7fWALXrrUtGMSW45f7li+fTC21xUACiouyqipRzay1z6/SoKORB25WOUW+QDSHXvvAGC4ZxtUDtcsZ19Zs/dymg7IJs6v9rxLsYVEJd3rsgg5fkXiqjX5GYTFNAO6fQyn5gIb2j6KQp9B2DzWImIxkPlndEFRjjcSx/8QDh19z+GQ83/sDezL4XAsEa8pZVdV/xjAH8/+/iaAd5z8kBwOx72GZ+45HC3EUot06qrGdPcO74oK+71zjgo9zq3ZGOtqLyyIqXPLgLbJb7AytLkCg8HR33Xc9FBisqkAp2poHHFAtO0WxYapLyN2KPbap3hzn0hed2iPmVIxS9wJBSe6tM06iYHkmb3W4wnlAnTs9kp2RdeJ/RYAUPfsGCYd67vIucCF/C/Tjr13E1JWPaDTnlLMvZTwXnERTpDXQWZFaplSUhFOxUU4dlBBSQ6rwzZsFQV+haPG6EU6DofjLvCJ73C0ED7xHY4WYqkcv6pq7GzdiUFnNeWsJzaPftizcd/Vvo1vA8D+geVYB5nlx2VGrIpy+zkmGkd2TDFx1zilBgoSctmSONkec1XisqlYXphS48Y+kbyzJNyxSly5k4QcP6HcgfU1m4tfjG3Mu7Nr/Sl13/LzsmvtKQuYxGF8uo7tGDK6dpzWXpLQRpHYY2T02uK80Jw4ftVwr2qh3HxuasmD4iYwJODJQhuBfCvT8yZKzjkTr6NhxnHwN77D0UL4xHc4Wgif+A5HC7FUjj/NSzxz9U7d9oRi4DuF5apbe5brdruW8wNASTHyxFJ+dCPiv8Tx+9TsouxY0jXJ7RhjtTwxIq4LAMICilRPnUTckJL9DHYMKfFQiL0umlLu9moohlTSMXeJP4M0AuoNW0+fU34D82fOgi8bhFFTOs8koXwE0lIA5WhMc2oKQveejyj8Xmvg+BBqHspxfbqXWnEXTd4hxeB5CETym9i7Bpx+8dq2Rb0B/sZ3OFoIn/gORwvhE9/haCGWyvHzCnhp7w4HmlJDwVFha6MPcvu9lHZs7BkA+sTziO4iBcXlK8qLp5xy7VIjxmrF2EWH+TrxUiCI9TLlj+j7lpsgxmSzD4B5XEWcsEjD26okHjcl3hjTdUio9n1aEd8uyaZ4dtXQKCKF5dMd8gOw/6WkY5RUO6/U4JK5MNe6J2no+4jomWPOz7n3wc0UrrdnHYPwOhwH4eA+1/AH4givPc7vb3yHo4Xwie9wtBA+8R2OFmK59fiSYBTdqbkvSbetqEnPLrb8KorDHPQOac5TyTYi3mdut++S5n3aJU37DvHOykasRcNGjODGiJTvzbQvkOZnHsk6cLS+oP2XC4R9+ZhJzP0JSPOeeGdG8epJYe9N0aBvl1CMPBH2l1DNv9p7U1fUsJKuc0QXNiXtBDQ0eyrKhGz7vJQl+2uY85NNh4wC/k26+wvE6JnTx5T/MG8vmtfvb3yHo4Xwie9wtBA+8R2OFmKpHF8Ro8Dmq3bJfIeJZ8Qcn+O2YV+5KKZ9EgWbHNiqbY7r18SfCsoTKJXGENRfh9rsAcfnMC3Hbbl/G/NIsrleIc/D68Q91VKqzx8OLP9d6VmOT6eECtSvr7Ra/1kWjqET2WVCPQo6MWshUP4CjZlaJqCmuD7H1NkGgISS6RM6ZkmcueL6/MA/c3TMXeh4bB9+BrQN3zt7r+Y5/rE6/68cY6GtHA7HdxV84jscLYRPfIejhVg6x691Y862xLEm3ihUhx43cHwhvbqIcquLgmLcudWLr/aszTHwkjXWiI/XDaJpvKwO6vOtGdRoK9eZ23OSyO4vL2wuwd7I9pADgIR0B1ZXbQ1Cmlg7WbccPy6Zq5Lvg1IsmvwMSWqXJR17jBXSEuxQ7zyhuoj9kfUr5DnpLbKvhfPuARClR4d0/Qruncg5GFwfwNoKdAD2rSQNdRUc+2dfSES+kPl+j87xHQ7HXeET3+FoIXziOxwthE98h6OFWKpzryxL3Lh1R2yTcyHYcZZTQ8KMRRMA1LyssJ/htg4scnEchJw1IFuloXGEHG3zty03QayDJg0kckF2UdlzzovQ4VhxBs7IOsKG1Fy0DPyRlHhC513TdeV7ewhyUgkn8Fh70LOJKt2edfaV1Hhkb4+coEEzi9C5FxZAkUAJOctqfh7AhUJUeBRTERgLjDY49+Lk6Mat7ICev9aLynL6G9/haCEWeuOLyPMA9nH49Vaq6iMicgbAJwA8BOB5AD+pqtv3ZpgOh+Mk8Vre+P9AVd+uqo/M7A8D+IyqvhXAZ2a2w+H4DsAb4fjvA/DO2d9PAvhjAB866gNlWeLmHMcP+DNxmSy3/H08DcUdSkraqOkzKfHrJLCpAIIEGjskQhlRU02Nw0tYU+FFzadJCRqcsFGRb6Mo2NdB6zlRJQ79GJxUVFWWH08z8iMEvR7ZUcG+DUp0eR0cP6Z9skjKcGgTePZGPIZjOH1DQRWLmhzP8bloh/bHohnR0Qk9LIAChI1ZhTh/VdjzqHkQC2DRN74C+F8i8kUReXy27KKqXpv9/TKAi6/56A6H41Sw6Bv/R1X1JRG5AOApEfmr+ZWqqsKu6RlmXxSPA8BgOGzaxOFwLBkLvfFV9aXZ/zcA/AGAdwC4LiKXAGD2/427fPYJVX1EVR/pUY23w+E4HRz7xheRIYBIVfdnf/8jAP8ewKcAPArgo7P/P3ncvrRWVHONDyNu4kB8mXlpUxxfw4CzQckcjhtJpBwzJf5NcVeJOH4dfndyzJxt5uAJmAdyEYYdQ8UilFQhw8VPhwuPFgPhZINQcIIKR4jyK62vGppmFjSsnM67IP6bs5BGZe9/TqIoLOzC9z4olgIQsRAHFdEE95L2UbIoCjlH2B8TURFQ2tDIs6Q8Dm4cUlAgf/5e6YKR/EV+6l8E8Aczp0UC4L+p6v8UkS8A+B0ReQzAtwD85EJHdDgcp45jJ76qfhPADzQsvw3gx+7FoBwOx72FZ+45HC3EUnP1ATV8lOP4IW+0fKUoG+KwxMH4m6wmrlpR0wXOQWelQxZB4Bx15UR8hOOuKH7M560syEidPxPyK3BfEeo7gkrDxiPs/AibeBwtOMGbc7ya4/xNufp89yr6TEXXvqRBFsrCp8ThueYhEHoJB8U5FDE1F40ozh5VR78r+Xkr6ZltEm4J98GfofOgBqbmkAsm6/sb3+FoIXziOxwthE98h6OFEOaj9/RgIjdxGPo7B+DWMZufNnyMJwMf48lg0TG+WVXPH7fRUif+qwcVeXquyu++hI/xZOBjPBmc9Bj9p77D0UL4xHc4WojTmvhPnNJxXwt8jCcDH+PJ4ETHeCoc3+FwnC78p77D0UIsdeKLyHtE5Osi8pyI3DcafSLyGyJyQ0S+MrfsjIg8JSLPzv7fPMXxPSginxWRr4nIV0Xkg/fhGHsi8nkR+fPZGH9xtvxhEfnc7J5/QkQ6x+1rCWONReRLIvLp+3iMz4vIX4rIl0Xk6dmyE7vfS5v4cpjk/p8B/GMAbwPw0yLytmUd/xj8JoD30LL7SUy0BPDzqvo2AD8E4F/Mrt39NMYMwLtU9QcAvB3Ae0TkhwD8EoBfUdW3ANgG8NgpjvEVfBDAM3P2/ThG4F4K3KrqUv4B+GEAfzRnfwTAR5Z1/AXG9xCAr8zZXwdwafb3JQBfP+0xzo3tkwDefb+OEcAAwJ8B+Hs4TDpJmp6BUxrbldmkeReAT+NQJfS+GuNsHM8DOEfLTux+L/On/mUAL8zZL86W3a+4L8VEReQhAD8I4HO4z8Y4+wn9ZRzKsD0F4BsAdlRfLau7H+75rwL4BeDV0r2zuP/GCNxjgdsll+V+Z0L17mKiy4SIrAD4PQA/q6p786Wx98MYVbUC8HYR2cChNuP3n+Z4GCLy4wBuqOoXReSdpz2eY/C6BW4XwTLf+C8BeHDOvjJbdr9iITHRZUFEUhxO+t9S1d+fLb6vxvgKVHUHwGdx+LN5Q+6IBp72Pf8RAD8x6wz1cRz+3P813F9jBPDGBG4XwTIn/hcAvHXmQe0A+CkcCnber3hFTBRYUEz0XkEOX+0fA/CMqv7y3Kr7aYznZ296iEgfhz6IZ3D4BfCB2WanOkZV/YiqXlHVh3D4/P0fVf0Z3EdjBA4FbkVk9ZW/cShw+xWc5P1essPivQD+Gofc79+ctgNlbly/DeAagAKHHO8xHHK/zwB4FsD/BnDmFMf3ozjkfH8B4Muzf++9z8b4twF8aTbGrwD4t7Pl3wvg8wCeA/C7ALqnfb9n43ongE/fj2OcjefPZ/+++spcOcn77Zl7DkcL4Zl7DkcL4RPf4WghfOI7HC2ET3yHo4Xwie9wtBA+8R2OFsInvsPRQvjEdzhaiP8PHwQd3rHzL9UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "img_rescaled = cv.resize(img, (53,53))\n",
    "print('Dimensions of the rescaled image:', img_rescaled.shape)\n",
    "plt.imshow(img_rescaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have a look what the average size of the images of class 1 are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average x-dim:  53.857142857142854\n",
      "Average y-dim:  52.90952380952381\n",
      "Average channels:  3.0\n"
     ]
    }
   ],
   "source": [
    "cur_path = 'Final_Training/Images/00000/'\n",
    "files_class1 = [f for f in listdir(cur_path) if isfile(join(cur_path, f))]\n",
    "dims = np.zeros((len(files_class1), 3))\n",
    "for i,f in enumerate(files_class1):\n",
    "    img = cv.imread(join(cur_path, f), cv.IMREAD_COLOR)\n",
    "    if img is not None:\n",
    "        dims[i] = img.shape\n",
    "\n",
    "# Remove error images \n",
    "err_imgs = (np.all(dims == 0, axis=1)) # Check where the dimension entry consists of only zeros (0,0,0)\n",
    "err_imgs_ind = np.arange(len(files_class1))[err_imgs] # Get the indices of theres entries\n",
    "dims = np.delete(dims, err_imgs_ind, axis=0) # And remove them\n",
    "\n",
    "print('Average x-dim: ', np.average(dims[:,0]))\n",
    "print('Average y-dim: ', np.average(dims[:,1]))\n",
    "print('Average channels: ', np.average(dims[:,2]))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see that the average size if roughly 53x53, so we will rescale all images to 52x52 (as we can easily devide this number by 2). (We could take the images from the other classses into account as well to achieve a more accurate result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Import and Serialisation\n",
    "\n",
    "We now import the whole data set and serialise it appropriately with pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data set loaded from files.\n"
     ]
    }
   ],
   "source": [
    "training_path = 'Final_Training/Images'\n",
    "try:\n",
    "    train_data = pickle.load(open('train_data.p', 'rb'))\n",
    "    train_labels = pickle.load(open('train_labels.p', 'rb'))\n",
    "    print('Data set loaded from files.')\n",
    "except:\n",
    "    train_data = np.zeros((40000,52,52,3), dtype=np.float32)\n",
    "    train_labels = np.zeros((40000, 43), dtype=np.float32)\n",
    "    i = 0\n",
    "    for folder in listdir(training_path):\n",
    "        label = int(folder[-2:])\n",
    "        cur_path = join(training_path, folder)\n",
    "        files = [f for f in listdir(cur_path) if isfile(join(cur_path, f))]\n",
    "        for img in files:\n",
    "            img = cv.imread(join(cur_path,img), cv.IMREAD_COLOR)\n",
    "            if img is not None:\n",
    "                img = cv.cvtColor(img, cv.COLOR_BGR2RGB)\n",
    "                img = cv.resize(img, (52,52))\n",
    "                train_data[i,:,:,:] = img\n",
    "                train_labels[i,:] = np.identity(43)[label,:]\n",
    "                i += 1\n",
    "    # Get error images\n",
    "    err_ind = np.arange(40000)[(np.sum(train_data, axis=(1,2,3)) == 0)]\n",
    "    train_data = np.delete(train_data, err_ind, axis=0)\n",
    "    train_labels = np.delete(train_labels, err_ind, axis=0)\n",
    "    assert len(train_labels) == len(train_data)\n",
    "    print('Length of the training set:', len(train_labels))\n",
    "    \n",
    "    #Normalise data\n",
    "    train_data /= 255.0\n",
    "    train_data -= 0.5\n",
    "    \n",
    "    pickle.dump(train_data, open('train_data.p', 'wb'))\n",
    "    pickle.dump(train_labels, open('train_labels.p', 'wb'))\n",
    "    print('Saved to file.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shuffle training data\n",
    "p = np.random.permutation(len(train_data))\n",
    "train_data = train_data[p]\n",
    "train_labels = train_labels[p]\n",
    "\n",
    "# Remove a validation set\n",
    "train_data, val_data = np.split(train_data, [-200])\n",
    "train_labels, val_labels = np.split(train_labels, [-200])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning\n",
    "\n",
    "First, we define the network we want to use. In this case, we use the low-level API of tensorflow to ensure maximum control over what is performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph = tf.Graph()\n",
    "\n",
    "with graph.as_default():\n",
    "    x = tf.placeholder(tf.float32, shape=(None, 52,52,3))\n",
    "    y = tf.placeholder(tf.float32, shape=(None,43))\n",
    "    val_x = tf.constant(val_data)\n",
    "\n",
    "    # Weights\n",
    "    c1 = tf.Variable(tf.truncated_normal(shape=[3,3,3,32], stddev=0.1))\n",
    "    b1 = tf.Variable(tf.constant(1.0, shape=[32]))\n",
    "    c2 = tf.Variable(tf.truncated_normal(shape=[3,3,32,64], stddev=0.1))\n",
    "    b2 = tf.Variable(tf.constant(1.0, shape=[64]))\n",
    "    c3 = tf.Variable(tf.truncated_normal(shape=[2,2,64,128], stddev=0.1))\n",
    "    b3 = tf.Variable(tf.constant(1.0, shape=[128]))\n",
    "    \n",
    "    #w1 = tf.Variable(tf.truncated_normal(shape=[7*7*128, 1024], stddev=0.1))\n",
    "    #b4 = tf.Variable(tf.constant(0.0, shape=[1024]))\n",
    "    \n",
    "    w1 = tf.Variable(tf.truncated_normal(shape=[7*7*128, 2048], stddev=0.1))\n",
    "    b4 = tf.Variable(tf.constant(0.0, shape=[2048]))\n",
    "    w2 = tf.Variable(tf.truncated_normal(shape=[2048, 1024], stddev=0.1))\n",
    "    b5 = tf.Variable(tf.constant(0.0, shape=[1024]))\n",
    "    \n",
    "    w3 = tf.Variable(tf.truncated_normal(shape=[1024,43], stddev=0.1))\n",
    "    b6 = tf.Variable(tf.constant(0.0, shape=[43]))\n",
    "\n",
    "\n",
    "    def cnn_model(x, istrain=False):\n",
    "    # Conv+Max Pool (52,52,3) -> (26,26,32)\n",
    "        conv = tf.nn.conv2d(input=x, filter=c1, strides=[1,1,1,1], padding='SAME')\n",
    "        conv = tf.nn.relu(conv + b1)\n",
    "        maxp = tf.nn.max_pool(conv, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
    "\n",
    "        # Conv+Max Pool (26,26,32) -> (13, 13, 64)\n",
    "        conv2 = tf.nn.conv2d(input=maxp, filter=c2, strides=[1,1,1,1], padding='SAME')\n",
    "        conv2 = tf.nn.relu(conv2 + b2)\n",
    "        maxp2 = tf.nn.max_pool(conv2, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
    "\n",
    "        # Conv+Max Pool (13, 13, 64) -> (7,7,128)\n",
    "        conv3 = tf.nn.conv2d(input=maxp2, filter=c3, strides=[1,1,1,1], padding='SAME')\n",
    "        conv3 = tf.nn.relu(conv3 + b3)\n",
    "        maxp3 = tf.nn.max_pool(conv3, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
    "        maxp3 = tf.reshape(maxp3, shape=[-1, 7*7*128])\n",
    "        \n",
    "        # Dense Layers\n",
    "        dl = tf.nn.relu(tf.matmul(maxp3, w1)+b4)\n",
    "        if istrain:\n",
    "            dl = tf.nn.dropout(dl, 0.5)\n",
    "        dl2 = tf.nn.relu(tf.matmul(dl, w2)+b5)\n",
    "        if istrain:\n",
    "            dl2 = tf.nn.dropout(dl2,0.5)\n",
    "        dl3 = tf.matmul(dl2, w3) + b6\n",
    "        return dl3\n",
    "\n",
    "    logits = cnn_model(x, True)\n",
    "    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels=y))\n",
    "    optimiser = tf.train.AdamOptimizer().minimize(loss)\n",
    "    val_predictions = tf.nn.softmax(cnn_model(val_x))\n",
    "    train_predictions = tf.nn.softmax(logits=logits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(y, yhat):\n",
    "    ''' Returns the accuracy of the batch predictions'''\n",
    "    l = len(y)\n",
    "    return np.sum((np.argmax(y, axis=1) == np.argmax(yhat, axis=1)).astype(np.float32)) / l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 done.\n",
      "Validation accuracy 0.045\n",
      "Test accuracy 0.015625\n",
      "Time: 1.8421900272369385 s\n",
      "Epoch 50 done.\n",
      "Validation accuracy 0.08\n",
      "Test accuracy 0.0390625\n",
      "Time: 38.23906707763672 s\n",
      "Epoch 100 done.\n",
      "Validation accuracy 0.08\n",
      "Test accuracy 0.078125\n",
      "Time: 40.208839893341064 s\n",
      "Epoch 150 done.\n",
      "Validation accuracy 0.085\n",
      "Test accuracy 0.0859375\n",
      "Time: 39.877641677856445 s\n",
      "Epoch 200 done.\n",
      "Validation accuracy 0.145\n",
      "Test accuracy 0.09375\n",
      "Time: 39.26720833778381 s\n",
      "Epoch 250 done.\n",
      "Validation accuracy 0.135\n",
      "Test accuracy 0.1171875\n",
      "Time: 43.17077589035034 s\n",
      "Epoch 300 done.\n",
      "Validation accuracy 0.19\n",
      "Test accuracy 0.1171875\n",
      "Time: 39.98659324645996 s\n",
      "Epoch 350 done.\n",
      "Validation accuracy 0.205\n",
      "Test accuracy 0.125\n",
      "Time: 40.1056969165802 s\n",
      "Epoch 400 done.\n",
      "Validation accuracy 0.195\n",
      "Test accuracy 0.109375\n",
      "Time: 39.330345153808594 s\n",
      "Epoch 450 done.\n",
      "Validation accuracy 0.23\n",
      "Test accuracy 0.125\n",
      "Time: 39.88817286491394 s\n",
      "Epoch 500 done.\n",
      "Validation accuracy 0.215\n",
      "Test accuracy 0.140625\n",
      "Time: 38.7406964302063 s\n",
      "Epoch 550 done.\n",
      "Validation accuracy 0.265\n",
      "Test accuracy 0.171875\n",
      "Time: 40.635931968688965 s\n",
      "Epoch 600 done.\n",
      "Validation accuracy 0.265\n",
      "Test accuracy 0.1484375\n",
      "Time: 40.22779703140259 s\n",
      "Epoch 650 done.\n",
      "Validation accuracy 0.275\n",
      "Test accuracy 0.15625\n",
      "Time: 38.362465143203735 s\n",
      "Epoch 700 done.\n",
      "Validation accuracy 0.37\n",
      "Test accuracy 0.1640625\n",
      "Time: 40.78031086921692 s\n",
      "Epoch 750 done.\n",
      "Validation accuracy 0.33\n",
      "Test accuracy 0.1796875\n",
      "Time: 40.867313385009766 s\n",
      "Epoch 800 done.\n",
      "Validation accuracy 0.375\n",
      "Test accuracy 0.234375\n",
      "Time: 40.02076506614685 s\n",
      "Epoch 850 done.\n",
      "Validation accuracy 0.38\n",
      "Test accuracy 0.3046875\n",
      "Time: 39.266048431396484 s\n",
      "Epoch 900 done.\n",
      "Validation accuracy 0.42\n",
      "Test accuracy 0.296875\n",
      "Time: 38.61926627159119 s\n",
      "Epoch 950 done.\n",
      "Validation accuracy 0.445\n",
      "Test accuracy 0.3359375\n",
      "Time: 39.728068113327026 s\n",
      "Epoch 1000 done.\n",
      "Validation accuracy 0.465\n",
      "Test accuracy 0.28125\n",
      "Time: 39.3569700717926 s\n",
      "Epoch 1050 done.\n",
      "Validation accuracy 0.47\n",
      "Test accuracy 0.3203125\n",
      "Time: 39.392679929733276 s\n",
      "Epoch 1100 done.\n",
      "Validation accuracy 0.545\n",
      "Test accuracy 0.3515625\n",
      "Time: 39.54079818725586 s\n",
      "Epoch 1150 done.\n",
      "Validation accuracy 0.545\n",
      "Test accuracy 0.4296875\n",
      "Time: 39.00639510154724 s\n",
      "Epoch 1200 done.\n",
      "Validation accuracy 0.595\n",
      "Test accuracy 0.390625\n",
      "Time: 38.726869106292725 s\n",
      "Epoch 1250 done.\n",
      "Validation accuracy 0.59\n",
      "Test accuracy 0.5\n",
      "Time: 40.2406804561615 s\n",
      "Epoch 1300 done.\n",
      "Validation accuracy 0.585\n",
      "Test accuracy 0.4296875\n",
      "Time: 41.236876487731934 s\n",
      "Epoch 1350 done.\n",
      "Validation accuracy 0.595\n",
      "Test accuracy 0.4921875\n",
      "Time: 38.98193717002869 s\n",
      "Epoch 1400 done.\n",
      "Validation accuracy 0.68\n",
      "Test accuracy 0.515625\n",
      "Time: 39.29127645492554 s\n",
      "Epoch 1450 done.\n",
      "Validation accuracy 0.685\n",
      "Test accuracy 0.515625\n",
      "Time: 38.678436517715454 s\n",
      "Epoch 1500 done.\n",
      "Validation accuracy 0.68\n",
      "Test accuracy 0.546875\n",
      "Time: 38.027671813964844 s\n",
      "Epoch 1550 done.\n",
      "Validation accuracy 0.65\n",
      "Test accuracy 0.5\n",
      "Time: 41.659752368927 s\n",
      "Epoch 1600 done.\n",
      "Validation accuracy 0.705\n",
      "Test accuracy 0.5\n",
      "Time: 41.19460844993591 s\n",
      "Epoch 1650 done.\n",
      "Validation accuracy 0.695\n",
      "Test accuracy 0.609375\n",
      "Time: 38.663252115249634 s\n",
      "Epoch 1700 done.\n",
      "Validation accuracy 0.745\n",
      "Test accuracy 0.5859375\n",
      "Time: 39.477153301239014 s\n",
      "Epoch 1750 done.\n",
      "Validation accuracy 0.765\n",
      "Test accuracy 0.625\n",
      "Time: 40.96799159049988 s\n",
      "Epoch 1800 done.\n",
      "Validation accuracy 0.77\n",
      "Test accuracy 0.6796875\n",
      "Time: 40.23512244224548 s\n",
      "Epoch 1850 done.\n",
      "Validation accuracy 0.75\n",
      "Test accuracy 0.6484375\n",
      "Time: 40.128188610076904 s\n",
      "Epoch 1900 done.\n",
      "Validation accuracy 0.77\n",
      "Test accuracy 0.640625\n",
      "Time: 39.36694836616516 s\n",
      "Epoch 1950 done.\n",
      "Validation accuracy 0.78\n",
      "Test accuracy 0.7265625\n",
      "Time: 39.21726584434509 s\n",
      "Epoch 2000 done.\n",
      "Validation accuracy 0.815\n",
      "Test accuracy 0.7109375\n",
      "Time: 39.043272495269775 s\n",
      "Epoch 2050 done.\n",
      "Validation accuracy 0.805\n",
      "Test accuracy 0.640625\n",
      "Time: 39.598146200180054 s\n",
      "Epoch 2100 done.\n",
      "Validation accuracy 0.83\n",
      "Test accuracy 0.7890625\n",
      "Time: 39.268961668014526 s\n",
      "Epoch 2150 done.\n",
      "Validation accuracy 0.845\n",
      "Test accuracy 0.8046875\n",
      "Time: 39.724517583847046 s\n",
      "Epoch 2200 done.\n",
      "Validation accuracy 0.85\n",
      "Test accuracy 0.796875\n",
      "Time: 40.71201014518738 s\n",
      "Epoch 2250 done.\n",
      "Validation accuracy 0.885\n",
      "Test accuracy 0.7421875\n",
      "Time: 40.721471071243286 s\n",
      "Epoch 2300 done.\n",
      "Validation accuracy 0.855\n",
      "Test accuracy 0.7421875\n",
      "Time: 39.47451090812683 s\n",
      "Epoch 2350 done.\n",
      "Validation accuracy 0.87\n",
      "Test accuracy 0.7421875\n",
      "Time: 38.22007727622986 s\n",
      "Epoch 2400 done.\n",
      "Validation accuracy 0.89\n",
      "Test accuracy 0.7890625\n",
      "Time: 41.352797746658325 s\n",
      "Epoch 2450 done.\n",
      "Validation accuracy 0.885\n",
      "Test accuracy 0.7890625\n",
      "Time: 40.78791856765747 s\n",
      "Epoch 2500 done.\n",
      "Validation accuracy 0.9\n",
      "Test accuracy 0.7734375\n",
      "Time: 38.5182569026947 s\n",
      "Epoch 2550 done.\n",
      "Validation accuracy 0.905\n",
      "Test accuracy 0.8046875\n",
      "Time: 37.97688150405884 s\n",
      "Epoch 2600 done.\n",
      "Validation accuracy 0.905\n",
      "Test accuracy 0.828125\n",
      "Time: 38.041362047195435 s\n",
      "Epoch 2650 done.\n",
      "Validation accuracy 0.88\n",
      "Test accuracy 0.7578125\n",
      "Time: 38.03758144378662 s\n",
      "Epoch 2700 done.\n",
      "Validation accuracy 0.89\n",
      "Test accuracy 0.7890625\n",
      "Time: 39.4310622215271 s\n",
      "Epoch 2750 done.\n",
      "Validation accuracy 0.9\n",
      "Test accuracy 0.828125\n",
      "Time: 38.55440092086792 s\n",
      "Epoch 2800 done.\n",
      "Validation accuracy 0.9\n",
      "Test accuracy 0.78125\n",
      "Time: 40.347190380096436 s\n",
      "Epoch 2850 done.\n",
      "Validation accuracy 0.89\n",
      "Test accuracy 0.84375\n",
      "Time: 39.511204957962036 s\n",
      "Epoch 2900 done.\n",
      "Validation accuracy 0.93\n",
      "Test accuracy 0.84375\n",
      "Time: 38.455241203308105 s\n",
      "Epoch 2950 done.\n",
      "Validation accuracy 0.91\n",
      "Test accuracy 0.890625\n",
      "Time: 38.2227840423584 s\n",
      "Epoch 3000 done.\n",
      "Validation accuracy 0.93\n",
      "Test accuracy 0.8515625\n",
      "Time: 38.581475257873535 s\n",
      "Epoch 3050 done.\n",
      "Validation accuracy 0.91\n",
      "Test accuracy 0.859375\n",
      "Time: 39.48576283454895 s\n",
      "Epoch 3100 done.\n",
      "Validation accuracy 0.925\n",
      "Test accuracy 0.875\n",
      "Time: 37.5674889087677 s\n",
      "Epoch 3150 done.\n",
      "Validation accuracy 0.925\n",
      "Test accuracy 0.890625\n",
      "Time: 37.65677762031555 s\n",
      "Epoch 3200 done.\n",
      "Validation accuracy 0.905\n",
      "Test accuracy 0.8515625\n",
      "Time: 39.1521680355072 s\n",
      "Epoch 3250 done.\n",
      "Validation accuracy 0.895\n",
      "Test accuracy 0.8828125\n",
      "Time: 38.73995280265808 s\n",
      "Epoch 3300 done.\n",
      "Validation accuracy 0.93\n",
      "Test accuracy 0.8828125\n",
      "Time: 39.12422156333923 s\n",
      "Epoch 3350 done.\n",
      "Validation accuracy 0.92\n",
      "Test accuracy 0.875\n",
      "Time: 42.54729413986206 s\n",
      "Epoch 3400 done.\n",
      "Validation accuracy 0.94\n",
      "Test accuracy 0.8515625\n",
      "Time: 41.86206579208374 s\n",
      "Epoch 3450 done.\n",
      "Validation accuracy 0.93\n",
      "Test accuracy 0.828125\n",
      "Time: 42.595492124557495 s\n",
      "Epoch 3500 done.\n",
      "Validation accuracy 0.95\n",
      "Test accuracy 0.875\n",
      "Time: 39.961353063583374 s\n",
      "Epoch 3550 done.\n",
      "Validation accuracy 0.935\n",
      "Test accuracy 0.9375\n",
      "Time: 40.63534188270569 s\n",
      "Epoch 3600 done.\n",
      "Validation accuracy 0.935\n",
      "Test accuracy 0.8984375\n",
      "Time: 43.450196981430054 s\n",
      "Epoch 3650 done.\n",
      "Validation accuracy 0.95\n",
      "Test accuracy 0.890625\n",
      "Time: 40.82423210144043 s\n",
      "Epoch 3700 done.\n",
      "Validation accuracy 0.935\n",
      "Test accuracy 0.875\n",
      "Time: 40.11289715766907 s\n",
      "Epoch 3750 done.\n",
      "Validation accuracy 0.93\n",
      "Test accuracy 0.890625\n",
      "Time: 43.99260497093201 s\n",
      "Epoch 3800 done.\n",
      "Validation accuracy 0.945\n",
      "Test accuracy 0.9140625\n",
      "Time: 43.30326843261719 s\n",
      "Epoch 3850 done.\n",
      "Validation accuracy 0.95\n",
      "Test accuracy 0.8984375\n",
      "Time: 38.27740931510925 s\n",
      "Epoch 3900 done.\n",
      "Validation accuracy 0.955\n",
      "Test accuracy 0.9140625\n",
      "Time: 38.47874617576599 s\n",
      "Epoch 3950 done.\n",
      "Validation accuracy 0.96\n",
      "Test accuracy 0.9296875\n",
      "Time: 37.50127720832825 s\n",
      "Epoch 4000 done.\n",
      "Validation accuracy 0.94\n",
      "Test accuracy 0.8984375\n",
      "Time: 45.05230212211609 s\n",
      "Epoch 4050 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.9296875\n",
      "Time: 40.10958909988403 s\n",
      "Epoch 4100 done.\n",
      "Validation accuracy 0.975\n",
      "Test accuracy 0.9296875\n",
      "Time: 42.55528545379639 s\n",
      "Epoch 4150 done.\n",
      "Validation accuracy 0.98\n",
      "Test accuracy 0.9296875\n",
      "Time: 42.28407096862793 s\n",
      "Epoch 4200 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.890625\n",
      "Time: 43.048405170440674 s\n",
      "Epoch 4250 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.96875\n",
      "Time: 39.828553915023804 s\n",
      "Epoch 4300 done.\n",
      "Validation accuracy 0.965\n",
      "Test accuracy 0.890625\n",
      "Time: 39.39187932014465 s\n",
      "Epoch 4350 done.\n",
      "Validation accuracy 0.965\n",
      "Test accuracy 0.890625\n",
      "Time: 41.94807767868042 s\n",
      "Epoch 4400 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.9375\n",
      "Time: 42.33235812187195 s\n",
      "Epoch 4450 done.\n",
      "Validation accuracy 0.955\n",
      "Test accuracy 0.8828125\n",
      "Time: 42.0391902923584 s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4500 done.\n",
      "Validation accuracy 0.945\n",
      "Test accuracy 0.9296875\n",
      "Time: 40.4966185092926 s\n",
      "Epoch 4550 done.\n",
      "Validation accuracy 0.955\n",
      "Test accuracy 0.90625\n",
      "Time: 42.72370100021362 s\n",
      "Epoch 4600 done.\n",
      "Validation accuracy 0.96\n",
      "Test accuracy 0.90625\n",
      "Time: 41.97453474998474 s\n",
      "Epoch 4650 done.\n",
      "Validation accuracy 0.975\n",
      "Test accuracy 0.90625\n",
      "Time: 45.26119112968445 s\n",
      "Epoch 4700 done.\n",
      "Validation accuracy 0.955\n",
      "Test accuracy 0.9609375\n",
      "Time: 45.94171595573425 s\n",
      "Epoch 4750 done.\n",
      "Validation accuracy 0.95\n",
      "Test accuracy 0.9296875\n",
      "Time: 40.266701459884644 s\n",
      "Epoch 4800 done.\n",
      "Validation accuracy 0.965\n",
      "Test accuracy 0.9609375\n",
      "Time: 40.57490062713623 s\n",
      "Epoch 4850 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.9375\n",
      "Time: 39.542888164520264 s\n",
      "Epoch 4900 done.\n",
      "Validation accuracy 0.955\n",
      "Test accuracy 0.96875\n",
      "Time: 40.06524300575256 s\n",
      "Epoch 4950 done.\n",
      "Validation accuracy 0.975\n",
      "Test accuracy 0.90625\n",
      "Time: 39.276992321014404 s\n",
      "Epoch 5000 done.\n",
      "Validation accuracy 0.97\n",
      "Test accuracy 0.953125\n",
      "Time: 40.72553324699402 s\n",
      "Full Validation accuracy: 0.97\n",
      "Full Time: 4018.236939430237 s\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "offset = 0\n",
    "start = time.time()\n",
    "epoch_time = time.time()\n",
    "with tf.Session(graph=graph) as session:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for i in range(5001):        \n",
    "        p = np.random.permutation(batch_size)\n",
    "        batch_data = train_data[offset:offset+batch_size,:,:,:][p]\n",
    "        batch_label = train_labels[offset:offset+batch_size,:][p]\n",
    "        feed_dict = {x: batch_data, y: batch_label}\n",
    "        op, lval,pre = session.run([optimiser, loss, train_predictions], feed_dict=feed_dict)\n",
    "        if i%50 == 0:\n",
    "            print('Epoch', i, 'done.')\n",
    "            print('Validation accuracy', accuracy(val_predictions.eval(), val_labels))\n",
    "            print('Test accuracy', accuracy(pre, batch_label))\n",
    "            print('Time:', time.time() - epoch_time, 's')\n",
    "            epoch_time = time.time()\n",
    "\n",
    "        offset += batch_size\n",
    "        if offset+batch_size >= len(train_data):\n",
    "            offset = (offset+batch_size)%len(train_data)\n",
    "    print('Full Validation accuracy:', accuracy(val_predictions.eval(), val_labels))\n",
    "    print('Full Time:', time.time()-start, 's')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensor-kernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}