cnn-scyfer-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network (LeNet)\n",
    "\n",
    "Update of the [example of CNN](http://deeplearning.net/tutorial/lenet.html#lenet) given on deeplearning.net. This notebook tries to explain all the code as if reader had no knowledge of Theano whatsoever.\n",
    "\n",
    "[Theano](http://deeplearning.net/software/theano/) is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano features:\n",
    "\n",
    "- **tight integration with NumPy** – Use numpy.ndarray in Theano-compiled functions.\n",
    "- **transparent use of a GPU** – Perform data-intensive calculations up to 140x faster than with CPU.(float32 only)\n",
    "- **efficient symbolic differentiation** – Theano does your derivatives for function with one or many inputs.\n",
    "- **speed and stability optimizations** – Get the right answer for log(1+x) even when x is really tiny.\n",
    "- **dynamic C code generation** – Evaluate expressions faster.\n",
    "- **extensive unit-testing and self-verification** – Detect and diagnose many types of mistake.\n",
    "\n",
    "Theano has been powering large-scale computationally intensive scientific investigations since 2007.\n",
    "\n",
    "# Outline of this document:\n",
    "\n",
    "**A. The tools to implement CNNs**\n",
    "   1. The Convolution Operator\n",
    "   2. Testing ConvOp on an image\n",
    "   3. MaxPooling\n",
    "   4. Convolution + MaxPooling layer\n",
    "\n",
    "**B. Full LeNet model**\n",
    "   1. HiddenLayer class\n",
    "   2. LogisticRegression class\n",
    "   3. Loading dataset\n",
    "   4. Putting it all together\n",
    "   \n",
    "**C. Implementation of Learning Rate Decay**\n",
    "\n",
    "**D. Implementation of dropout**\n",
    "   1. Creating dropout function\n",
    "   2. Creating dropout classes\n",
    "   3. Rewriting evaluate_lenet5\n",
    "   \n",
    "**E. Visualization of the convolutional filters**\n",
    "   1. Visualization function\n",
    "   2. Testing the function on a single untrained LeNetConvPoolLayer\n",
    "   3. Displaying the learned filters after training\n",
    "   \n",
    "**F. Automated creation of a CNN + MLP**\n",
    "\n",
    "## A. The tools to implement CNNs\n",
    "\n",
    "### 1. The Convolution Operator \n",
    "\n",
    "ConvOp is the main workhorse for implementing a convolutional layer in Theano. ConvOp is used by theano.tensor.signal.conv2d, which takes two symbolic inputs:\n",
    "\n",
    "- a 4D tensor corresponding to a mini-batch of input images. The shape of the tensor is as follows: [mini-batch size, number of input feature maps, image height, image width].\n",
    "- a 4D tensor corresponding to the weight matrix W. The shape of the tensor is: [number of feature maps at layer m, number of feature maps at layer m-1, filter height, filter width]\n",
    "\n",
    "Below is the Theano code for implementing a convolutional layer similar to the one of Figure 1. The input consists of 3 features maps (an RGB color image) of size 120x160. We use two convolutional filters with 9x9 receptive fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cPickle\n",
    "import gzip\n",
    "import os\n",
    "import sys\n",
    "import timeit\n",
    "\n",
    "import numpy\n",
    "\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "from theano.tensor.signal import downsample\n",
    "from theano.tensor.nnet import conv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rng = numpy.random.RandomState(23455)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# instantiate 4D tensor for input\n",
    "input = T.tensor4(name='input')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "w_shp = (2, 3, 9, 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of the 4D tensor corresponding to the weight matrix W is:\n",
    "\n",
    "- number of feature maps at layer 2: as we chose to have only 2 convolutional filters, we will have 2 resulting feature maps.\n",
    "- number of feature maps at layer 1: the original image being RGB, it has 3 layers on top of each other, so 3 feature maps.\n",
    "- filter height: the convolutional filters has 9x9 receptive fields, so height = 9 pixels\n",
    "- filter width: similarly, width = 9 pixels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w_bound = numpy.sqrt(3 * 9 * 9) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we use the same weight initialization formula as with the MLP. Weights are sampled randomly from a uniform distribution in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit. For MLPs, this was the number of units in the layer below. For CNNs however, we have to take into account the number of input feature maps and the size of the receptive fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = theano.shared( numpy.asarray(\n",
    "            rng.uniform(\n",
    "                low=-1.0 / w_bound,\n",
    "                high=1.0 / w_bound,\n",
    "                size=w_shp),\n",
    "            dtype=input.dtype), name ='W')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **RandomState.uniform**(low=0.0, high=1.0, size=None): draw samples from a uniform distribution: samples are uniformly distributed over the half-open interval [low, high) (includes low, but excludes high). In other words, any value within the given interval is equally likely to be drawn by uniform. [source](http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.RandomState.uniform.html#numpy.random.RandomState.uniform)\n",
    "\n",
    "- **theano.shared**: the main benefits of using shared constructors are you can use them to initialise important variables with predefined numerical values (weight matrices in a neural network, for example). [source](https://www.quora.com/What-is-the-meaning-and-benefit-of-shared-variables-in-Theano)\n",
    "\n",
    "The distinction between Theano-managed memory and user-managed memory can be broken down by some Theano functions (e.g. shared, get_value and the constructors for In and Out) by using a **borrow=True flag**. This can make those methods faster (by avoiding copy operations) at the expense of risking subtle bugs in the overall program (by aliasing memory).\n",
    "\n",
    "*Take home message*:\n",
    "\n",
    "It is a **safe practice** (and a good idea) to use **borrow=True** in a shared variable constructor when the shared variable stands for a **large object (in terms of memory footprint)** and you do not want to create copies of it in memory.\n",
    "\n",
    "It is not a reliable technique to use borrow=True to modify shared variables through side-effect, because with some devices (e.g. GPU devices) this technique will not work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# initialize shared variable for bias (1D tensor) with random values\n",
    "# IMPORTANT: biases are usually initialized to zero. However in this\n",
    "# particular application, we simply apply the convolutional layer to\n",
    "# an image without learning the parameters. We therefore initialize\n",
    "# them to random values to \"simulate\" learning.\n",
    "b_shp = (2,)\n",
    "b = theano.shared(numpy.asarray(\n",
    "            rng.uniform(low=-.5, high=.5, size=b_shp),\n",
    "            dtype=input.dtype), name ='b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We chose to have only 2 filters, so 2 bias terms need to be initialized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build symbolic expression that computes the convolution of input with filters in w\n",
    "conv_out = conv.conv2d(input, W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`nnet.conv2d`**: This is the standard operator for convolutional neural networks working with batches of multi-channel 2D images, available for CPU and GPU. [source](http://deeplearning.net/software/theano/library/tensor/nnet/conv.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build symbolic expression to add bias and apply activation function, i.e. produce neural net layer output\n",
    "output = T.nnet.sigmoid(conv_out + b.dimshuffle('x', 0, 'x', 'x'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tensor.nnet.sigmoid(x)`: returns the standard sigmoid nonlinearity applied to x. \n",
    "\n",
    "- Parameters: x - symbolic Tensor (or compatible)\n",
    "- Return type: same as x\n",
    "- Returns:\telement-wise sigmoid: $$sigmoid(x) = \\frac{1}{1 + \\exp(-x)}$$.\n",
    "\n",
    "**Note:** in numpy and in Theano, the transpose of a vector is exactly the same vector! Use reshape or **dimshuffle** to turn your vector into a row or column matrix. [source](http://deeplearning.net/software/theano/library/tensor/basic.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create theano function to compute filtered images\n",
    "f = theano.function([input], output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Testing ConvOp on an image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pylab\n",
    "from PIL import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# open random image of dimensions 1936×2592\n",
    "img = Image.open(open('images/profilepic4.jpg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "img = numpy.asarray(img, dtype='float64') / 256. # divide by 256 to have RGB 0-1 scale and not 0 - 256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#put image in 4D tensor of shape (1, 3, height, width)\n",
    "img_ = img.transpose(2, 0, 1).reshape(1, 3, 2592, 1936)\n",
    "filtered_img = f(img_)\n",
    "\n",
    "# plot original image and first and second components of output\n",
    "pylab.subplot(1, 3, 1); pylab.axis('off'); pylab.imshow(img)\n",
    "pylab.gray();\n",
    "# recall that the convOp output (filtered image) is actually a \"minibatch\",\n",
    "# of size 1 here, so we take index 0 in the first dimension:\n",
    "pylab.subplot(1, 3, 2); pylab.axis('off'); pylab.imshow(filtered_img[0, 0, :, :])\n",
    "pylab.subplot(1, 3, 3); pylab.axis('off'); pylab.imshow(filtered_img[0, 1, :, :])\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/figure_3.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. MaxPooling\n",
    "\n",
    "Another important concept of CNNs is *max-pooling*, which is a form of non-linear down-sampling. Max-pooling partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value.\n",
    "\n",
    "Max-pooling is useful in vision for two reasons:\n",
    "\n",
    "1. By eliminating non-maximal values, it reduces computation for upper layers.\n",
    "\n",
    "2. It provides a form of **translation invariance**. Imagine cascading a max-pooling layer with a convolutional layer. There are 8 directions in which one can translate the input image by a single pixel. If max-pooling is done over a 2x2 region, 3 out of these 8 possible configurations will produce exactly the same output at the convolutional layer. For max-pooling over a 3x3 window, this jumps to 5/8.\n",
    "\n",
    "    Since it provides additional robustness to position, max-pooling is a “smart” **way of reducing the dimensionality of intermediate representations**.\n",
    "\n",
    "Max-pooling is done in Theano by way of **`theano.tensor.signal.downsample.max_pool_2d`**. This function takes as input an N dimensional tensor (where N >= 2) and a downscaling factor and performs max-pooling over the 2 trailing dimensions of the tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from theano.tensor.signal import downsample\n",
    "\n",
    "input = T.dtensor4('input')\n",
    "maxpool_shape = (2, 2)\n",
    "pool_out = downsample.max_pool_2d(input, maxpool_shape, ignore_border=True)\n",
    "g = theano.function([input],pool_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With ignore_border set to True:\n",
      "invals[0, 0, :, :] =\n",
      "[[  4.17022005e-01   7.20324493e-01   1.14374817e-04   3.02332573e-01\n",
      "    1.46755891e-01]\n",
      " [  9.23385948e-02   1.86260211e-01   3.45560727e-01   3.96767474e-01\n",
      "    5.38816734e-01]\n",
      " [  4.19194514e-01   6.85219500e-01   2.04452250e-01   8.78117436e-01\n",
      "    2.73875932e-02]\n",
      " [  6.70467510e-01   4.17304802e-01   5.58689828e-01   1.40386939e-01\n",
      "    1.98101489e-01]\n",
      " [  8.00744569e-01   9.68261576e-01   3.13424178e-01   6.92322616e-01\n",
      "    8.76389152e-01]]\n",
      "output[0, 0, :, :] =\n",
      "[[ 0.72032449  0.39676747]\n",
      " [ 0.6852195   0.87811744]]\n"
     ]
    }
   ],
   "source": [
    "invals = numpy.random.RandomState(1).rand(3, 2, 5, 5)\n",
    "print 'With ignore_border set to True:'\n",
    "print 'invals[0, 0, :, :] =\\n', invals[0, 0, :, :]\n",
    "print 'output[0, 0, :, :] =\\n', g(invals)[0, 0, :, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With ignore_border set to False:\n",
      "invals[0, 0, :, :] =\n",
      "[[  4.17022005e-01   7.20324493e-01   1.14374817e-04   3.02332573e-01\n",
      "    1.46755891e-01]\n",
      " [  9.23385948e-02   1.86260211e-01   3.45560727e-01   3.96767474e-01\n",
      "    5.38816734e-01]\n",
      " [  4.19194514e-01   6.85219500e-01   2.04452250e-01   8.78117436e-01\n",
      "    2.73875932e-02]\n",
      " [  6.70467510e-01   4.17304802e-01   5.58689828e-01   1.40386939e-01\n",
      "    1.98101489e-01]\n",
      " [  8.00744569e-01   9.68261576e-01   3.13424178e-01   6.92322616e-01\n",
      "    8.76389152e-01]]\n",
      "output[0, 0, :, :] =\n",
      "[[ 0.72032449  0.39676747  0.53881673]\n",
      " [ 0.6852195   0.87811744  0.19810149]\n",
      " [ 0.96826158  0.69232262  0.87638915]]\n"
     ]
    }
   ],
   "source": [
    "pool_out = downsample.max_pool_2d(input, maxpool_shape, ignore_border=False)\n",
    "g = theano.function([input],pool_out)\n",
    "print 'With ignore_border set to False:'\n",
    "print 'invals[0, 0, :, :] =\\n', invals[0, 0, :, :]\n",
    "print 'output[0, 0, :, :] =\\n', g(invals)[0, 0, :, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`theano.tensor.signal.downsample.max_pool_2d`**`(input, ds, ignore_border=None, st=None, padding=(0, 0), mode='max')`: takes as input a N-D tensor, where **N >= 2**. It downscales the input image by the specified factor, by *keeping only the maximum value of non-overlapping patches of size* (ds[0],ds[1])\n",
    "\n",
    "Parameters:\t\n",
    "\n",
    "- **input (N-D theano tensor of input images)** – Input images. **Max pooling will be done over the 2 last dimensions**.\n",
    "- **ds** (tuple of length 2) – **Factor by which to downscale** (vertical ds, horizontal ds). (2,2) will halve the image in each dimension.\n",
    "- ignore_border (bool (default None, will print a warning and set to False)) – When True, (5,5) input with ds=(2,2) will generate a (2,2) output. (3,3) otherwise.\n",
    "- st (tuple of lenght 2) – Stride size, which is the number of shifts over rows/cols to get the next pool region. If st is None, it is considered equal to ds (no overlap on pooling regions).\n",
    "- padding (tuple of two ints) – (pad_h, pad_w), pad zeros to extend beyond four borders of the images, pad_h is the size of the top and bottom margins, and pad_w is the size of the left and right margins.\n",
    "- mode ({‘max’, ‘sum’, ‘average_inc_pad’, ‘average_exc_pad’}) – Operation executed on each window. max and sum always exclude the padding in the computation. average gives you the choice to include or exclude it.\n",
    "\n",
    "[source](http://deeplearning.net/software/theano/library/tensor/signal/downsample.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Convolution + MaxPooling layer\n",
    "\n",
    "We now have all we need to implement a LeNet model in Theano. We start with the `LeNetConvPoolLayer` class, which implements a {convolution + max-pooling} layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LeNetConvPoolLayer(object):\n",
    "    \"\"\"Pool Layer of a convolutional network \"\"\"\n",
    "\n",
    "    def __init__(self, rng, input, filter_shape, image_shape, poolsize=(2, 2)):\n",
    "        \"\"\"\n",
    "        Allocate a LeNetConvPoolLayer with shared variable internal parameters.\n",
    "\n",
    "        :type rng: numpy.random.RandomState\n",
    "        :param rng: a random number generator used to initialize weights\n",
    "\n",
    "        :type input: theano.tensor.dtensor4\n",
    "        :param input: symbolic image tensor, of shape image_shape\n",
    "\n",
    "        :type filter_shape: tuple or list of length 4\n",
    "        :param filter_shape: (number of filters, num input feature maps,\n",
    "                              filter height, filter width)\n",
    "\n",
    "        :type image_shape: tuple or list of length 4\n",
    "        :param image_shape: (batch size, num input feature maps,\n",
    "                             image height, image width)\n",
    "\n",
    "        :type poolsize: tuple or list of length 2\n",
    "        :param poolsize: the downsampling (pooling) factor (#rows, #cols)\n",
    "        \"\"\"\n",
    "\n",
    "        assert image_shape[1] == filter_shape[1]\n",
    "        \n",
    "        # assert just checks if the number of feature maps is consistent between filter shape and image_shape\n",
    "        \n",
    "        self.input = input\n",
    "\n",
    "        # there are \"num input feature maps * filter height * filter width\"\n",
    "        # inputs to each hidden unit\n",
    "        # reminder: Weights are sampled randomly from a uniform distribution \n",
    "        # in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit\n",
    "        fan_in = numpy.prod(filter_shape[1:])\n",
    "        # each unit in the lower layer receives a gradient from:\n",
    "        # \"num output feature maps * filter height * filter width\" /\n",
    "        #   pooling size\n",
    "        fan_out = (filter_shape[0] * numpy.prod(filter_shape[2:]) /\n",
    "                   numpy.prod(poolsize))\n",
    "        # initialize weights with random weights\n",
    "        W_bound = numpy.sqrt(6. / (fan_in + fan_out))\n",
    "        self.W = theano.shared(\n",
    "            numpy.asarray(\n",
    "                rng.uniform(low=-W_bound, high=W_bound, size=filter_shape),\n",
    "                dtype=theano.config.floatX\n",
    "            ),\n",
    "            borrow=True # see above the def of theano.shared for explanation of borrow\n",
    "        )\n",
    "\n",
    "        # the bias is a 1D tensor -- one bias per output feature map\n",
    "        b_values = numpy.zeros((filter_shape[0],), dtype=theano.config.floatX)\n",
    "        self.b = theano.shared(value=b_values, borrow=True)\n",
    "\n",
    "        # convolve input feature maps with filters\n",
    "        conv_out = conv.conv2d(\n",
    "            input=input,\n",
    "            filters=self.W,\n",
    "            filter_shape=filter_shape,\n",
    "            image_shape=image_shape\n",
    "        )\n",
    "\n",
    "        # downsample each feature map individually, using maxpooling\n",
    "        pooled_out = downsample.max_pool_2d(\n",
    "            input=conv_out,\n",
    "            ds=poolsize,\n",
    "            ignore_border=True\n",
    "        )\n",
    "\n",
    "        # add the bias term. Since the bias is a vector (1D array), we first\n",
    "        # reshape it to a tensor of shape (1, n_filters, 1, 1). Each bias will\n",
    "        # thus be broadcasted across mini-batches and feature map\n",
    "        # width & height\n",
    "        self.output = T.tanh(pooled_out + self.b.dimshuffle('x', 0, 'x', 'x'))\n",
    "\n",
    "        # store parameters of this layer\n",
    "        self.params = [self.W, self.b]\n",
    "\n",
    "        # keep track of model input\n",
    "        self.input = input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when initializing the weight values, the fan-in is determined by the size of the receptive fields and the number of input feature maps.\n",
    "\n",
    "## B. Full LeNet model\n",
    "\n",
    "Sparse, convolutional layers and max-pooling are at the heart of the LeNet family of models. While the exact details of the model will vary greatly, the figure below shows a graphical depiction of a LeNet model.\n",
    "\n",
    "<img src=\"images/mylenet.png\">\n",
    "\n",
    "The **lower-layers** are composed to **alternating convolution and max-pooling layers**. The **upper-layers** however are fully-connected and correspond to a **traditional MLP (hidden layer + logistic regression)**. The input to the first fully-connected layer is the set of all features maps at the layer below.\n",
    "\n",
    "From an implementation point of view, this means lower-layers operate on 4D tensors. These are then flattened to a 2D matrix of rasterized feature maps, to be compatible with our previous MLP implementation.\n",
    "\n",
    "Using the LogisticRegression class defined in [Classifying MNIST digits using Logistic Regression](http://deeplearning.net/tutorial/logreg.html) and the HiddenLayer class defined in [Multilayer Perceptron](http://deeplearning.net/tutorial/mlp.html), we can instantiate the network as follows.\n",
    "\n",
    "### 1. HiddenLayer class\n",
    "\n",
    "The original code for this class can be found here: [source](http://deeplearning.net/tutorial/mlp.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class HiddenLayer(object):\n",
    "    def __init__(self, rng, input, n_in, n_out, W=None, b=None,\n",
    "                 activation=T.tanh):\n",
    "        \"\"\"\n",
    "        Typical hidden layer of a MLP: units are fully-connected and have\n",
    "        sigmoidal activation function. Weight matrix W is of shape (n_in,n_out)\n",
    "        and the bias vector b is of shape (n_out,).\n",
    "\n",
    "        NOTE : The nonlinearity used here is tanh\n",
    "\n",
    "        Hidden unit activation is given by: tanh(dot(input,W) + b)\n",
    "\n",
    "        :type rng: numpy.random.RandomState\n",
    "        :param rng: a random number generator used to initialize weights\n",
    "\n",
    "        :type input: theano.tensor.dmatrix\n",
    "        :param input: a symbolic tensor of shape (n_examples, n_in)\n",
    "\n",
    "        :type n_in: int\n",
    "        :param n_in: dimensionality of input\n",
    "\n",
    "        :type n_out: int\n",
    "        :param n_out: number of hidden units\n",
    "\n",
    "        :type activation: theano.Op or function\n",
    "        :param activation: Non linearity to be applied in the hidden\n",
    "                           layer\n",
    "        \"\"\"\n",
    "        self.input = input\n",
    "\n",
    "        # `W` is initialized with `W_values` which is uniformely sampled\n",
    "        # from sqrt(-6./(n_in+n_hidden)) and sqrt(6./(n_in+n_hidden))\n",
    "        # for tanh activation function\n",
    "        # the output of uniform if converted using asarray to dtype\n",
    "        # theano.config.floatX so that the code is runable on GPU\n",
    "        # Note : optimal initialization of weights is dependent on the\n",
    "        #        activation function used (among other things).\n",
    "        #        For example, results presented in [Xavier10] suggest that you\n",
    "        #        should use 4 times larger initial weights for sigmoid\n",
    "        #        compared to tanh\n",
    "        #        We have no info for other function, so we use the same as\n",
    "        #        tanh.\n",
    "        if W is None:\n",
    "            W_values = numpy.asarray(\n",
    "                rng.uniform(\n",
    "                    low=-numpy.sqrt(6. / (n_in + n_out)),\n",
    "                    high=numpy.sqrt(6. / (n_in + n_out)),\n",
    "                    size=(n_in, n_out)\n",
    "                ),\n",
    "                dtype=theano.config.floatX\n",
    "            )\n",
    "            if activation == theano.tensor.nnet.sigmoid:\n",
    "                W_values *= 4\n",
    "\n",
    "            W = theano.shared(value=W_values, name='W', borrow=True)\n",
    "\n",
    "        if b is None:\n",
    "            b_values = numpy.zeros((n_out,), dtype=theano.config.floatX)\n",
    "            b = theano.shared(value=b_values, name='b', borrow=True)\n",
    "\n",
    "        self.W = W\n",
    "        self.b = b\n",
    "\n",
    "        lin_output = T.dot(input, self.W) + self.b\n",
    "        self.output = (\n",
    "            lin_output if activation is None\n",
    "            else activation(lin_output)\n",
    "        )\n",
    "        # parameters of the model\n",
    "        self.params = [self.W, self.b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class uses tanh as activation function by default. This can be supported by the results presented in the scientific paper by called [Performance Analysis of Various Activation Functions in Generalized MLP Architectures of Neural Networks](http://www.researchgate.net/profile/Bekir_Karlik/publication/228813985_Performance_Analysis_of_Various_Activation_Functions_in_Generalized_MLP_Architectures_of_Neural_Networks/links/004635229e9d608b3a000000.pdf) by Ahmet V Olgac and Bekir Karlik.\n",
    "\n",
    "> In this study, we have used five conventional differentiable and monotonic activation functions for\n",
    "the evolution of MLP architecture along with Generalized Delta rule learning. These proposed\n",
    "well-known and effective activation functions are Bi-polar sigmoid, Uni-polar sigmoid, Tanh, Conic\n",
    "Section, and Radial Bases Function (RBF). Having compared their performances, simulation\n",
    "results show that **Tanh (hyperbolic tangent) function performs better recognition accuracy than\n",
    "those of the other functions**. In other words, the neural network computed good results when\n",
    "“Tanh-Tanh” combination of activation functions was used for both neurons (or nodes) of hidden\n",
    "and output layers. \n",
    "\n",
    "The paper by Xavier can be found at:\n",
    "\n",
    "### 2. LogisticRegression class\n",
    "\n",
    "The original code for this class can be found here: [source]('http://deeplearning.net/tutorial/logreg.html')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LogisticRegression(object):\n",
    "    \"\"\"Multi-class Logistic Regression Class\n",
    "\n",
    "    The logistic regression is fully described by a weight matrix :math:`W`\n",
    "    and bias vector :math:`b`. Classification is done by projecting data\n",
    "    points onto a set of hyperplanes, the distance to which is used to\n",
    "    determine a class membership probability.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, input, n_in, n_out):\n",
    "        \"\"\" Initialize the parameters of the logistic regression\n",
    "\n",
    "        :type input: theano.tensor.TensorType\n",
    "        :param input: symbolic variable that describes the input of the\n",
    "                      architecture (one minibatch)\n",
    "\n",
    "        :type n_in: int\n",
    "        :param n_in: number of input units, the dimension of the space in\n",
    "                     which the datapoints lie\n",
    "\n",
    "        :type n_out: int\n",
    "        :param n_out: number of output units, the dimension of the space in\n",
    "                      which the labels lie\n",
    "\n",
    "        \"\"\"\n",
    "        # initialize with 0 the weights W as a matrix of shape (n_in, n_out)\n",
    "        self.W = theano.shared(\n",
    "            value=numpy.zeros(\n",
    "                (n_in, n_out),\n",
    "                dtype=theano.config.floatX\n",
    "            ),\n",
    "            name='W',\n",
    "            borrow=True\n",
    "        )\n",
    "        # initialize the biases b as a vector of n_out 0s\n",
    "        self.b = theano.shared(\n",
    "            value=numpy.zeros(\n",
    "                (n_out,),\n",
    "                dtype=theano.config.floatX\n",
    "            ),\n",
    "            name='b',\n",
    "            borrow=True\n",
    "        )\n",
    "\n",
    "        # symbolic expression for computing the matrix of class-membership\n",
    "        # probabilities\n",
    "        # Where:\n",
    "        # W is a matrix where column-k represent the separation hyperplane for\n",
    "        # class-k\n",
    "        # x is a matrix where row-j  represents input training sample-j\n",
    "        # b is a vector where element-k represent the free parameter of\n",
    "        # hyperplane-k\n",
    "        self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W) + self.b)\n",
    "\n",
    "        # symbolic description of how to compute prediction as class whose\n",
    "        # probability is maximal\n",
    "        self.y_pred = T.argmax(self.p_y_given_x, axis=1)\n",
    "\n",
    "        # parameters of the model\n",
    "        self.params = [self.W, self.b]\n",
    "\n",
    "        # keep track of model input\n",
    "        self.input = input\n",
    "\n",
    "    def negative_log_likelihood(self, y):\n",
    "        \"\"\"Return the mean of the negative log-likelihood of the prediction\n",
    "        of this model under a given target distribution.\n",
    "\n",
    "        .. math::\n",
    "\n",
    "            \\frac{1}{|\\mathcal{D}|} \\mathcal{L} (\\theta=\\{W,b\\}, \\mathcal{D}) =\n",
    "            \\frac{1}{|\\mathcal{D}|} \\sum_{i=0}^{|\\mathcal{D}|}\n",
    "                \\log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\\\\n",
    "            \\ell (\\theta=\\{W,b\\}, \\mathcal{D})\n",
    "\n",
    "        :type y: theano.tensor.TensorType\n",
    "        :param y: corresponds to a vector that gives for each example the\n",
    "                  correct label\n",
    "\n",
    "        Note: we use the mean instead of the sum so that\n",
    "              the learning rate is less dependent on the batch size\n",
    "        \"\"\"\n",
    "        # y.shape[0] is (symbolically) the number of rows in y, i.e.,\n",
    "        # number of examples (call it n) in the minibatch\n",
    "        # T.arange(y.shape[0]) is a symbolic vector which will contain\n",
    "        # [0,1,2,... n-1] T.log(self.p_y_given_x) is a matrix of\n",
    "        # Log-Probabilities (call it LP) with one row per example and\n",
    "        # one column per class LP[T.arange(y.shape[0]),y] is a vector\n",
    "        # v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ...,\n",
    "        # LP[n-1,y[n-1]]] and T.mean(LP[T.arange(y.shape[0]),y]) is\n",
    "        # the mean (across minibatch examples) of the elements in v,\n",
    "        # i.e., the mean log-likelihood across the minibatch.\n",
    "        return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])\n",
    "\n",
    "    def errors(self, y):\n",
    "        \"\"\"Return a float representing the number of errors in the minibatch\n",
    "        over the total number of examples of the minibatch ; zero one\n",
    "        loss over the size of the minibatch\n",
    "\n",
    "        :type y: theano.tensor.TensorType\n",
    "        :param y: corresponds to a vector that gives for each example the\n",
    "                  correct label\n",
    "        \"\"\"\n",
    "\n",
    "        # check if y has same dimension of y_pred\n",
    "        if y.ndim != self.y_pred.ndim:\n",
    "            raise TypeError(\n",
    "                'y should have the same shape as self.y_pred',\n",
    "                ('y', y.type, 'y_pred', self.y_pred.type)\n",
    "            )\n",
    "        # check if y is of the correct datatype\n",
    "        if y.dtype.startswith('int'):\n",
    "            # the T.neq operator returns a vector of 0s and 1s, where 1\n",
    "            # represents a mistake in prediction\n",
    "            return T.mean(T.neq(self.y_pred, y))\n",
    "        else:\n",
    "            raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.negative_log_likelihood(y)`: this method returns the mean of the negative log-likelihood of the prediction of this model under a given target distribution:\n",
    "$$\\frac{1}{|\\mathcal{D}|} \\mathcal{L} (\\theta=\\{W,b\\}, \\mathcal{D}) =\\frac{1}{|\\mathcal{D}|} \\sum_{i=0}^{|\\mathcal{D}|} \\log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\\\ \\ell (\\theta=\\{W,b\\}, \\mathcal{D})$$\n",
    "\n",
    "type y: theano.tensor.TensorType\n",
    "\n",
    "param y: corresponds to a vector that gives for each example the correct label\n",
    "\n",
    "Note: we use the mean instead of the sum so that the learning rate is less dependent on the batch size.\n",
    "\n",
    "### 3. Loading dataset\n",
    "\n",
    "Original code can be found [here]('http://deeplearning.net/tutorial/code/logistic_sgd.py'). This piece of code loads the dataset and partitions it into: train set, validation set and test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Putting it all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_lenet5(learning_rate=0.1, n_epochs=200,\n",
    "                    dataset='mnist.pkl.gz',\n",
    "                    nkerns=[20, 50], batch_size=500):\n",
    "    \"\"\" Demonstrates lenet on MNIST dataset\n",
    "\n",
    "    :type learning_rate: float\n",
    "    :param learning_rate: learning rate used (factor for the stochastic\n",
    "                          gradient)\n",
    "\n",
    "    :type n_epochs: int\n",
    "    :param n_epochs: maximal number of epochs to run the optimizer\n",
    "\n",
    "    :type dataset: string\n",
    "    :param dataset: path to the dataset used for training /testing (MNIST here)\n",
    "\n",
    "    :type nkerns: list of ints\n",
    "    :param nkerns: number of kernels on each layer (so 20 convolutional filters, and then 50 activation units)\n",
    "    \"\"\"\n",
    "\n",
    "    rng = numpy.random.RandomState(23455)\n",
    "\n",
    "    datasets = load_data(dataset)\n",
    "\n",
    "    train_set_x, train_set_y = datasets[0]\n",
    "    valid_set_x, valid_set_y = datasets[1]\n",
    "    test_set_x, test_set_y = datasets[2]\n",
    "\n",
    "    # compute number of minibatches for training, validation and testing\n",
    "    n_train_batches = train_set_x.get_value(borrow=True).shape[0]\n",
    "    n_valid_batches = valid_set_x.get_value(borrow=True).shape[0]\n",
    "    n_test_batches = test_set_x.get_value(borrow=True).shape[0]\n",
    "    n_train_batches /= batch_size\n",
    "    n_valid_batches /= batch_size\n",
    "    n_test_batches /= batch_size\n",
    "\n",
    "    # allocate symbolic variables for the data\n",
    "    index = T.lscalar()  # index to a [mini]batch\n",
    "\n",
    "    # start-snippet-1\n",
    "    x = T.matrix('x')   # the data is presented as rasterized images\n",
    "    y = T.ivector('y')  # the labels are presented as 1D vector of\n",
    "                        # [int] labels\n",
    "\n",
    "    ######################\n",
    "    # BUILD ACTUAL MODEL #\n",
    "    ######################\n",
    "    print '... building the model'\n",
    "\n",
    "    # Reshape matrix of rasterized images of shape (batch_size, 28 * 28)\n",
    "    # to a 4D tensor, compatible with our LeNetConvPoolLayer\n",
    "    # (28, 28) is the size of MNIST images.\n",
    "    layer0_input = x.reshape((batch_size, 1, 28, 28))\n",
    "\n",
    "    # Construct the first convolutional pooling layer:\n",
    "    # filtering reduces the image size to (28-5+1 , 28-5+1) = (24, 24)\n",
    "    # maxpooling reduces this further to (24/2, 24/2) = (12, 12)\n",
    "    # 4D output tensor is thus of shape (batch_size, nkerns[0], 12, 12)\n",
    "    layer0 = LeNetConvPoolLayer(\n",
    "        rng,\n",
    "        input=layer0_input,\n",
    "        image_shape=(batch_size, 1, 28, 28),\n",
    "        filter_shape=(nkerns[0], 1, 5, 5),\n",
    "        poolsize=(2, 2)\n",
    "    )\n",
    "    '''\n",
    "    Reminder of LeNetConvPoolLayer input parameters and types\n",
    "    :type rng: numpy.random.RandomState\n",
    "    :param rng: a random number generator used to initialize weights\n",
    "\n",
    "    :type input: theano.tensor.dtensor4\n",
    "    :param input: symbolic image tensor, of shape image_shape\n",
    "\n",
    "    :type filter_shape: tuple or list of length 4\n",
    "    :param filter_shape: (number of filters, num input feature maps,\n",
    "                              filter height, filter width)\n",
    "\n",
    "    :type image_shape: tuple or list of length 4\n",
    "    :param image_shape: (batch size, num input feature maps,\n",
    "                             image height, image width)\n",
    "\n",
    "    :type poolsize: tuple or list of length 2\n",
    "    :param poolsize: the downsampling (pooling) factor (#rows, #cols)\n",
    "    '''\n",
    "    # Construct the second convolutional pooling layer\n",
    "    # filtering reduces the image size to (12-5+1, 12-5+1) = (8, 8)\n",
    "    # maxpooling reduces this further to (8/2, 8/2) = (4, 4)\n",
    "    # 4D output tensor is thus of shape (batch_size, nkerns[1], 4, 4)\n",
    "    layer1 = LeNetConvPoolLayer(\n",
    "        rng,\n",
    "        input=layer0.output,\n",
    "        image_shape=(batch_size, nkerns[0], 12, 12),\n",
    "        filter_shape=(nkerns[1], nkerns[0], 5, 5),\n",
    "        poolsize=(2, 2)\n",
    "    )\n",
    "\n",
    "    # the HiddenLayer being fully-connected, it operates on 2D matrices of\n",
    "    # shape (batch_size, num_pixels) (i.e matrix of rasterized images).\n",
    "    # This will generate a matrix of shape (batch_size, nkerns[1] * 4 * 4),\n",
    "    # or (500, 50 * 4 * 4) = (500, 800) with the default values.\n",
    "    layer2_input = layer1.output.flatten(2)\n",
    "\n",
    "    # construct a fully-connected sigmoidal layer\n",
    "    layer2 = HiddenLayer(\n",
    "        rng,\n",
    "        input=layer2_input,\n",
    "        n_in=nkerns[1] * 4 * 4,\n",
    "        n_out=500,\n",
    "        activation=T.tanh\n",
    "    )\n",
    "\n",
    "    # classify the values of the fully-connected sigmoidal layer\n",
    "    layer3 = LogisticRegression(input=layer2.output, n_in=500, n_out=10)\n",
    "\n",
    "    # the cost we minimize during training is the NLL of the model\n",
    "    cost = layer3.negative_log_likelihood(y)\n",
    "\n",
    "    # create a function to compute the mistakes that are made by the model\n",
    "    test_model = theano.function(\n",
    "        [index],\n",
    "        layer3.errors(y),\n",
    "        givens={\n",
    "            x: test_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: test_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    validate_model = theano.function(\n",
    "        [index],\n",
    "        layer3.errors(y),\n",
    "        givens={\n",
    "            x: valid_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: valid_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    # create a list of all model parameters to be fit by gradient descent\n",
    "    params = layer3.params + layer2.params + layer1.params + layer0.params\n",
    "\n",
    "    # create a list of gradients for all model parameters\n",
    "    grads = T.grad(cost, params)\n",
    "\n",
    "    # train_model is a function that updates the model parameters by\n",
    "    # SGD Since this model has many parameters, it would be tedious to\n",
    "    # manually create an update rule for each model parameter. We thus\n",
    "    # create the updates list by automatically looping over all\n",
    "    # (params[i], grads[i]) pairs.\n",
    "    updates = [\n",
    "        (param_i, param_i - learning_rate * grad_i)\n",
    "        for param_i, grad_i in zip(params, grads)\n",
    "    ]\n",
    "\n",
    "    train_model = theano.function(\n",
    "        [index],\n",
    "        cost,\n",
    "        updates=updates,\n",
    "        givens={\n",
    "            x: train_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: train_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "    # end-snippet-1\n",
    "\n",
    "    ###############\n",
    "    # TRAIN MODEL #\n",
    "    ###############\n",
    "    print '... training'\n",
    "    # early-stopping parameters\n",
    "    patience = 10000  # look as this many examples regardless\n",
    "    patience_increase = 2  # wait this much longer when a new best is\n",
    "                           # found\n",
    "    improvement_threshold = 0.995  # a relative improvement of this much is\n",
    "                                   # considered significant\n",
    "    validation_frequency = min(n_train_batches, patience / 2)\n",
    "                                  # go through this many\n",
    "                                  # minibatche before checking the network\n",
    "                                  # on the validation set; in this case we\n",
    "                                  # check every epoch\n",
    "\n",
    "    best_validation_loss = numpy.inf\n",
    "    best_iter = 0\n",
    "    test_score = 0.\n",
    "    start_time = timeit.default_timer()\n",
    "\n",
    "    epoch = 0\n",
    "    done_looping = False\n",
    "\n",
    "    while (epoch < n_epochs) and (not done_looping):\n",
    "        epoch = epoch + 1\n",
    "        for minibatch_index in xrange(n_train_batches):\n",
    "            \n",
    "            # This function is very similar to range(), but returns an xrange object instead of a list. \n",
    "            # This is an opaque sequence type which yields the same values as the corresponding list, \n",
    "            # without actually storing them all simultaneously. The advantage of xrange() over range() \n",
    "            # is minimal (since xrange() still has to create the values when asked for them) except when a \n",
    "            # very large range is used on a memory-starved machine or when all of the range’s elements \n",
    "            # are never used (such as when the loop is usually terminated with break). \n",
    "            # For more information on xrange objects, see XRange Type and Sequence Types — str, \n",
    "            # unicode, list, tuple, bytearray, buffer, xrange\n",
    "\n",
    "            iter = (epoch - 1) * n_train_batches + minibatch_index\n",
    "            \n",
    "            # for epoch = 1 (first value while entering the \"while\" loop; iter = 0 * n_train_batches + minibtach_index\n",
    "            # so iter = 0. This will call train_model over the index of train_set_x[0:500] and train_set_y[0:500].\n",
    "            # the (epoch -1) * n_train_batches keep track of the iteration number while looping over and over on\n",
    "            # the train set.\n",
    "\n",
    "            if iter % 100 == 0:\n",
    "                print 'training @ iter = ', iter\n",
    "            cost_ij = train_model(minibatch_index)\n",
    "            \n",
    "            # Only at this moment all the symbolic expression that were called during \"Building the model\" are\n",
    "            # called with real values replacing the symbolic tensors. That is how theano works.\n",
    "\n",
    "            if (iter + 1) % validation_frequency == 0:\n",
    "\n",
    "                # compute zero-one loss on validation set\n",
    "                validation_losses = [validate_model(i) for i\n",
    "                                     in xrange(n_valid_batches)]\n",
    "                this_validation_loss = numpy.mean(validation_losses)\n",
    "                print('epoch %i, minibatch %i/%i, validation error %f %%' %\n",
    "                      (epoch, minibatch_index + 1, n_train_batches,\n",
    "                       this_validation_loss * 100.))\n",
    "\n",
    "                # if we got the best validation score until now\n",
    "                if this_validation_loss < best_validation_loss:\n",
    "\n",
    "                    #improve patience if loss improvement is good enough\n",
    "                    if this_validation_loss < best_validation_loss *  \\\n",
    "                       improvement_threshold:\n",
    "                        patience = max(patience, iter * patience_increase)\n",
    "\n",
    "                    # save best validation score and iteration number\n",
    "                    best_validation_loss = this_validation_loss\n",
    "                    best_iter = iter\n",
    "\n",
    "                    # test it on the test set\n",
    "                    test_losses = [\n",
    "                        test_model(i)\n",
    "                        for i in xrange(n_test_batches)\n",
    "                    ]\n",
    "                    test_score = numpy.mean(test_losses)\n",
    "                    print(('     epoch %i, minibatch %i/%i, test error of '\n",
    "                           'best model %f %%') %\n",
    "                          (epoch, minibatch_index + 1, n_train_batches,\n",
    "                           test_score * 100.))\n",
    "\n",
    "            if patience <= iter:\n",
    "                done_looping = True\n",
    "                break\n",
    "\n",
    "    end_time = timeit.default_timer()\n",
    "    print('Optimization complete.')\n",
    "    print('Best validation score of %f %% obtained at iteration %i, '\n",
    "          'with test performance %f %%' %\n",
    "          (best_validation_loss * 100., best_iter + 1, test_score * 100.))\n",
    "    print >> sys.stderr, ('The code for file ' +\n",
    "                          os.path.split(__file__)[1] +\n",
    "                          ' ran for %.2fm' % ((end_time - start_time) / 60.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C. Implementation of Learning Rate Decay\n",
    "\n",
    "Let's modify the code of evaluate_lenet5 function so it allows *Learning Rate Decay*.\n",
    "\n",
    "**Definition:** the *learning rate* is the step-size of the update of the parameters during gradient descent. It is typically between 0.1 and 0.01. However, if it is too big, gradient descent can overshoot the minimum and diverge. If it is too small, the optimization is very slow and may get stuck into a local minimum. The *learning rate decay* allows for the learning rate to be big at the beginning and then slowly decrease when nearing the global minimum: \n",
    "\n",
    "- initial learning rate: $$\\alpha = \\alpha_0$$\n",
    "- learning rate decay: $$\\alpha_d$$\n",
    "- at each iteration update: $$\\alpha = \\alpha_d*\\alpha$$\n",
    "\n",
    "The **full code** can be found at code/**convolutional_mlp_ldr.py**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_lenet5_ldr(learning_rate=0.1, learning_rate_decay = 0.98, n_epochs=200,\n",
    "                    dataset='mnist.pkl.gz',\n",
    "                    nkerns=[20, 50], batch_size=500):\n",
    "    \"\"\" \n",
    "    :type learning_rate_decay: float\n",
    "    :param learning_rate_decay: learning rate decay used \n",
    "    \"\"\"\n",
    "\n",
    "    rng = numpy.random.RandomState(23455)\n",
    "\n",
    "   \"\"\"\n",
    "   ...\n",
    "   \"\"\"\n",
    "    \n",
    "    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!#\n",
    "    # Theano function to decay the learning rate, this is separate from the\n",
    "    # training function because we only want to do this once each epoch instead\n",
    "    # of after each minibatch.\n",
    "    decay_learning_rate = theano.function(inputs=[], outputs=learning_rate,\n",
    "            updates={learning_rate: learning_rate * learning_rate_decay})\n",
    "    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!#\n",
    "\n",
    "    ###############\n",
    "    # TRAIN MODEL #\n",
    "    ###############\n",
    "    \n",
    "    \"\"\"\n",
    "    ...\n",
    "    \"\"\"\n",
    "\n",
    "    while (epoch < n_epochs) and (not done_looping):\n",
    "        epoch = epoch + 1\n",
    "        for minibatch_index in xrange(n_train_batches):\n",
    "\n",
    "            iter = (epoch - 1) * n_train_batches + minibatch_index\n",
    "            \n",
    "            if iter % 100 == 0:\n",
    "                print 'training @ iter = ', iter\n",
    "            cost_ij = train_model(minibatch_index)\n",
    "\n",
    "            if (iter + 1) % validation_frequency == 0:\n",
    "\n",
    "                # compute zero-one loss on validation set\n",
    "                validation_losses = [validate_model(i) for i\n",
    "                                     in xrange(n_valid_batches)]\n",
    "                this_validation_loss = numpy.mean(validation_losses)\n",
    "                print('epoch %i, minibatch %i/%i, validation error %f %%' %\n",
    "                      (epoch, minibatch_index + 1, n_train_batches,\n",
    "                       this_validation_loss * 100.))\n",
    "\n",
    "                # if we got the best validation score until now\n",
    "                if this_validation_loss < best_validation_loss:\n",
    "\n",
    "                    #improve patience if loss improvement is good enough\n",
    "                    if this_validation_loss < best_validation_loss *  \\\n",
    "                       improvement_threshold:\n",
    "                        patience = max(patience, iter * patience_increase)\n",
    "\n",
    "                    # save best validation score and iteration number\n",
    "                    best_validation_loss = this_validation_loss\n",
    "                    best_iter = iter\n",
    "\n",
    "                    # test it on the test set\n",
    "                    test_losses = [\n",
    "                        test_model(i)\n",
    "                        for i in xrange(n_test_batches)\n",
    "                    ]\n",
    "                    test_score = numpy.mean(test_losses)\n",
    "                    print(('     epoch %i, minibatch %i/%i, test error of '\n",
    "                           'best model %f %%') %\n",
    "                          (epoch, minibatch_index + 1, n_train_batches,\n",
    "                           test_score * 100.))\n",
    "                \n",
    "                #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n",
    "                new_learning_rate = decay_learning_rate()\n",
    "                #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n",
    "\n",
    "            if patience <= iter:\n",
    "                done_looping = True\n",
    "                break\n",
    "\n",
    "\"\"\"\n",
    "...\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## D. Implementation of dropout\n",
    "\n",
    "Dropout is a technique that was presented in [G. Hinton's work](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) \"Dropout: A simple Way to Prevent Neural Networks from Overfitting\". As can be read in the abstract:\n",
    "\n",
    "> Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different “thinned” networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights\n",
    "\n",
    "<img src=\"images/hinton1.png\" width=650>\n",
    "\n",
    "> Dropping out 20% of the input units and 50% of the hidden units was often found to be optimal\n",
    "\n",
    "Implementation solution presented here is greatly **inspired from GitHub user [mdenil work on dropout](https://github.com/mdenil/dropout)**, who implemented Hinton's dropout on a Multi-Layer Perceptron (base code from [deeplearning.net](http://deeplearning.net/tutorial/mlp.html)).\n",
    "\n",
    "### 1. Creating dropout function\n",
    "\n",
    "This function takes a layer (which can be either a layer of units in an MLP or a layer of feature maps in a CNN) and drop units from the layer with a probability of p (or in the case of CNN pixels from feature maps with a probability of p)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _dropout_from_layer(rng, layer, p):\n",
    "    \"\"\"p is the probablity of dropping a unit\n",
    "    \"\"\"\n",
    "    srng = theano.tensor.shared_randomstreams.RandomStreams(\n",
    "            rng.randint(999999))\n",
    "    # p=1-p because 1's indicate keep and p is probability of dropping\n",
    "    mask = srng.binomial(n=1, p=1-p, size=layer.shape)\n",
    "    # The cast is important because\n",
    "    # int * float32 = float64 which pulls things off the gpu\n",
    "    output = layer * T.cast(mask, theano.config.floatX)\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Creating dropout classes\n",
    "\n",
    "We create child classes from `HiddenLayer` and `LeNetConvPoolLayer` so that they take into account dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DropoutHiddenLayer(HiddenLayer):\n",
    "    def __init__(self, rng, input, n_in, n_out,\n",
    "                 activation, dropout_rate, W=None, b=None):\n",
    "        super(DropoutHiddenLayer, self).__init__(\n",
    "                rng=rng, input=input, n_in=n_in, n_out=n_out, W=W, b=b,\n",
    "                activation=activation)\n",
    "\n",
    "        self.output = _dropout_from_layer(rng, self.output, p=dropout_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DropoutLeNetConvPoolLayer(LeNetConvPoolLayer):\n",
    "    def __init__(self, rng, input, filter_shape, image_shape, poolsize,\n",
    "               dropout_rate, W=None, b=None):\n",
    "    super(DropoutLeNetConvPoolLayer, self).__init__(\n",
    "      rng=rng, input=input, filter_shape=filter_shape, image_shape=image_shape,\n",
    "      poolsize=poolsize, W=W, b=b)\n",
    "\n",
    "    self.output = _dropout_from_layer(rng, self.output, p=dropout_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** we dropout pixels after pooling.\n",
    "\n",
    "### 3. Rewriting evaluate_lenet5\n",
    "\n",
    "Each time a layer is instantiated, two actual layers need to be actually created in parallel: the dropout layer which drops out some of its units with a probability of p, and an associated layer sharing the same coefficient W and b except W is scaled using p.\n",
    "\n",
    "Again, **full code** can be found at code/**convolutional_mlp_dropout.py**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_lenet5(initial_learning_rate=0.1, learning_rate_decay = 1, \n",
    "                    dropout_rates = [0.2, 0.2, 0.2, 0.5], n_epochs=200,\n",
    "                    dataset='mnist.pkl.gz',\n",
    "                    nkerns=[20, 50], batch_size=500):\n",
    "    \"\"\"\n",
    "    :type dropout_rates: list of float\n",
    "    :param dropout_rates: dropout rate used for each layer (input layer, \n",
    "    1st filtered layer, 2nd filtered layer, fully connected layer)\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    \"\"\"\n",
    "    ...\n",
    "    \"\"\"\n",
    "\n",
    "    ######################\n",
    "    # BUILD ACTUAL MODEL #\n",
    "    ######################\n",
    "    print '... building the model'\n",
    "\n",
    "    # Reshape matrix of rasterized images of shape (batch_size, 28 * 28)\n",
    "    # to a 4D tensor, compatible with our LeNetConvPoolLayer\n",
    "    # (28, 28) is the size of MNIST images.\n",
    "    layer0_input = x.reshape((batch_size, 1, 28, 28))\n",
    "    # Dropping out pixels from original image randomly, with a probability of dropping\n",
    "    # low enough not too drop too much information (20% was found to be ideal)\n",
    "    layer0_input_dropout = _dropout_from_layer(rng, layer0_input, dropout_rates[0])\n",
    "\n",
    "\n",
    "    # Construct the first convolutional pooling layer:\n",
    "    # filtering reduces the image size to (28-5+1 , 28-5+1) = (24, 24)\n",
    "    # maxpooling reduces this further to (24/2, 24/2) = (12, 12)\n",
    "    # 4D output tensor is thus of shape (batch_size, nkerns[0], 12, 12)\n",
    "    layer0_dropout = DropoutLeNetConvPoolLayer(\n",
    "        rng,\n",
    "        input=layer0_input_dropout,\n",
    "        image_shape=(batch_size, 1, 28, 28),\n",
    "        filter_shape=(nkerns[0], 1, 5, 5),\n",
    "        poolsize=(2, 2),\n",
    "        dropout_rate= dropout_rates[1]\n",
    "    )\n",
    "    \n",
    "    # Creating in parallel a normal LeNetConvPoolLayer that share the same\n",
    "    # W and b as the dropout layer, with W scaled with p.\n",
    "    layer0 = LeNetConvPoolLayer(\n",
    "      rng,\n",
    "      input=layer0_input,\n",
    "      image_shape=(batch_size, 1, 28, 28),\n",
    "      filter_shape=(nkerns[0], 1, 5, 5),\n",
    "      poolsize=(2, 2),\n",
    "      W=layer0_dropout.W * (1 - dropout_rates[0]),\n",
    "      b=layer0_dropout.b\n",
    "    )\n",
    "\n",
    "    # Construct the second convolutional pooling layer\n",
    "    # filtering reduces the image size to (12-5+1, 12-5+1) = (8, 8)\n",
    "    # maxpooling reduces this further to (8/2, 8/2) = (4, 4)\n",
    "    # 4D output tensor is thus of shape (batch_size, nkerns[1], 4, 4)\n",
    "    layer1_dropout = DropoutLeNetConvPoolLayer(\n",
    "        rng,\n",
    "        input=layer0_dropout.output,\n",
    "        image_shape=(batch_size, nkerns[0], 12, 12),\n",
    "        filter_shape=(nkerns[1], nkerns[0], 5, 5),\n",
    "        poolsize=(2, 2),\n",
    "        dropout_rate = dropout_rates[2]\n",
    "    )\n",
    "\n",
    "    layer1 = LeNetConvPoolLayer(\n",
    "      rng,\n",
    "      input=layer0.output,\n",
    "      image_shape=(batch_size, nkerns[0], 12, 12),\n",
    "      filter_shape=(nkerns[1], nkerns[0], 5, 5),\n",
    "      poolsize=(2, 2),\n",
    "      W=layer1_dropout.W * (1 - dropout_rates[1]),\n",
    "      b=layer1_dropout.b\n",
    "    )\n",
    "\n",
    "    # the HiddenLayer being fully-connected, it operates on 2D matrices of\n",
    "    # shape (batch_size, num_pixels) (i.e matrix of rasterized images).\n",
    "    # This will generate a matrix of shape (batch_size, nkerns[1] * 4 * 4),\n",
    "    # or (500, 50 * 4 * 4) = (500, 800) with the default values.\n",
    "    layer2_dropout_input = layer1_dropout.output.flatten(2)\n",
    "    layer2_input = layer1.output.flatten(2)\n",
    "\n",
    "    # construct a fully-connected sigmoidal layer\n",
    "    layer2_dropout = DropoutHiddenLayer(\n",
    "        rng,\n",
    "        input=layer2_dropout_input,\n",
    "        n_in=nkerns[1] * 4 * 4,\n",
    "        n_out=500,\n",
    "        activation=T.tanh,\n",
    "        dropout_rate = dropout_rates[3]\n",
    "    )\n",
    "\n",
    "    layer2 = HiddenLayer(\n",
    "      rng,\n",
    "      input=layer2_input,\n",
    "      n_in=nkerns[1] * 4 * 4,\n",
    "      n_out=500,\n",
    "      activation=T.tanh,\n",
    "      W=layer2_dropout.W * (1 - dropout_rates[2]),\n",
    "      b=layer2_dropout.b\n",
    "    )\n",
    "\n",
    "\n",
    "    # classify the values of the fully-connected sigmoidal layer\n",
    "    layer3_dropout = LogisticRegression(\n",
    "      input = layer2_dropout.output,\n",
    "      n_in = 500, n_out = 10)\n",
    "\n",
    "    layer3 = LogisticRegression(\n",
    "      input=layer2.output,\n",
    "      n_in=500, n_out=10,\n",
    "      W=layer3_dropout.W * (1 - dropout_rates[-1]),\n",
    "      b=layer3_dropout.b\n",
    "    )\n",
    "\n",
    "\n",
    "    # the cost we minimize during training is the NLL of the model\n",
    "    cost = layer3.negative_log_likelihood(y)\n",
    "    dropout_cost = layer3_dropout.negative_log_likelihood(y)\n",
    "\n",
    "    # create a function to compute the mistakes that are made by the model\n",
    "    test_model = theano.function(\n",
    "        [index],\n",
    "        layer3.errors(y),\n",
    "        givens={\n",
    "            x: test_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: test_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    validate_model = theano.function(\n",
    "        [index],\n",
    "        layer3.errors(y),\n",
    "        givens={\n",
    "            x: valid_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: valid_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    # create a list of all model parameters to be fit by gradient descent\n",
    "    params = layer3_dropout.params + layer2_dropout.params + layer1_dropout.params + layer0_dropout.params\n",
    "\n",
    "    # create a list of gradients for all model parameters\n",
    "    grads = T.grad(dropout_cost, params)\n",
    "\n",
    "    # train_model is a function that updates the model parameters by SGD\n",
    "    updates = [\n",
    "        (param_i, param_i - learning_rate * grad_i)\n",
    "        for param_i, grad_i in zip(params, grads)\n",
    "    ]\n",
    "\n",
    "    train_model = theano.function(\n",
    "        [index],\n",
    "        dropout_cost,\n",
    "        updates=updates,\n",
    "        givens={\n",
    "            x: train_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: train_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    \"\"\"\n",
    "    ...\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the code for 50 epochs (237 minutes of computation) we get:\n",
    "\n",
    "`Best validation score of 1.560000 % obtained at iteration 5000, with test performance 1.450000 %`\n",
    "\n",
    "Full **result file** at results/**dropout_good_percent.txt**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## E. Visualization of the convolutional filters\n",
    "\n",
    "Read this article on [understanding the convolutional neural networks]('http://cs231n.github.io/understanding-cnn/'). Many methods of visualization what the convolutional networks learned are descrived. We will retain the first one, as it is the most straight-forward to implement:\n",
    "\n",
    "**Visualizing the activations and first-layer weights:**\n",
    "\n",
    "- **Layer Activations**: the most straight-forward visualization technique is to show the activations of the network during the forward pass. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse and localized. One dangerous pitfall that can be easily noticed with this visualization is that some activation maps may be all zero for many different inputs, which can indicate dead filters, and can be a symptom of high learning rates.\n",
    "\n",
    "- **Conv/FC Filters**: The second common strategy is to visualize the weights. These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn't been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.\n",
    "\n",
    "I would like to visualize the filters, so implement the second most common strategy to see the **first 20 filters**.\n",
    "\n",
    "M. D. Zeiler wrote an [interesting paper](http://arxiv.org/abs/1311.2901) about Deconvolutional Networks (DeConvNet) for visualizing and understanding convolutional filter. The only code I found for this subject can be found [here](https://github.com/ChienliMa/DeConvNet).\n",
    "\n",
    "### 1. Visualization function\n",
    "\n",
    "Let's create a function that displays the weight of the filters if fed the weight parameter W."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pylab\n",
    "from PIL import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def display_filter(W, n_cols = 5):\n",
    "    \n",
    "    \"\"\"\n",
    "    :type W: numpy_nd_array\n",
    "    :param W: parameter W of a convolutional + max pooling layer\n",
    "    \n",
    "    :type image_width: int\n",
    "    : param image_width: width of the final image representing the different filters\n",
    "    \"\"\"\n",
    "    \n",
    "    W_shape = W.shape\n",
    "    n_filters = W_shape[0]\n",
    "    \n",
    "    #param filter_shape: (number of filters, num input feature maps, filter height, filter width)\n",
    "    filter_height = W_shape[2]\n",
    "    filter_width = W_shape[3]\n",
    "    \n",
    "    n_lines = numpy.ceil(n_filters / n_cols)\n",
    "    \n",
    "    for n in range(n_filters):\n",
    "        Wn = W[n,0,:,:]\n",
    "        Wn = Wn / Wn.max() # Scaling W to get 0-1 gray scale \n",
    "        pylab.subplot(n_lines, n_cols, n + 1); pylab.axis('off'); pylab.imshow(W[n,0,:,:], cmap=pylab.gray())\n",
    "    pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Testing the function on a single untrained LeNetConvPoolLayer\n",
    "\n",
    "To test the function let's take back the example of the image of me when I was 5 years old. I will feed it to a `LeNetConvPoolLayer`, retrieve the weights, and display them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rng = numpy.random.RandomState(1234)\n",
    "\n",
    "img = Image.open(open('images/profilepic4.jpg'))\n",
    "img = numpy.asarray(img, dtype='float64') / 256. # divide by 256 to have RGB 0-1 scale and not 0 - 256\n",
    "img_ = img.transpose(2, 0, 1).reshape(1, 3, 2592, 1936)\n",
    "\n",
    "input = img_\n",
    "\n",
    "filter_shape = [20,3,12,12]\n",
    "image_shape = [1,3,2592,1936]\n",
    "\n",
    "poolsize = (2, 2)\n",
    "\n",
    "layer_test = LeNetConvPoolLayer(rng, input, filter_shape, image_shape, poolsize)\n",
    "\n",
    "f = theano.function([], layer_test.params)\n",
    "\n",
    "W = f[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "display_filter(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/filters2.png\" width = 400 >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the **weights are randomly initialized** we of course see **random pattern** in each filter.\n",
    "\n",
    "### 3. Displaying the learned filters after training\n",
    "\n",
    "Let's now modify the code of evaluate_lenet5 so that it displays the filters after training. Full code can be found at code/**filter_visualization.py**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_lenet5(initial_learning_rate=0.1, learning_rate_decay = 1, \n",
    "                    dropout_rates = [0.2, 0.2, 0.2, 0.5], n_epochs=200,\n",
    "                    dataset='mnist.pkl.gz', display_filters = True,\n",
    "                    nkerns=[20, 50], batch_size=500):\n",
    "    \"\"\"\n",
    "    :type display_filters: Bool\n",
    "    :param display_filters: True if we want to display the learned filters after training\n",
    "    \n",
    "    \n",
    "    we skip to the very end of the code, after training is done\n",
    "    \"\"\"\n",
    "    \n",
    "    if display_filters:\n",
    "        # Retrieving the filters from first and second layer\n",
    "        first_convlayer_params = theano.function([], layer0_dropout.params)\n",
    "        second_convlayer_params = theano.function([], layer1_dropout.params)\n",
    "        \n",
    "        W0 = first_convlayer_params[0]\n",
    "        W1 = second_convlayer_params[0]\n",
    "        \n",
    "        # Display filters from first layer (20 filters)\n",
    "        display_filter(W0)\n",
    "        \n",
    "        # Display filters from second layer (50 filters)\n",
    "        display_filter(W1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filters from the first layer after 50 epochs (159.21 minutes of computation):\n",
    "\n",
    "<img src=\"images/filters_50epoch_1.png\" width = 400>\n",
    "\n",
    "The filters from the second layer after 50 epochs:\n",
    "\n",
    "<img src=\"images/filters_50epoch_2.png\">\n",
    "\n",
    "We can see **less randomness and patterns appearing on some filters**.\n",
    "\n",
    "## F. Automated creation of a CNN + MLP\n",
    "\n",
    "The idea is to create a `cnn_mlp class` that enable the direct instatiation of a CNN followed by a MLP, in order to have a **cleaner, more flexible code**. The full code will not be presented here but can be found at code/**cnn_mlp.py**. It is greatly inspired from mlp.py created by [mdenil](https://github.com/mdenil/dropout)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}