deep-learning-with-python-notebooks/2.1-a-first-look-at-a-neural-network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文代码作者：François Chollet\n",
    "\n",
    "github：https://github.com/fchollet/deep-learning-with-python-notebooks\n",
    "\n",
    "中文注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：keras 2.2.1（原文是2.0.8，运行结果一致），tensorflow 1.8，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](data/gongzhong.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A first look at a neural network\n",
    "# 初识神经网络\n",
    "\n",
    "This notebook contains the code samples found in Chapter 2, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "We will now take a look at a first concrete example of a neural network, which makes use of the Python library Keras to learn to classify \n",
    "hand-written digits. Unless you already have experience with Keras or similar libraries, you will not understand everything about this \n",
    "first example right away. You probably haven't even installed Keras yet. Don't worry, that is perfectly fine. In the next chapter, we will \n",
    "review each element in our example and explain them in detail. So don't worry if some steps seem arbitrary or look like magic to you! \n",
    "We've got to start somewhere.\n",
    "\n",
    "The problem we are trying to solve here is to classify grayscale images of handwritten digits (28 pixels by 28 pixels), into their 10 \n",
    "categories (0 to 9). The dataset we will use is the MNIST dataset, a classic dataset in the machine learning community, which has been \n",
    "around for almost as long as the field itself and has been very intensively studied. It's a set of 60,000 training images, plus 10,000 test \n",
    "images, assembled by the National Institute of Standards and Technology (the NIST in MNIST) in the 1980s. You can think of \"solving\" MNIST \n",
    "as the \"Hello World\" of deep learning -- it's what you do to verify that your algorithms are working as expected. As you become a machine \n",
    "learning practitioner, you will see MNIST come up over and over again, in scientific papers, blog posts, and so on.\n",
    "\n",
    "我们来看一个具体的神经网络示例，使用 Python 的 Keras 库来学习手写数字分类。如果你 没用过 Keras 或类似的库，可能无法立刻搞懂这个例子中的全部内容。甚至你可能还没有安装 Keras。没关系，下一章会详细解释这个例子中的每个步骤。因此，如果其中某些步骤看起来有 些随意，或者像魔法一样，也请你不要担心。下面我们要开始了。\n",
    "\n",
    "我们这里要解决的问题是，将手写数字的灰度图像（28 像素×28 像素）划分到 10 个类别 中（0~9）。我们将使用 MNIST 数据集，它是机器学习领域的一个经典数据集，其历史几乎和这 个领域一样长，而且已被人们深入研究。这个数据集包含 60 000 张训练图像和 10 000 张测试图 像，由美国国家标准与技术研究院（National Institute of Standards and Technology，即 MNIST 中 的 NIST）在 20 世纪 80 年代收集得到。你可以将“解决”MNIST 问题看作深度学习的“Hello World”，正是用它来验证你的算法是否按预期运行。当你成为机器学习从业者后，会发现 MNIST 一次又一次地出现在科学论文、博客文章等中。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset comes pre-loaded in Keras, in the form of a set of four Numpy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 5s 0us/step\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
    "#这里下载要多试几次，不翻墙能下。train_images 和 train_labels 组成了训练集（training set），模型将从这些数据中进行学习。\n",
    "#然后在测试集（test set，即 test_images 和 test_labels）上对模型进行测试。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`train_images` and `train_labels` form the \"training set\", the data that the model will learn from. The model will then be tested on the \n",
    "\"test set\", `test_images` and `test_labels`. Our images are encoded as Numpy arrays, and the labels are simply an array of digits, ranging \n",
    "from 0 to 9. There is a one-to-one correspondence between the images and the labels.\n",
    "\n",
    "Let's have a look at the training data:\n",
    "\n",
    "train_images 和 train_labels 组成了训练集（training set），模型将从这些数据中进行学习。然后在测试集（test set，即 test_images 和 test_labels）上对模型进行测试。\n",
    "\n",
    "图像被编码为 Numpy 数组，而标签是数字数组，取值范围为 0~9。图像和标签一一对应。 \n",
    "\n",
    "我们来看一下训练数据：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_images.shape#看下数据的维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 4, ..., 5, 6, 8], dtype=uint8)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the test data:\n",
    "\n",
    "看下测试数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7, 2, 1, ..., 4, 5, 6], dtype=uint8)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our workflow will be as follow: first we will present our neural network with the training data, `train_images` and `train_labels`. The \n",
    "network will then learn to associate images and labels. Finally, we will ask the network to produce predictions for `test_images`, and we \n",
    "will verify if these predictions match the labels from `test_labels`.\n",
    "\n",
    "Let's build our network -- again, remember that you aren't supposed to understand everything about this example just yet.\n",
    "\n",
    "接下来的工作流程如下：\n",
    "\n",
    "首先，将训练数据（train_images 和 train_labels）输入神经网络；\n",
    "\n",
    "其次，网络学习将图像和标签关联在一起；\n",
    "\n",
    "最后，网络对 test_images 生成预测， 而我们将验证这些预测与 test_labels 中的标签是否匹配。\n",
    "\n",
    "下面我们来构建网络。再说一遍，你现在不需要理解这个例子的全部内容。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "network = models.Sequential()\n",
    "network.add(layers.Dense(512, activation='relu', input_shape=(28 * 28,)))\n",
    "network.add(layers.Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The core building block of neural networks is the \"layer\", a data-processing module which you can conceive as a \"filter\" for data. Some \n",
    "data comes in, and comes out in a more useful form. Precisely, layers extract _representations_ out of the data fed into them -- hopefully \n",
    "representations that are more meaningful for the problem at hand. Most of deep learning really consists of chaining together simple layers \n",
    "which will implement a form of progressive \"data distillation\". A deep learning model is like a sieve for data processing, made of a \n",
    "succession of increasingly refined data filters -- the \"layers\".\n",
    "\n",
    "Here our network consists of a sequence of two `Dense` layers, which are densely-connected (also called \"fully-connected\") neural layers. \n",
    "The second (and last) layer is a 10-way \"softmax\" layer, which means it will return an array of 10 probability scores (summing to 1). Each \n",
    "score will be the probability that the current digit image belongs to one of our 10 digit classes.\n",
    "\n",
    "To make our network ready for training, we need to pick three more things, as part of \"compilation\" step:\n",
    "\n",
    "* A loss function: the is how the network will be able to measure how good a job it is doing on its training data, and thus how it will be \n",
    "able to steer itself in the right direction.\n",
    "* An optimizer: this is the mechanism through which the network will update itself based on the data it sees and its loss function.\n",
    "* Metrics to monitor during training and testing. Here we will only care about accuracy (the fraction of the images that were correctly \n",
    "classified).\n",
    "\n",
    "The exact purpose of the loss function and the optimizer will be made clear throughout the next two chapters.\n",
    "\n",
    "神经网络的核心组件是层（layer），它是一种数据处理模块，你可以将它看成数据过滤器。 进去一些数据，出来的数据变得更加有用。具体来说，层从输入数据中提取表示——我们期望这种表示有助于解决手头的问题。大多数深度学习都是将简单的层链接起来，从而实现渐进式 的数据蒸馏（data distillation）。深度学习模型就像是数据处理的筛子，包含一系列越来越精细的数据过滤器（即层）。\n",
    "\n",
    "本例中的网络包含 2 个 Dense 层，它们是密集连接（也叫全连接）的神经层。第二层（也 是最后一层）是一个 10 路 softmax 层，它将返回一个由 10 个概率值（总和为 1）组成的数组。每个概率值表示当前数字图像属于 10 个数字类别中某一个的概率。\n",
    "\n",
    "要想训练网络，我们还需要选择编译（compile）步骤的三个参数。\n",
    "\n",
    "* 损失函数（loss function）：网络如何衡量在训练数据上的性能，即网络如何朝着正确的方向前进。\n",
    "\n",
    "* 优化器（optimizer）：基于训练数据和损失函数来更新网络的机制。\n",
    "\n",
    "* 在训练和测试过程中需要监控的指标（metric）：本例只关心精度，即正确分类的图像所占的比例。\n",
    "\n",
    "后续两章会详细解释损失函数和优化器的确切用途。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "network.compile(optimizer='rmsprop',\n",
    "                loss='categorical_crossentropy',\n",
    "                metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Before training, we will preprocess our data by reshaping it into the shape that the network expects, and scaling it so that all values are in \n",
    "the `[0, 1]` interval. Previously, our training images for instance were stored in an array of shape `(60000, 28, 28)` of type `uint8` with \n",
    "values in the `[0, 255]` interval. We transform it into a `float32` array of shape `(60000, 28 * 28)` with values between 0 and 1.\n",
    "\n",
    "在开始训练之前，我们将对数据进行预处理，将其变换为网络要求的形状，并缩放到所 有值都在 [0, 1] 区间。比如，之前训练图像保存在一个 uint8 类型的数组中，其形状为 (60000, 28, 28)，取值区间为 [0, 255]。我们需要将其变换为一个 float32 数组，其形 状为 (60000, 28 * 28)，取值范围为 0~1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype('float32') / 255\n",
    "\n",
    "test_images = test_images.reshape((10000, 28 * 28))\n",
    "test_images = test_images.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to categorically encode the labels, a step which we explain in chapter 3:\n",
    "\n",
    "我们还需要对标签进行分类编码，第 3 章将会对这一步骤进行解释。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical\n",
    "\n",
    "train_labels = to_categorical(train_labels)\n",
    "test_labels = to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to train our network, which in Keras is done via a call to the `fit` method of the network: \n",
    "we \"fit\" the model to its training data.\n",
    "\n",
    "现在我们准备开始训练网络，在 Keras 中这一步是通过调用网络的 fit 方法来完成的——\t2\n",
    "我们在训练数据上拟合（fit）模型。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 5s 87us/step - loss: 0.2587 - acc: 0.9251\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 0.1042 - acc: 0.9687\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 2s 32us/step - loss: 0.0680 - acc: 0.9796\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 0.0494 - acc: 0.9853\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 0.0374 - acc: 0.9887\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x29fb6500400>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.fit(train_images, train_labels, epochs=5, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two quantities are being displayed during training: the \"loss\" of the network over the training data, and the accuracy of the network over \n",
    "the training data.\n",
    "\n",
    "We quickly reach an accuracy of 0.989 (i.e. 98.9%) on the training data. Now let's check that our model performs well on the test set too:\n",
    "\n",
    "训练过程中显示了两个数字：一个是网络在训练数据上的损失（loss），另一个是网络在 训练数据上的精度（acc）。\n",
    "我们很快就在训练数据上达到了 0.989（98.9%）的精度。现在我们来检查一下模型在测试 集上的性能。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 46us/step\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = network.evaluate(test_images, test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_acc: 0.9785\n"
     ]
    }
   ],
   "source": [
    "print('test_acc:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Our test set accuracy turns out to be 97.8% -- that's quite a bit lower than the training set accuracy. \n",
    "This gap between training accuracy and test accuracy is an example of \"overfitting\", \n",
    "the fact that machine learning models tend to perform worse on new data than on their training data. \n",
    "Overfitting will be a central topic in chapter 3.\n",
    "\n",
    "This concludes our very first example -- you just saw how we could build and a train a neural network to classify handwritten digits, in \n",
    "less than 20 lines of Python code. In the next chapter, we will go in detail over every moving piece we just previewed, and clarify what is really \n",
    "going on behind the scenes. You will learn about \"tensors\", the data-storing objects going into the network, about tensor operations, which \n",
    "layers are made of, and about gradient descent, which allows our network to learn from its training examples.\n",
    "\n",
    "测试集精度为 97.8%，比训练集精度低不少。训练精度和测试精度之间的这种差距是过拟合（overfit）造成的。过拟合是指机器学习模型在新数据上的性能往往比在训练数据上要差，它 是第 3 章的核心主题。\n",
    "\n",
    "第一个例子到这里就结束了。你刚刚看到了如何构建和训练一个神经网络，用不到 20 行的 Python 代码对手写数字进行分类。下一章会详细介绍这个例子中的每一个步骤，并讲解其背后的原理。接下来你将要学到张量（输入网络的数据存储对象）、张量运算（层的组成要素）和梯度下降（可以让网络从训练样本中进行学习）。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}