06-Everything-about-TF2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Everything about Tensorflow 2.x\n",
    "\n",
    "## Eager execution by default\n",
    "\n",
    "Probably the most wanted update is eager execution as the main way for working with Tensorflow. For those of you who don't know with Tensorflow v1.x the abstract data structures need to be defined in something called a Graph. To then actually, execute the code a session must be used.\n",
    "\n",
    "This system had a steep learning curve because it's so different from the normal way of Python programming. It's also really hard to debug such programs because when printing a graph node we aren't getting a specific value, rather we would only see a reference.\n",
    "\n",
    "With version 2.0 Tensorflow moved away from the graph/session architecture and embraced eager execution.\n",
    "\n",
    "Eager execution is an imperative programming environment that evaluates operations immediately, without building graphs. For more information check out the official eager execution guide.\n",
    "\n",
    "## Keras as the main API for model building\n",
    "\n",
    "Tensorflow 1.x had multiple APIs for building networks. tf.slim, tf.layers, tf.contrib.layers, tf.keras could all be used to create Neural Networks. Most of the time it wasn't clear to beginners which API to use. Furthermore, if you trained a model with one API, it wasn't straight forward to use the code with another API.\n",
    "\n",
    "In Tensorflow 2.0 Keras is the recommended high-level API. Keras is a high-level neural networks API, capable of running on top of Tensorflow, Theano, and CNTK. It enables fast experimentation through a high level, user-friendly, modular and extensible API.\n",
    "\n",
    "## Model Subclassing – The pythonic way of creating models\n",
    "\n",
    "Tensorflow 2.0 also adds support for subclassing. Model subclassing is a user-friendly modular way of creating models or layers.\n",
    "\n",
    "If you are already familiar with PyTorch you might already have heard of subclassing. In PyTorch, you need to overwrite the __init__ and forward methods. In Tensorflow 2.0 the prediction method isn't called forward, rather it's called call.\n",
    "\n",
    "To create a neural network with a convolutional and two fully connected layers the below code can be used:\n",
    "\n",
    "```python\n",
    "class MyModel(Model):\n",
    "  def __init__(self):\n",
    "    super(MyModel, self).__init__()\n",
    "    self.conv1 = Conv2D(32, 3, activation='relu')\n",
    "    self.flatten = Flatten()\n",
    "    self.d1 = Dense(128, activation='relu')\n",
    "    self.d2 = Dense(10, activation='softmax')\n",
    "\n",
    "  def call(self, x):\n",
    "    x = self.conv1(x)\n",
    "    x = self.flatten(x)\n",
    "    x = self.d1(x)\n",
    "    return self.d2(x)\n",
    "\n",
    "\n",
    "# Create an instance of the model\n",
    "`model = MyModel()`\n",
    "```\n",
    "\n",
    "## Creating your own layers\n",
    "\n",
    "Tensorflow 2.0 also supports writing your custom layers. A layer encapsulates both a state (the layer's \"weights\") and a transformation from inputs to outputs (a \"call\", the layer's forward pass).\n",
    "\n",
    "Writing your own layer is quite similar to the way you create a model with subclassing only that for a layer you won't inherit from Model. Rather you will inherit from layers.Layer.\n",
    "\n",
    "For example, a linear layer can be created with the following code:\n",
    "\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "\n",
    "```python\n",
    "class Linear(layers.Layer):\n",
    "\n",
    "  def __init__(self, units=32, input_dim=32):\n",
    "    super(Linear, self).__init__()\n",
    "    w_init = tf.random_normal_initializer()\n",
    "    self.w = tf.Variable(initial_value=w_init(shape=(input_dim, units),\n",
    "                                              dtype='float32'),\n",
    "                         trainable=True)\n",
    "    b_init = tf.zeros_initializer()\n",
    "    self.b = tf.Variable(initial_value=b_init(shape=(units,),\n",
    "                                              dtype='float32'),\n",
    "                         trainable=True)\n",
    "\n",
    "  def call(self, inputs):\n",
    "    return tf.matmul(inputs, self.w) + self.b\n",
    "```\n",
    "\n",
    "After creating the layer it can be used as a normal python method:\n",
    "\n",
    "```python\n",
    "x = tf.ones((2, 2))\n",
    "linear_layer = Linear(4, 2)\n",
    "y = linear_layer(x)\n",
    "print(y)\n",
    "```\n",
    "\n",
    "```shell\n",
    "Output:\n",
    "tf.Tensor(\n",
    "[[ 0.04828779  0.07988431 -0.07137472  0.10912687]\n",
    " [ 0.04828779  0.07988431 -0.07137472  0.10912687]], shape=(2, 4), dtype=float32)\n",
    "```\n",
    "\n",
    "## Train your model\n",
    "\n",
    "In Tensorflow 2.0 we have two main ways of training our models. The fit method and the [tf.GradientTape API](https://www.tensorflow.org/api_docs/python/tf/GradientTape).\n",
    "\n",
    "When using Keras Sequential or Model API the fit method can be used without any modifications. For training a Subclassing model with the fit method the comput_output_shape method must be overridden.\n",
    "\n",
    "If you want easier access to the gradients and loss as well as a clearer understanding of the training it might be worth to use tf.GradientTape instead of the fit method.\n",
    "\n",
    "Using GradientTape, one can manually define each training step in the training procedure. The basic steps include:\n",
    "\n",
    "- Forward pass\n",
    "- Calculating loss\n",
    "- Backward pass\n",
    "- Updating the gradients\n",
    "\n",
    "For image classification GradientTape could be used like in the following example:\n",
    "\n",
    "```python\n",
    "@tf.function\n",
    "def train_step(images, labels):\n",
    "  with tf.GradientTape() as tape:\n",
    "    predictions = model(images)\n",
    "    loss = loss_object(labels, predictions)\n",
    "  gradients = tape.gradient(loss, model.trainable_variables)\n",
    "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "\n",
    "  train_loss(loss)\n",
    "  train_accuracy(labels, predictions)\n",
    "```\n",
    "## Converting from Tensorflow v1.x to v2.0\n",
    "\n",
    "Upgrading all your code from Tensorflow v1.x to 2.0 manually would be tedious and error-prone. To streamline the changes, and to make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the tf_upgrade_v2 utility to help transition legacy code to the new API.\n",
    "\n",
    "Typically the utility can be used like:\n",
    "\n",
    "```shell\n",
    "tf_upgrade_v2 \\\n",
    "  --intree my_project/ \\\n",
    "  --outtree my_project_v2/ \\\n",
    "  --reportfile report.txt\n",
    "```\n",
    "In most cases, the tf_upgrade_v2 utility uses the tf.compat.v1 module and doesn't make any further changes.\n",
    "\n",
    "Tensorflow v1.x:\n",
    "\n",
    "```python\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Hello World\n",
    "hello = tf.constant('Hello, TensorFlow!')\n",
    "\n",
    "# Start tf session\n",
    "sess = tf.Session()\n",
    "\n",
    "# Run the op\n",
    "print(sess.run(hello))\n",
    "tf_upgrade_v2 --infile tf1_hello_world.py --outfile tf2_hello_world.py\n",
    "```\n",
    "\n",
    "#### Tensorflow 2.0:\n",
    "\n",
    "```python\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Hello World\n",
    "hello = tf.constant('Hello, TensorFlow!')\n",
    "\n",
    "# Start tf session\n",
    "sess = tf.compat.v1.Session()\n",
    "\n",
    "# Run the op\n",
    "print(sess.run(hello))\n",
    "```\n",
    "For more information about how to update your code-bases check out the [official migration] guide(https://www.tensorflow.org/guide/migrate).\n",
    "\n",
    "## Other changes\n",
    "\n",
    "Besides the changes above Tensorflow 2.0 also significantly cleaned their API for easier usage and added lots of performance updates including multiple GPU support and their Distribution Strategy API, allowing us to distribute training with minimal code changes.\n",
    "\n",
    "Also do check out the [official Tensorflow page](https://www.tensorflow.org/).\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}