07_Attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "After this lecture you will learn:\n",
    "* The concept of attention and its relevance in NNs\n",
    "* How attention is used in seq2seq with RNNs\n",
    "* How attention is used in seq2seq without RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sequence-to-sequence (seq2seq) model [(Sutskever et al., 2014)](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf):\n",
    "* One RNN reads (encodes) the source sentence S into a fixed-length vector\n",
    "* Another RNN produces a target sentence T from the encoded vector (e.g. a translation)\n",
    "Note that $length(S)$ not necessarily equal to $length(T)$\n",
    "<img src=\"pics/seq2seq_lite.png\">\n",
    "What is the problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We put the meaning of a whole sentence into a fixed-length vector. This does not work for long sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Attention in seq2seq with RNNs [(Bahdanau et al., 2014)](https://arxiv.org/pdf/1409.0473.pdf)\n",
    "* Consider all the hidden states from the encoder (not only the last one)\n",
    "* When decoding, the NN may decide to pay more **attention** to some source tokens than to others\n",
    "\n",
    "<img src=\"pics/seq2seq_attention.png\" width=\"30%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"pics/seq2seq_attention.png\" width=\"15%\">\n",
    "\n",
    "* Input: $x=[x_1, x_2, ..., x_n]$\n",
    "* Output: $y=[y_1, y_2, ..., y_m]$\n",
    "* Encoder's hidden-state: $h_i=[\\overrightarrow{h_i};\\overleftarrow{h_i}]$\n",
    "* Decoder's hidden-state: $s_t=f(s_{t-1}, y_{t-1}, c_t)$\n",
    "* Context vector: $c_t=\\sum_{i=1}^{n}\\alpha_{t,i}h_i$\n",
    "* $\\alpha_{t,i}=\\frac{exp(score(s_{t-1},h_i))}{\\sum_{i'=1}^{n}exp(score(s_{t-1},h_{i'}))}$\n",
    "* $score(s_t,h_i)=s_t^T W_a h_i$ [(Luong et al., 2015)](https://arxiv.org/pdf/1508.4025.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"pics/seq2seq_attention_length.png\" width=\"60%\">\n",
    "\n",
    "Performance of RNN-based seq2seq without (RNNenc) and with attention (RNNsearch) for input sentences of different lengths."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer [(Vaswani et al., 2017)](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)\n",
    "\n",
    "* A seq2seq architecture that does not use RNNs nor CNNs.\n",
    "* Instead, an attention mechanism draws global dependencies between input and output\n",
    "* Advantage: training is parallelizable. No backpropagation in time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Self-attention and path length\n",
    "\n",
    "**Self-attention**: attention mechanism that relates different positions of a single sequence $s$.\n",
    "\n",
    "**Path length** between long-range dependencies: RNN: $O(n)$, CNN: $O(log_k(n))$, self-attention: $O(1)$\n",
    "* Self-attention: constant path length between any pair of positions.\n",
    "\n",
    "<img src=\"pics/wavenet.jpg\" width=\"60%\">\n",
    "Paths in WaveNet (CNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"pics/transformer_overview.png\" width=\"30%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Masked attention: prevents attending to future words (they have not been predicted yet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Attention in Transformer\n",
    "\n",
    "<img src=\"pics/transformer_attention.png\" width=\"25%\">\n",
    "\n",
    "* Query Q. Current word\n",
    "* Keys Ks. Previous words\n",
    "* Values Vs. Keys' values\n",
    "\n",
    "$Attention(Q, K, V) = softmax(\\frac{QK^T}{\\sqrt{d_k}})V$\n",
    "\n",
    "Find the Ks that are most similar to Q. Then get Ks' values V."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Results with Transformer\n",
    "\n",
    "<img src=\"pics/transformer_results.png\" width=\"75%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## References\n",
    "\n",
    "* D. Rao & B. McMahan's NLP with PyTorch (chapter 8)\n",
    "* [Attention? Attention! Blog post by Lilian Weng](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)\n",
    "* [The annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)\n",
    "* [Video: Transformer presented by one of its authors](https://www.youtube.com/watch?v=rBCqOTEfxvg)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}