EXTRA - ATARI DEEP-Q LEARNING.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ff362182-a847-4f29-ac05-a6e24ad3d8e8",
   "metadata": {},
   "source": [
    "# ATARI Deep-Q Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f60ffd47-fa3f-4ce9-8ae9-1ec21b6d4f6d",
   "metadata": {},
   "source": [
    "## Introdução\n",
    "\n",
    "Código apresenta uma implementação da rede neural **Deep Q-Learning**,\n",
    "desenvolvida pelo grupo de pesquisadores da Deep Mind e, publicado no seguinte [artigo](https://arxiv.org/pdf/1312.5602.pdf) em 2013.\n",
    "\n",
    "\n",
    "## Deep-Q Learning\n",
    "\n",
    "À medida que um agente realiza ações e se move através de um ambiente, ele aprende a mapear o estado observado do ambiente para uma ação.\n",
    "Um agente escolherá uma ação em um determinado estado com base em um \"valor Q\", que é uma recompensa ponderada com base na maior recompensa de longo prazo esperada.\n",
    "Um Agente Q-Learning aprende a realizar sua tarefa de forma que a ação recomendada maximize as possíveis recompensas futuras.\n",
    "Este método é considerado um método \"Off-Policy\", ou seja, seus valores Q são atualizados assumindo que a melhor ação foi escolhida, mesmo que a melhor ação não tenha sido escolhida.\n",
    "\n",
    "\n",
    "## Atari Breakout\n",
    "\n",
    "No jogo em questão, o jogador poode movimentar uma barra para o lado esquerdo, ou direito da tela, procurando rebater uma bola que então destroi blocos distribuídos em camadas na parte superior da tela. O objetivo do jogo é remover todos os blocos da tela. Neste ambiente, o agente deve aprender a controlar o tabuleiro movendo-se para a esquerda e para a direita, devolvendo a bola e removendo todos os blocos sem que a bola passe no tabuleiro.\n",
    "\n",
    "## Inputs\n",
    "\n",
    "Semelhante a um jogador humano, o agente recebe uma série de imagens representando o estado em tempo real do jogo, e deve a partir destas imagens inferir se o mesmo deve se mover para esquerda ou direita da tela. Em outras palavras, o agente recebe uma série de imagens da tela e, deve determinar para qual dos dois lados possíveis ele deve se mover.\n",
    "\n",
    "\n",
    ":::{attention}\n",
    "\n",
    "Em uma máquina moderna, é necessário de cerca de 24 horas de treino, até que o agente atinga um nível razoável de experiência no jogo.\n",
    "\n",
    ":::\n",
    "\n",
    "\n",
    "## Instruções\n",
    "\n",
    "\n",
    "Nenhum tipo de instrução é passada ao agente no início do jogo. Apenas os possíveis movimentos legais que o agente pode escolher são informadas.\n",
    "Durante o treino, melhoras no score e tempo até o GameOver são pontuadas, de modo a incentivar o agente, na tomada de escolhas que façam com que o mesmo\n",
    "se aproxime do objetivo final.\n",
    "\n",
    "\n",
    "## Resultado\n",
    "\n",
    "\n",
    "O video a seguir mostra o agente no início do treino. O vdeo em questão foi capurado mais ou menos uma hora após inicio do treino.\n",
    "\n",
    "![](docs/_static/DEEPQ_TRAIN_START.gif)\n",
    "\n",
    "\n",
    "Já no próximo vídeo, o agente possui por volta de 30 horas de experiência com o jogo.\n",
    "\n",
    "\n",
    "![](docs/_static/DEEPQ_TRAIN_MID_END.gif)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd62d082-3f06-4abf-9449-1e95d99fcdaf",
   "metadata": {},
   "source": [
    "# Objetivo\n",
    "\n",
    "O objetivo deste notebook é apresentar o potêncial dos modelos de análise de imagem apresentados no presente tutorial, para análise de videos também. Embora o exemplo em questão não apresente um grando valor em termos de utilide no mundo real, ele apresenta um caso de uso extremamente utilizado por outras áreas, como para a aálise de \"self-driving\" cars.\n",
    "\n",
    "\n",
    ":::{warning}\n",
    "\n",
    "Configurar o jogo da Atari requer níveis avançado de conheciment em ciencia de dados, mas também de gerenciamento de pacotes externos.\n",
    "\n",
    ":::\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "569945d7-8444-4200-a62e-c0a8fa7e457b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from baselines.common.atari_wrappers import make_atari, wrap_deepmind\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "be5fd6eb-a775-4860-a8ba-55315d458d1b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Configuration paramaters for the whole setup\n",
    "seed = 42\n",
    "\n",
    "# Discount factor for past rewards\n",
    "gamma = 0.99\n",
    "\n",
    "# Epsilon greedy parameter\n",
    "epsilon = 1.0\n",
    "epsilon_min = 0.1  # Minimum epsilon greedy parameter.\n",
    "epsilon_max = 1.0  # Maximum epsilon greedy parameter.\n",
    "\n",
    "# Rate at which to reduce chance of random action being taken.\n",
    "epsilon_interval = epsilon_max - epsilon_min\n",
    "\n",
    "# Size of batch taken from replay buffer\n",
    "batch_size = 32\n",
    "max_steps_per_episode = 10_000\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a40dbcd4-19ba-47dd-8f06-c6c75fccf682",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/erikingwersen/Library/Caches/pypoetry/virtualenvs/pyimg-oSL_kg38-py3.8/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: \u001B[33mWARN: obs_type \"image\" should be replaced with the image type, one of: rgb, grayscale\u001B[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('BreakoutNoFrameskip-v4', render_mode='human')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b9ceef5e-7f63-4588-99d6-d278cd1d0769",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/erikingwersen/Library/Caches/pypoetry/virtualenvs/pyimg-oSL_kg38-py3.8/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: \u001B[33mWARN: Box bound precision lowered by casting to float32\u001B[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3444837047, 2669555309)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env = gym.make('BreakoutNoFrameskip-v4', render_mode='human')\n",
    "\n",
    "# Warp the frames, grey scale, stake four frame and scale to smaller ratio\n",
    "env = wrap_deepmind(env, frame_stack=True, scale=True)\n",
    "env.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "da8b092f-4dca-4977-aba2-e59a7d2e7de2",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_actions = 4\n",
    "\n",
    "def create_q_model():\n",
    "    # Network defined by the Deepmind paper\n",
    "    inputs = layers.Input(shape=(84, 84, 4,))\n",
    "\n",
    "    # Convolutions on the frames on the screen\n",
    "    layer1 = layers.Conv2D(32, 8, strides=4, activation=\"relu\")(inputs)\n",
    "    layer2 = layers.MaxPooling2D(2, 2)(layer1)\n",
    "    layer3 = layers.Conv2D(64, 4, strides=2, activation=\"relu\")(layer2)\n",
    "    # layer4 = layers.MaxPooling2D(2, 2)(layer3)\n",
    "    layer5 = layers.Conv2D(64, 3, strides=1, activation=\"relu\")(layer3)\n",
    "\n",
    "    layer6 = layers.Flatten()(layer5)\n",
    "\n",
    "    layer7 = layers.Dense(512, activation=\"relu\")(layer6)\n",
    "    action = layers.Dense(num_actions, activation=\"linear\")(layer7)\n",
    "\n",
    "    return keras.Model(inputs=inputs, outputs=action)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a111568f-7b36-45cd-bd79-59d1a50ff746",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The first model makes the predictions for Q-values which are used to\n",
    "# make a action.\n",
    "model = create_q_model()\n",
    "\n",
    "# Build a target model for the prediction of future rewards.\n",
    "# The weights of a target model get updated every 10000 steps thus when the\n",
    "# loss between the Q-values is calculated the target Q-value is stable.\n",
    "model_target = create_q_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6b927ae-55b0-4a66-89b1-c7f23dd39873",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running reward: 0.29 at episode 70, frame count 10000\n",
      "running reward: 0.17 at episode 149, frame count 20000\n",
      "running reward: 0.17 at episode 229, frame count 30000\n",
      "running reward: 0.24 at episode 297, frame count 40000\n",
      "running reward: 0.23 at episode 367, frame count 50000\n",
      "running reward: 0.18 at episode 448, frame count 60000\n",
      "running reward: 0.19 at episode 527, frame count 70000\n",
      "running reward: 0.16 at episode 607, frame count 80000\n",
      "running reward: 0.28 at episode 678, frame count 90000\n",
      "running reward: 0.14 at episode 764, frame count 100000\n",
      "running reward: 0.13 at episode 846, frame count 110000\n",
      "running reward: 0.27 at episode 918, frame count 120000\n",
      "running reward: 0.17 at episode 1000, frame count 130000\n",
      "running reward: 0.15 at episode 1081, frame count 140000\n",
      "running reward: 0.17 at episode 1159, frame count 150000\n",
      "running reward: 0.25 at episode 1230, frame count 160000\n",
      "running reward: 0.20 at episode 1304, frame count 170000\n",
      "running reward: 0.24 at episode 1376, frame count 180000\n",
      "running reward: 0.32 at episode 1445, frame count 190000\n",
      "running reward: 0.17 at episode 1527, frame count 200000\n",
      "running reward: 0.20 at episode 1602, frame count 210000\n",
      "running reward: 0.20 at episode 1681, frame count 220000\n",
      "running reward: 0.18 at episode 1760, frame count 230000\n",
      "running reward: 0.17 at episode 1833, frame count 240000\n",
      "running reward: 0.22 at episode 1904, frame count 250000\n",
      "running reward: 0.19 at episode 1982, frame count 260000\n",
      "running reward: 0.22 at episode 2061, frame count 270000\n",
      "running reward: 0.21 at episode 2134, frame count 280000\n",
      "running reward: 0.19 at episode 2215, frame count 290000\n",
      "running reward: 0.22 at episode 2291, frame count 300000\n",
      "running reward: 0.22 at episode 2369, frame count 310000\n",
      "running reward: 0.24 at episode 2442, frame count 320000\n",
      "running reward: 0.22 at episode 2518, frame count 330000\n",
      "running reward: 0.18 at episode 2605, frame count 340000\n",
      "running reward: 0.21 at episode 2682, frame count 350000\n",
      "running reward: 0.20 at episode 2757, frame count 360000\n",
      "running reward: 0.22 at episode 2833, frame count 370000\n",
      "running reward: 0.22 at episode 2906, frame count 380000\n",
      "running reward: 0.29 at episode 2980, frame count 390000\n",
      "running reward: 0.28 at episode 3054, frame count 400000\n",
      "running reward: 0.28 at episode 3121, frame count 410000\n",
      "running reward: 0.16 at episode 3206, frame count 420000\n",
      "running reward: 0.28 at episode 3272, frame count 430000\n",
      "running reward: 0.24 at episode 3353, frame count 440000\n"
     ]
    }
   ],
   "source": [
    "# In the Deepmind paper they use RMSProp however then Adam optimizer\n",
    "# improves training time\n",
    "optimizer = keras.optimizers.Adam(learning_rate=0.00025, clipnorm=1.0)\n",
    "\n",
    "# Experience replay buffers\n",
    "action_history = []\n",
    "state_history = []\n",
    "state_next_history = []\n",
    "rewards_history = []\n",
    "done_history = []\n",
    "episode_reward_history = []\n",
    "running_reward = 0\n",
    "episode_count = 0\n",
    "frame_count = 0\n",
    "\n",
    "# Number of frames to take random action and observe output\n",
    "epsilon_random_frames = 50000\n",
    "\n",
    "# Number of frames for exploration\n",
    "epsilon_greedy_frames = 1000000.0\n",
    "\n",
    "# Maximum replay length\n",
    "# Note: The Deepmind paper suggests 1000000 however this causes memory issues\n",
    "max_memory_length = 100000\n",
    "\n",
    "# Train the model after 4 actions\n",
    "update_after_actions = 4\n",
    "\n",
    "# How often to update the target network\n",
    "update_target_network = 10_000\n",
    "\n",
    "# Using huber loss for stability\n",
    "loss_function = keras.losses.Huber()\n",
    "\n",
    "\n",
    "while True:  # Run until solved\n",
    "\n",
    "    state = np.array(env.reset())\n",
    "    episode_reward = 0\n",
    "\n",
    "    for timestep in range(1, max_steps_per_episode):\n",
    "        # gym.make(timestep, render_mode='human')\n",
    "\n",
    "        # of the agent in a pop up window.\n",
    "        frame_count += 1\n",
    "\n",
    "        # Use epsilon-greedy for exploration\n",
    "        if frame_count < epsilon_random_frames or epsilon > np.random.rand(1)[0]:\n",
    "            # Take random action\n",
    "            action = np.random.choice(num_actions)\n",
    "        else:\n",
    "            # Predict action Q-values\n",
    "            # From environment state\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_probs = model(state_tensor, training=False)\n",
    "\n",
    "            # Take best action\n",
    "            action = tf.argmax(action_probs[0]).numpy()\n",
    "\n",
    "        # Decay probability of taking random action\n",
    "        epsilon -= epsilon_interval / epsilon_greedy_frames\n",
    "        epsilon = max(epsilon, epsilon_min)\n",
    "\n",
    "        # Apply the sampled action in our environment\n",
    "        state_next, reward, done, _ = env.step(action)\n",
    "        state_next = np.array(state_next)\n",
    "\n",
    "        episode_reward += reward\n",
    "\n",
    "        # Save actions and states in replay buffer\n",
    "        action_history.append(action)\n",
    "        state_history.append(state)\n",
    "        state_next_history.append(state_next)\n",
    "        done_history.append(done)\n",
    "        rewards_history.append(reward)\n",
    "        state = state_next\n",
    "\n",
    "        # Update every fourth frame and once batch size is over 32\n",
    "        if frame_count % update_after_actions == 0 and len(done_history) > batch_size:\n",
    "\n",
    "            # Get indices of samples for replay buffers\n",
    "            indices = np.random.choice(range(len(done_history)), size=batch_size)\n",
    "\n",
    "            # Using list comprehension to sample from replay buffer\n",
    "            state_sample = np.array([state_history[i] for i in indices])\n",
    "            state_next_sample = np.array([state_next_history[i] for i in indices])\n",
    "            rewards_sample = [rewards_history[i] for i in indices]\n",
    "            action_sample = [action_history[i] for i in indices]\n",
    "            done_sample = tf.convert_to_tensor(\n",
    "                [float(done_history[i]) for i in indices]\n",
    "            )\n",
    "\n",
    "            # Build the updated Q-values for the sampled future states\n",
    "            # Use the target model for stability\n",
    "            future_rewards = model_target.predict(state_next_sample)\n",
    "\n",
    "            # Q value = reward + discount factor * expected future reward\n",
    "            updated_q_values = rewards_sample + gamma * tf.reduce_max(\n",
    "                future_rewards, axis=1\n",
    "            )\n",
    "\n",
    "            # If final frame set the last value to -1\n",
    "            updated_q_values = updated_q_values * (1 - done_sample) - done_sample\n",
    "\n",
    "            # Create a mask so we only calculate loss on the updated Q-values\n",
    "            masks = tf.one_hot(action_sample, num_actions)\n",
    "\n",
    "            with tf.GradientTape() as tape:\n",
    "                # Train the model on the states and updated Q-values\n",
    "                q_values = model(state_sample)\n",
    "\n",
    "                # Apply the masks to the Q-values to get the Q-value for action taken\n",
    "                q_action = tf.reduce_sum(tf.multiply(q_values, masks), axis=1)\n",
    "\n",
    "                # Calculate loss between new Q-value and old Q-value\n",
    "                loss = loss_function(updated_q_values, q_action)\n",
    "\n",
    "            # Backpropagation\n",
    "            grads = tape.gradient(loss, model.trainable_variables)\n",
    "            optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "        if frame_count % update_target_network == 0:\n",
    "\n",
    "            # update the the target network with new weights\n",
    "            model_target.set_weights(model.get_weights())\n",
    "\n",
    "            # Log details\n",
    "            template = \"running reward: {:.2f} at episode {}, frame count {}\"\n",
    "            print(template.format(running_reward, episode_count, frame_count))\n",
    "\n",
    "        # Limit the state and reward history\n",
    "        if len(rewards_history) > max_memory_length:\n",
    "            del rewards_history[:1]\n",
    "            del state_history[:1]\n",
    "            del state_next_history[:1]\n",
    "            del action_history[:1]\n",
    "            del done_history[:1]\n",
    "\n",
    "        if done:\n",
    "            break\n",
    "\n",
    "    # Update running reward to check condition for solving\n",
    "    episode_reward_history.append(episode_reward)\n",
    "    if len(episode_reward_history) > 100:\n",
    "        del episode_reward_history[:1]\n",
    "\n",
    "    running_reward = np.mean(episode_reward_history)\n",
    "\n",
    "    episode_count += 1\n",
    "\n",
    "    if running_reward > 40:  # Condition to consider the task solved\n",
    "        print(\"Solved at episode {}!\".format(episode_count))\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f603fe8-db56-4c95-a8fc-72d6388c6e43",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}