model_code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Machine Learning Workshop\n",
    "### Assignment 5: CarNet\n",
    "### Description of the Solution\n",
    "### Priyanka Rotte\n",
    " \n",
    "#### Individual Models:\n",
    "For individual models, I’m using transfer learning from the EfficientNetV2B1 model. I tried various models available in the Keras library based on Size, Accuracy, Parameters, Time, etc. and EfficientNetV2B1 performed the best for the given data. I’m removing the fully-connected layer at the top of the model and using global average pooling on the output of the layer before. Average pooling worked better than flattening, as the output shape was getting too high after flattening. <br> \n",
    "After EfficientNetV2B1, I’m using two separate models for vehicles and signals, as to enable the models to learn features specific to the type. After several experiments, I’m going with two dense layers of sizes 64 and 32, with ReLU activations. Finally, there’s an output dense layer with 1 neuron. I’m not using any dropouts or regularization as the ensemble method with bootstrap replicates should prevent overfitting. I’m using Adam optimizer with a 0.001 learning rate, MSE loss, and early stopping.\n",
    "#### Ensemble Model (Blender):\n",
    "I’m training 7 individual models with the same architecture with different kernel initializations and bootstrap replicates i.e., random resampling of the dataset with replacement. <br>\n",
    "After training the individual models, I’m training an ensemble model with the outputs of these 7 models as inputs to the ensemble. Instead of just getting an average, I’m using a blender i.e., a neural network on top of the individual models. Here too, I’m using two separate models for vehicles and signals, just with one dense layer of 128 and ReLU, and one dropout (even dropout will act as an ensemble of models and prevent overfitting). This architecture gave the lowest validation MSE. Finally, there’s an output dense layer with 1 neuron. The training is performed similarly as for the individual models. I’m saving the final model with the least validation loss. Finally, I’m getting a validation MSE of 4.29 and a test MSE of 4.36.\n",
    "#### Submission:\n",
    "- Jupyter notebook with code for the model\n",
    "- Parquet file with predictions on score\n",
    "- Zip file\n",
    "    1. individual_models has individual folders for each model with their model definition, parameters and predictions on train, validate, test, and score.\n",
    "    2. ensemble_model has ensemble model definition, parameters, and predictions on score set. Folder inside, ensemble_alternate has the other way of creating the same ensemble.\n",
    "- PDF with solution description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import json\n",
    "\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import math\n",
    "import seaborn as sns\n",
    "sns.set_context('talk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-31 04:05:09.954006: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:10.009021: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:10.010090: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n"
     ]
    }
   ],
   "source": [
    "# This configures the GPU to be used by Tensorflow.\n",
    "import tensorflow as tf\n",
    "gpus = tf.config.experimental.list_physical_devices('GPU')\n",
    "tf.config.experimental.set_memory_growth(gpus[0], True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Training Images and Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>car</th>\n",
       "      <th>bus</th>\n",
       "      <th>truck</th>\n",
       "      <th>train</th>\n",
       "      <th>motorcycle</th>\n",
       "      <th>bicycle</th>\n",
       "      <th>airplane</th>\n",
       "      <th>boat</th>\n",
       "      <th>traffic light</th>\n",
       "      <th>stop sign</th>\n",
       "      <th>vehicle</th>\n",
       "      <th>signal</th>\n",
       "      <th>file_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>000000000064.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>000000000073.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>000000000074.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>000000000081.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>000000000086.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   car  bus  truck  train  motorcycle  bicycle  airplane  boat  traffic light  \\\n",
       "0  1.0  0.0    1.0    0.0         0.0      0.0       0.0   0.0            0.0   \n",
       "1  0.0  0.0    0.0    0.0         2.0      0.0       0.0   0.0            0.0   \n",
       "2  0.0  0.0    0.0    0.0         0.0      1.0       0.0   0.0            0.0   \n",
       "3  0.0  0.0    0.0    0.0         0.0      0.0       1.0   0.0            0.0   \n",
       "4  0.0  0.0    0.0    0.0         1.0      0.0       0.0   0.0            0.0   \n",
       "\n",
       "   stop sign  vehicle  signal         file_name  \n",
       "0        1.0      2.0     1.0  000000000064.jpg  \n",
       "1        0.0      2.0     0.0  000000000073.jpg  \n",
       "2        0.0      1.0     0.0  000000000074.jpg  \n",
       "3        0.0      1.0     0.0  000000000081.jpg  \n",
       "4        0.0      1.0     0.0  000000000086.jpg  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_metadata = pd.read_csv('carnet_dataset/train/metadata.csv')\n",
    "train_metadata.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score Images and Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000000000071.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000000000149.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000000000260.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000000000307.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>000000000690.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          file_name\n",
       "0  000000000071.jpg\n",
       "1  000000000149.jpg\n",
       "2  000000000260.jpg\n",
       "3  000000000307.jpg\n",
       "4  000000000690.jpg"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score_metadata = pd.read_csv('carnet_dataset/score/metadata.csv')\n",
    "score_metadata.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Tensorflow Dataset (Single Image, Labeled Segment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Splitting the train data into train, validate, and test \n",
    "from sklearn.model_selection import train_test_split\n",
    "train_metadata, test_metadata = train_test_split(train_metadata, test_size=0.20, random_state=42)\n",
    "train_metadata, val_metadata = train_test_split(train_metadata, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See Python Generator\n",
    "# https://peps.python.org/pep-0255/\n",
    "def build_generator_labeled_single(metadata: pd.DataFrame):\n",
    "    def generator():\n",
    "        for _, row in metadata.iterrows():\n",
    "            \n",
    "            training_path = 'carnet_dataset/train/images/' + row['file_name']\n",
    "            training_np = np.array(Image.open(training_path)).astype(np.float32)\n",
    "            \n",
    "            ### Passing image as input\n",
    "            model_input = training_np\n",
    "            \n",
    "            ### Passing count of vehicles and count of signals as outputs\n",
    "            model_output_1 = (row[\"vehicle\"], )\n",
    "            model_output_2 = (row[\"signal\"], )\n",
    "            yield (model_input, (model_output_1, model_output_2))\n",
    "            \n",
    "    return generator\n",
    "\n",
    "\n",
    "# See Tensorflow Dataset\n",
    "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n",
    "def build_dataset_labeled_single(metadata: pd.DataFrame) -> tf.data.Dataset:\n",
    "    model_input = tf.TensorSpec(shape=(224, 224, 3), dtype=tf.float32)  # type: ignore\n",
    "    model_output = tf.TensorSpec(shape=(1, ), dtype=tf.float32)  # type: ignore\n",
    "\n",
    "    dataset_signature = (model_input, (model_output, model_output))\n",
    "\n",
    "    dataset = tf.data.Dataset.from_generator(\n",
    "        build_generator_labeled_single(metadata), \n",
    "        output_signature=dataset_signature\n",
    "    )\n",
    "\n",
    "    return dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-31 04:05:11.934278: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-10-31 04:05:11.935000: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:11.935925: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:11.936752: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:12.640707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:12.641566: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:12.642375: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-10-31 04:05:12.643113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 10780 MB memory:  -> device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7\n"
     ]
    }
   ],
   "source": [
    "val_single_dataset = build_dataset_labeled_single(val_metadata).batch(batch_size)\n",
    "# model_input, (model_output_1, model_output_2) = next(iter(val_single_dataset))\n",
    "# display(model_input.shape, model_output_1.shape, model_output_2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_single_dataset = build_dataset_labeled_single(test_metadata).batch(batch_size)\n",
    "# model_input, (model_output_1, model_output_2) = next(iter(test_single_dataset))\n",
    "# display(model_input.shape, model_output_1.shape, model_output_2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Tensorflow Dataset (Single Image, Score Segment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# See Python Generator\n",
    "# https://peps.python.org/pep-0255/\n",
    "\n",
    "def generator_score_single():\n",
    "    metadata = pd.read_csv('carnet_dataset/score/metadata.csv')\n",
    "    \n",
    "    for _, row in metadata.iterrows():\n",
    "        image_path = 'carnet_dataset/score/images/' + row['file_name']\n",
    "\n",
    "        image_np = np.array(Image.open(image_path))\n",
    "\n",
    "        model_input = image_np\n",
    "        yield model_input\n",
    "\n",
    "\n",
    "# See Tensorflow Dataset\n",
    "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n",
    "def build_dataset_score_single() -> tf.data.Dataset:\n",
    "    model_input = tf.TensorSpec(shape=(224, 224, 3), dtype=tf.float32)  # type: ignore\n",
    "\n",
    "    dataset_signature = model_input\n",
    "\n",
    "    dataset = tf.data.Dataset.from_generator(\n",
    "        generator_score_single, \n",
    "        output_signature=dataset_signature\n",
    "    )\n",
    "\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_single_dataset = build_dataset_score_single().batch(batch_size)\n",
    "# model_input = next(iter(score_single_dataset))\n",
    "# display(model_input.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.initializers import GlorotUniform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Function to build individual models of the Ensemble\n",
    "def build_model(kernel_seed):\n",
    "    \n",
    "    ### Getting pre-trained EfficientNetV2B1 Architecture, \n",
    "    ### using global average pooling at the end and excluding the top dense layer\n",
    "    efficient_net = tf.keras.applications.EfficientNetV2B1(\n",
    "        include_top=False,\n",
    "        input_shape=(224, 224, 3),\n",
    "        pooling=\"avg\",\n",
    "        weights=\"imagenet\"\n",
    "    )\n",
    "\n",
    "    ### Freezing the parameters of the MobileNet layers\n",
    "    efficient_net.trainable = False\n",
    "    \n",
    "    ### Input: image\n",
    "    input = tf.keras.layers.Input((224, 224, 3))\n",
    "\n",
    "    ### Using MobileNet for the image\n",
    "    outputs = efficient_net(input)\n",
    "\n",
    "    ### Creating two seperate models for Vehicles and Signals using Dense Layers\n",
    "    outputs_1 = tf.keras.layers.Dense(64, activation='relu',\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+100))(outputs)\n",
    "    outputs_1 = tf.keras.layers.Dense(32, activation='relu',\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+200))(outputs_1)\n",
    "    outputs_1 = tf.keras.layers.Dense(1,\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+300))(outputs_1)\n",
    "\n",
    "    \n",
    "    outputs_2 = tf.keras.layers.Dense(64, activation='relu',\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+400))(outputs)\n",
    "    outputs_2 = tf.keras.layers.Dense(32, activation='relu',\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+500))(outputs_2)\n",
    "    outputs_2 = tf.keras.layers.Dense(1,\n",
    "                                      kernel_initializer=GlorotUniform(seed=kernel_seed+600))(outputs_2)\n",
    "\n",
    "    model = tf.keras.models.Model(inputs=input, outputs=[outputs_1, outputs_2])\n",
    "\n",
    "    ### Adam optimizer with 0.001 learning rate\n",
    "    model.compile(\n",
    "        optimizer=tf.keras.optimizers.Adam(\n",
    "            lr=0.001\n",
    "        ),\n",
    "        ### MSE loss function\n",
    "        loss='mean_squared_error'\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Function to train individual models of the Ensemble\n",
    "def train_model(model, train_single_dataset, val_single_dataset, seed):\n",
    "    np.random.seed(seed)\n",
    "    tf.random.set_seed(seed)\n",
    "    \n",
    "    ### Checkpoint path \n",
    "    checkpoint_path = \"weights_27/car_{0}\".format(seed)\n",
    "    checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "    ### Callback to save model log / results after each epoch\n",
    "    csv_logger = tf.keras.callbacks.CSVLogger(\"weights_27/model{0}_history_log.csv\".format(seed), append=True)\n",
    "\n",
    "    ### Callback to save weights after every epoch i.e., checkpoint\n",
    "    cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
    "                                                     save_weights_only=True,\n",
    "                                                     verbose=1, save_best_only=True, monitor = 'val_loss') \n",
    "\n",
    "    ### Callback to stop the training if there's no improvement in validation loss for 3 epochs\n",
    "    stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n",
    "    \n",
    "    ### Fitting the model\n",
    "    model.fit(\n",
    "        train_single_dataset,\n",
    "        epochs=128,\n",
    "        shuffle=True,\n",
    "        validation_data=(val_single_dataset),\n",
    "        callbacks=[stop_early, cp_callback, csv_logger]\n",
    "    )\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Function to save individual models of the Ensemble\n",
    "def save_model(model, seed):\n",
    "    model_dir = 'weights_27/model{0}'.format(seed)\n",
    "    os.makedirs(model_dir, exist_ok=True)\n",
    "    with open(f'{model_dir}/keras_model.json', 'w') as f:\n",
    "        model_json = json.dumps(json.loads(model.to_json()), indent=True)\n",
    "        f.write(model_json)\n",
    "\n",
    "    model.save_weights(f\"{model_dir}/keras_parameters.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Function to predict and save individual models' outputs \n",
    "    ### for train, validate, test, and score in parquet files\n",
    "def make_predictions(model, train, val, test, score, seed):\n",
    "    model_dir = 'weights_27/model{0}'.format(seed)\n",
    "    \n",
    "    y_hat = model.predict(train)\n",
    "    y_hat = pd.DataFrame(np.array(y_hat).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "    y_hat.to_parquet(f'{model_dir}/train_y_hat.parquet')\n",
    "    \n",
    "    y_hat = model.predict(val)\n",
    "    y_hat = pd.DataFrame(np.array(y_hat).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "    y_hat.to_parquet(f'{model_dir}/val_y_hat.parquet')\n",
    "    \n",
    "    y_hat = model.predict(test)\n",
    "    y_hat = pd.DataFrame(np.array(y_hat).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "    y_hat.to_parquet(f'{model_dir}/test_y_hat.parquet')\n",
    "    \n",
    "    y_hat = model.predict(score)\n",
    "    y_hat = pd.DataFrame(np.array(y_hat).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "    y_hat.to_parquet(f'{model_dir}/score_y_hat.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Creating 7 models\n",
    "model_dir = 'weights_27'\n",
    "os.makedirs(model_dir, exist_ok=True)\n",
    "np.random.seed(123)\n",
    "n_models = 7\n",
    "seeds = np.random.randint(0, 1e2, size=n_models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "66\n",
      "Epoch 1/128\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Adam, self).__init__(name, **kwargs)\n",
      "2022-10-31 00:03:45.922000: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8401\n",
      "2022-10-31 00:03:46.454267: W tensorflow/stream_executor/gpu/asm_compiler.cc:111] *** WARNING *** You are using ptxas 10.0.145, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.\n",
      "\n",
      "You may not need to update to CUDA 11.1; cherry-picking the ptxas binary is often sufficient.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    109/Unknown - 86s 687ms/step - loss: 11.9509 - dense_3_loss: 10.4316 - dense_6_loss: 1.5193\n",
      "Epoch 1: val_loss improved from inf to 9.66044, saving model to weights_27/car_66\n",
      "109/109 [==============================] - 114s 947ms/step - loss: 11.9509 - dense_3_loss: 10.4316 - dense_6_loss: 1.5193 - val_loss: 9.6604 - val_dense_3_loss: 8.3635 - val_dense_6_loss: 1.2969\n",
      "Epoch 2/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 8.6230 - dense_3_loss: 7.5001 - dense_6_loss: 1.1229\n",
      "Epoch 2: val_loss improved from 9.66044 to 9.35475, saving model to weights_27/car_66\n",
      "109/109 [==============================] - 98s 905ms/step - loss: 8.6230 - dense_3_loss: 7.5001 - dense_6_loss: 1.1229 - val_loss: 9.3547 - val_dense_3_loss: 8.0747 - val_dense_6_loss: 1.2800\n",
      "Epoch 3/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 6.9881 - dense_3_loss: 6.1378 - dense_6_loss: 0.8503\n",
      "Epoch 3: val_loss did not improve from 9.35475\n",
      "109/109 [==============================] - 97s 894ms/step - loss: 6.9881 - dense_3_loss: 6.1378 - dense_6_loss: 0.8503 - val_loss: 9.4361 - val_dense_3_loss: 8.1340 - val_dense_6_loss: 1.3020\n",
      "Epoch 4/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 5.4959 - dense_3_loss: 4.8762 - dense_6_loss: 0.6197\n",
      "Epoch 4: val_loss did not improve from 9.35475\n",
      "109/109 [==============================] - 98s 899ms/step - loss: 5.4959 - dense_3_loss: 4.8762 - dense_6_loss: 0.6197 - val_loss: 9.5485 - val_dense_3_loss: 8.2131 - val_dense_6_loss: 1.3354\n",
      "Epoch 5/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 4.1721 - dense_3_loss: 3.7311 - dense_6_loss: 0.4410\n",
      "Epoch 5: val_loss did not improve from 9.35475\n",
      "109/109 [==============================] - 98s 898ms/step - loss: 4.1721 - dense_3_loss: 3.7311 - dense_6_loss: 0.4410 - val_loss: 9.7801 - val_dense_3_loss: 8.4030 - val_dense_6_loss: 1.3772\n",
      "92\n",
      "Epoch 1/128\n",
      "    109/Unknown - 80s 671ms/step - loss: 12.5610 - dense_9_loss: 11.0551 - dense_12_loss: 1.5058\n",
      "Epoch 1: val_loss improved from inf to 9.69592, saving model to weights_27/car_92\n",
      "109/109 [==============================] - 107s 919ms/step - loss: 12.5610 - dense_9_loss: 11.0551 - dense_12_loss: 1.5058 - val_loss: 9.6959 - val_dense_9_loss: 8.4524 - val_dense_12_loss: 1.2435\n",
      "Epoch 2/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 8.6111 - dense_27_loss: 7.5119 - dense_30_loss: 1.0991\n",
      "Epoch 2: val_loss improved from 9.55394 to 8.98463, saving model to weights_27/car_83\n",
      "109/109 [==============================] - 98s 900ms/step - loss: 8.6111 - dense_27_loss: 7.5119 - dense_30_loss: 1.0991 - val_loss: 8.9846 - val_dense_27_loss: 7.7588 - val_dense_30_loss: 1.2258\n",
      "Epoch 3/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 7.0633 - dense_33_loss: 6.3010 - dense_36_loss: 0.7623\n",
      "Epoch 3: val_loss did not improve from 9.10349\n",
      "109/109 [==============================] - 99s 908ms/step - loss: 7.0633 - dense_33_loss: 6.3010 - dense_36_loss: 0.7623 - val_loss: 9.1936 - val_dense_33_loss: 7.9655 - val_dense_36_loss: 1.2281\n",
      "Epoch 4/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 5.6332 - dense_33_loss: 5.0695 - dense_36_loss: 0.5637\n",
      "Epoch 4: val_loss did not improve from 9.10349\n",
      "109/109 [==============================] - 98s 903ms/step - loss: 5.6332 - dense_33_loss: 5.0695 - dense_36_loss: 0.5637 - val_loss: 9.4153 - val_dense_33_loss: 8.1473 - val_dense_36_loss: 1.2680\n",
      "Epoch 5/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 4.3591 - dense_33_loss: 3.9495 - dense_36_loss: 0.4096\n",
      "Epoch 5: val_loss did not improve from 9.10349\n",
      "109/109 [==============================] - 98s 899ms/step - loss: 4.3591 - dense_33_loss: 3.9495 - dense_36_loss: 0.4096 - val_loss: 9.6522 - val_dense_33_loss: 8.3258 - val_dense_36_loss: 1.3264\n",
      "86\n",
      "Epoch 1/128\n",
      "    109/Unknown - 81s 673ms/step - loss: 12.2274 - dense_39_loss: 10.8092 - dense_42_loss: 1.4181\n",
      "Epoch 1: val_loss improved from inf to 9.76862, saving model to weights_27/car_86\n",
      "109/109 [==============================] - 107s 921ms/step - loss: 12.2274 - dense_39_loss: 10.8092 - dense_42_loss: 1.4181 - val_loss: 9.7686 - val_dense_39_loss: 8.5101 - val_dense_42_loss: 1.2586\n",
      "Epoch 2/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 8.8134 - dense_39_loss: 7.7725 - dense_42_loss: 1.0409\n",
      "Epoch 2: val_loss improved from 9.76862 to 9.10522, saving model to weights_27/car_86\n",
      "109/109 [==============================] - 98s 902ms/step - loss: 8.8134 - dense_39_loss: 7.7725 - dense_42_loss: 1.0409 - val_loss: 9.1052 - val_dense_39_loss: 7.8539 - val_dense_42_loss: 1.2513\n",
      "Epoch 3/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 7.2522 - dense_39_loss: 6.4579 - dense_42_loss: 0.7943\n",
      "Epoch 3: val_loss improved from 9.10522 to 9.07340, saving model to weights_27/car_86\n",
      "109/109 [==============================] - 99s 908ms/step - loss: 7.2522 - dense_39_loss: 6.4579 - dense_42_loss: 0.7943 - val_loss: 9.0734 - val_dense_39_loss: 7.8124 - val_dense_42_loss: 1.2610\n",
      "Epoch 4/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 5.8644 - dense_39_loss: 5.2822 - dense_42_loss: 0.5822\n",
      "Epoch 4: val_loss did not improve from 9.07340\n",
      "109/109 [==============================] - 98s 903ms/step - loss: 5.8644 - dense_39_loss: 5.2822 - dense_42_loss: 0.5822 - val_loss: 9.4313 - val_dense_39_loss: 8.1201 - val_dense_42_loss: 1.3112\n",
      "Epoch 5/128\n",
      " 60/109 [===============>..............] - ETA: 33s - loss: 4.8881 - dense_39_loss: 4.4626 - dense_42_loss: 0.4255"
     ]
    }
   ],
   "source": [
    "### Train individual models\n",
    "for seed in seeds:\n",
    "    print(seed)\n",
    "    ### Random resampling of the dataset with replacement (bootstrap replicate)\n",
    "    train_data = train_metadata.sample(frac = 1, replace = True, random_state = seed)\n",
    "    train_single_dataset = build_dataset_labeled_single(train_data).batch(batch_size)\n",
    "    model = build_model(seed)\n",
    "    model = train_model(model, train_single_dataset, val_single_dataset, seed)\n",
    "    save_model(model, seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Making predictions on the whole train set and validation, test, and score sets\n",
    "train_single_dataset = build_dataset_labeled_single(train_metadata).batch(batch_size)\n",
    "for seed in seeds:\n",
    "    model_dir = 'weights_27/model{0}'.format(seed)\n",
    "    model = json.load(open(f'{model_dir}/keras_model.json'))\n",
    "    model = tf.keras.models.model_from_config(model)\n",
    "    model.load_weights(f\"{model_dir}/keras_parameters.h5\")\n",
    "    make_predictions(model, train_single_dataset, val_single_dataset, test_single_dataset, \n",
    "                     score_single_dataset, seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Getting predictions of all models and adding them to the list as inputs to the ensemble model\n",
    "train_pred = []\n",
    "val_pred = []\n",
    "test_pred = []\n",
    "score_pred = []\n",
    "for seed in seeds:\n",
    "    model_dir = 'weights_27/model{0}/'.format(seed)\n",
    "    train_pred.append(pd.read_parquet(f'{model_dir}/train_y_hat.parquet', engine='pyarrow').values)\n",
    "    val_pred.append(pd.read_parquet(f'{model_dir}/val_y_hat.parquet', engine='pyarrow').values)\n",
    "    test_pred.append(pd.read_parquet(f'{model_dir}/test_y_hat.parquet', engine='pyarrow').values)\n",
    "    score_pred.append(pd.read_parquet(f'{model_dir}/score_y_hat.parquet', engine='pyarrow').values)\n",
    "\n",
    "### Train\n",
    "ip1 = np.array(train_pred).transpose()[0,:,:]\n",
    "ip2 = np.array(train_pred).transpose()[1,:,:]\n",
    "op1 = train_metadata[\"vehicle\"].values\n",
    "op2 = train_metadata[\"signal\"].values\n",
    "\n",
    "### Validate\n",
    "ip1v = np.array(val_pred).transpose()[0,:,:]\n",
    "ip2v = np.array(val_pred).transpose()[1,:,:]\n",
    "op1v = val_metadata[\"vehicle\"].values\n",
    "op2v = val_metadata[\"signal\"].values\n",
    "\n",
    "### Test\n",
    "ip1t = np.array(test_pred).transpose()[0,:,:]\n",
    "ip2t = np.array(test_pred).transpose()[1,:,:]\n",
    "op1t = test_metadata[\"vehicle\"].values\n",
    "op2t = test_metadata[\"signal\"].values\n",
    "\n",
    "### Score\n",
    "ip1s = np.array(score_pred).transpose()[0,:,:]\n",
    "ip2s = np.array(score_pred).transpose()[1,:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Creating the Ensemble model with outputs of 7 inidividual models as inputs\n",
    "model_input_1 = tf.keras.layers.Input((7))  ### Vehicles\n",
    "model_input_2 = tf.keras.layers.Input((7))  ### Signals\n",
    "\n",
    "### Creating individual models for vehicles and signals with dense layers and dropouts to reduce overfitting\n",
    "ensemble_output_1 = tf.keras.layers.Dense(128, activation = \"relu\")(model_input_1)\n",
    "ensemble_output_1 = tf.keras.layers.Dropout(0.25)(ensemble_output_1)\n",
    "ensemble_output_1 = tf.keras.layers.Dense(1)(ensemble_output_1)\n",
    "\n",
    "ensemble_output_2 = tf.keras.layers.Dense(128, activation = \"relu\")(model_input_2)\n",
    "ensemble_output_2 = tf.keras.layers.Dropout(0.25)(ensemble_output_2)\n",
    "ensemble_output_2 = tf.keras.layers.Dense(1)(ensemble_output_2)\n",
    "\n",
    "ensemble_model = tf.keras.Model(inputs=[model_input_1, model_input_2], \n",
    "                                outputs=[ensemble_output_1, ensemble_output_2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Checkpoint path \n",
    "checkpoint_path = \"weights/car_27\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Adam optimizer with 0.001 learning rate\n",
    "ensemble_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(\n",
    "        lr=0.001\n",
    "    ),\n",
    "    ### MSE loss\n",
    "    loss='mean_squared_error'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/128\n",
      "427/436 [============================>.] - ETA: 0s - loss: 5.5672 - dense_45_loss: 4.9973 - dense_47_loss: 0.5699\n",
      "Epoch 1: val_loss improved from inf to 8.60365, saving model to weights/car_271\n",
      "436/436 [==============================] - 3s 5ms/step - loss: 5.5533 - dense_45_loss: 4.9848 - dense_47_loss: 0.5685 - val_loss: 8.6037 - val_dense_45_loss: 7.4257 - val_dense_47_loss: 1.1779\n",
      "Epoch 2/128\n",
      "426/436 [============================>.] - ETA: 0s - loss: 4.9869 - dense_45_loss: 4.4855 - dense_47_loss: 0.5013\n",
      "Epoch 2: val_loss did not improve from 8.60365\n",
      "436/436 [==============================] - 2s 4ms/step - loss: 4.9794 - dense_45_loss: 4.4773 - dense_47_loss: 0.5021 - val_loss: 8.7114 - val_dense_45_loss: 7.5336 - val_dense_47_loss: 1.1779\n",
      "Epoch 3/128\n",
      "432/436 [============================>.] - ETA: 0s - loss: 4.7844 - dense_45_loss: 4.3029 - dense_47_loss: 0.4815\n",
      "Epoch 3: val_loss did not improve from 8.60365\n",
      "436/436 [==============================] - 2s 4ms/step - loss: 4.7885 - dense_45_loss: 4.3062 - dense_47_loss: 0.4823 - val_loss: 8.7046 - val_dense_45_loss: 7.4732 - val_dense_47_loss: 1.2314\n",
      "Epoch 4/128\n",
      "423/436 [============================>.] - ETA: 0s - loss: 4.8174 - dense_45_loss: 4.3454 - dense_47_loss: 0.4720\n",
      "Epoch 4: val_loss did not improve from 8.60365\n",
      "436/436 [==============================] - 2s 4ms/step - loss: 4.8397 - dense_45_loss: 4.3672 - dense_47_loss: 0.4724 - val_loss: 8.7165 - val_dense_45_loss: 7.4964 - val_dense_47_loss: 1.2201\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fa8d1b10160>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Training the Ensemble model\n",
    "np.random.seed(11)\n",
    "tf.random.set_seed(11)\n",
    "\n",
    "### Callback to save model log / results after each epoch\n",
    "csv_logger = tf.keras.callbacks.CSVLogger(\"model27_history_log.csv\", append=True)\n",
    "\n",
    "### Callback to save weights after every epoch i.e., checkpoint\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1, save_best_only=True, monitor = 'val_loss') \n",
    "\n",
    "### Callback to stop the training if there's no improvement in validation loss for 3 epochs\n",
    "stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n",
    "ensemble_model.fit(\n",
    "    x = [ip1, ip2],\n",
    "    y = [op1, op2],\n",
    "    epochs=128,\n",
    "    shuffle=True,\n",
    "    validation_data=([ip1v, ip2v], [op1v, op2v]),\n",
    "    callbacks=[stop_early, cp_callback, csv_logger]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7fa8d1a6e7d0>"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Loading the weights from the best model\n",
    "ensemble_model.load_weights(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "146/146 [==============================] - 1s 2ms/step - loss: 8.5805 - dense_49_loss: 7.4031 - dense_51_loss: 1.1774\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4.29025936126709"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Evaluating model on Validation\n",
    "total_loss, _, _ = ensemble_model.evaluate([ip1v, ip2v], [op1v, op2v])\n",
    "total_loss/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "146/146 [==============================] - 0s 2ms/step - loss: 8.7264 - dense_49_loss: 7.4870 - dense_51_loss: 1.2394\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4.363196849822998"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Evaluating model on Test\n",
    "total_loss, _, _ = ensemble_model.evaluate([ip1t, ip2t], [op1t, op2t])\n",
    "total_loss/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Predicting on Score\n",
    "score = ensemble_model.predict([ip1s, ip2s])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_y_hat = pd.DataFrame(np.array(score).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "score_y_hat = score_y_hat[[\"signal\", \"vehicle\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following asserts to check the type and shape of the final predictions.\n",
    "assert type(score_y_hat) == pd.DataFrame\n",
    "assert score_y_hat.shape == (score_metadata.shape[0], 2)\n",
    "assert (score_y_hat.columns == ['signal', 'vehicle']).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following code to save the final predictions.\n",
    "import os \n",
    "model_dir = 'carnet_model'\n",
    "os.makedirs(model_dir, exist_ok=True)\n",
    "score_y_hat.to_parquet(f'{model_dir}/score_y_hat.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Saving the Ensemble model\n",
    "with open(f'{model_dir}/keras_model.json', 'w') as f:\n",
    "    ensemble_model_json = json.dumps(json.loads(ensemble_model.to_json()), indent=True)\n",
    "    f.write(ensemble_model_json)\n",
    "ensemble_model.save_weights(f\"{model_dir}/keras_parameters.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alternate Way for Ensemble (Takes More Time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Load all individual models\n",
    "models = []\n",
    "for seed in seeds:\n",
    "    model_dir = 'weights_27/model{0}'.format(seed)\n",
    "    model = json.load(open(f'{model_dir}/keras_model.json'))\n",
    "    model = tf.keras.models.model_from_config(model)\n",
    "    model.load_weights(f\"{model_dir}/keras_parameters.h5\")\n",
    "    ### Freeze the layers\n",
    "    model.trainable = False\n",
    "    models.append(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Image as the input\n",
    "model_input = tf.keras.layers.Input((224, 224, 3))\n",
    "\n",
    "### Outputs of individual models\n",
    "model_outputs_1 = [model(model_input)[0] for model in models] ### Vehicles\n",
    "model_outputs_2 = [model(model_input)[1] for model in models] ### Signals\n",
    "\n",
    "### Creating individual models for vehicles and signals with dense layers and dropouts to reduce overfitting\n",
    "ensemble_output_1 = tf.keras.layers.Concatenate()(model_outputs_1)\n",
    "ensemble_output_1 = tf.keras.layers.Dense(128, activation = \"relu\")(ensemble_output_1)\n",
    "ensemble_output_1 = tf.keras.layers.Dropout(0.25)(ensemble_output_1)\n",
    "ensemble_output_1 = tf.keras.layers.Dense(1)(ensemble_output_1)\n",
    "\n",
    "ensemble_output_2 = tf.keras.layers.Concatenate()(model_outputs_2)\n",
    "ensemble_output_2 = tf.keras.layers.Dense(128, activation = \"relu\")(ensemble_output_2)\n",
    "ensemble_output_2 = tf.keras.layers.Dropout(0.25)(ensemble_output_2)\n",
    "ensemble_output_2 = tf.keras.layers.Dense(1)(ensemble_output_2)\n",
    "\n",
    "ensemble_model = tf.keras.Model(inputs=model_input, outputs=[ensemble_output_1, ensemble_output_2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 224, 224, 3  0           []                               \n",
      "                                )]                                                                \n",
      "                                                                                                  \n",
      " model (Functional)             [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_1 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_2 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_3 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_4 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_5 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " model_6 (Functional)           [(None, 1),          7099318     ['input_1[0][0]',                \n",
      "                                 (None, 1)]                       'input_1[0][0]']                \n",
      "                                                                                                  \n",
      " concatenate (Concatenate)      (None, 7)            0           ['model[0][0]',                  \n",
      "                                                                  'model_1[0][0]',                \n",
      "                                                                  'model_2[0][0]',                \n",
      "                                                                  'model_3[0][0]',                \n",
      "                                                                  'model_4[0][0]',                \n",
      "                                                                  'model_5[0][0]',                \n",
      "                                                                  'model_6[0][0]']                \n",
      "                                                                                                  \n",
      " concatenate_1 (Concatenate)    (None, 7)            0           ['model[1][1]',                  \n",
      "                                                                  'model_1[1][1]',                \n",
      "                                                                  'model_2[1][1]',                \n",
      "                                                                  'model_3[1][1]',                \n",
      "                                                                  'model_4[1][1]',                \n",
      "                                                                  'model_5[1][1]',                \n",
      "                                                                  'model_6[1][1]']                \n",
      "                                                                                                  \n",
      " dense (Dense)                  (None, 128)          1024        ['concatenate[0][0]']            \n",
      "                                                                                                  \n",
      " dense_1 (Dense)                (None, 128)          1024        ['concatenate_1[0][0]']          \n",
      "                                                                                                  \n",
      " dropout (Dropout)              (None, 128)          0           ['dense[0][0]']                  \n",
      "                                                                                                  \n",
      " dropout_1 (Dropout)            (None, 128)          0           ['dense_1[0][0]']                \n",
      "                                                                                                  \n",
      " dense_2 (Dense)                (None, 1)            129         ['dropout[0][0]']                \n",
      "                                                                                                  \n",
      " dense_3 (Dense)                (None, 1)            129         ['dropout_1[0][0]']              \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 49,697,532\n",
      "Trainable params: 2,306\n",
      "Non-trainable params: 49,695,226\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "ensemble_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Checkpoint path \n",
    "checkpoint_path = \"weights/car_27\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Adam, self).__init__(name, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "### Adam optimizer with 0.001 learning rate\n",
    "ensemble_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(\n",
    "        lr=0.001\n",
    "    ),\n",
    "    ### MSE loss\n",
    "    loss='mean_squared_error'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Whole train data\n",
    "train_single_dataset = build_dataset_labeled_single(train_metadata).batch(batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/128\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-31 02:44:50.732480: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8401\n",
      "2022-10-31 02:44:51.353386: W tensorflow/stream_executor/gpu/asm_compiler.cc:111] *** WARNING *** You are using ptxas 10.0.145, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.\n",
      "\n",
      "You may not need to update to CUDA 11.1; cherry-picking the ptxas binary is often sufficient.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    109/Unknown - 794s 6s/step - loss: 7.2389 - dense_2_loss: 6.4384 - dense_3_loss: 0.8005\n",
      "Epoch 1: val_loss improved from inf to 8.73772, saving model to weights/car_27_2\n",
      "109/109 [==============================] - 1061s 9s/step - loss: 7.2389 - dense_2_loss: 6.4384 - dense_3_loss: 0.8005 - val_loss: 8.7377 - val_dense_2_loss: 7.4917 - val_dense_3_loss: 1.2460\n",
      "Epoch 2/128\n",
      "109/109 [==============================] - ETA: 0s - loss: 5.4629 - dense_2_loss: 4.9267 - dense_3_loss: 0.5361\n",
      "Epoch 2: val_loss did not improve from 8.73772\n",
      "109/109 [==============================] - 919s 8s/step - loss: 5.4629 - dense_2_loss: 4.9267 - dense_3_loss: 0.5361 - val_loss: 8.8938 - val_dense_2_loss: 7.6510 - val_dense_3_loss: 1.2428\n",
      "Epoch 3/128\n",
      " 15/109 [===>..........................] - ETA: 9:57 - loss: 5.0414 - dense_2_loss: 4.5256 - dense_3_loss: 0.5158 "
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn [26], line 16\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;66;03m### Callback to stop the training if there's no improvement in validation loss for 3 epochs\u001b[39;00m\n\u001b[1;32m     15\u001b[0m stop_early \u001b[38;5;241m=\u001b[39m tf\u001b[38;5;241m.\u001b[39mkeras\u001b[38;5;241m.\u001b[39mcallbacks\u001b[38;5;241m.\u001b[39mEarlyStopping(monitor\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mval_loss\u001b[39m\u001b[38;5;124m'\u001b[39m, patience\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m3\u001b[39m)\n\u001b[0;32m---> 16\u001b[0m \u001b[43mensemble_model\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m     17\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtrain_single_dataset\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     18\u001b[0m \u001b[43m    \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m128\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m     19\u001b[0m \u001b[43m    \u001b[49m\u001b[43mshuffle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m     20\u001b[0m \u001b[43m    \u001b[49m\u001b[43mvalidation_data\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mval_single_dataset\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     21\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m[\u001b[49m\u001b[43mstop_early\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcp_callback\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcsv_logger\u001b[49m\u001b[43m]\u001b[49m\n\u001b[1;32m     22\u001b[0m \u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/keras/utils/traceback_utils.py:64\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m     63\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 64\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     65\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:  \u001b[38;5;66;03m# pylint: disable=broad-except\u001b[39;00m\n\u001b[1;32m     66\u001b[0m   filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/keras/engine/training.py:1384\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1377\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m   1378\u001b[0m     \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m   1379\u001b[0m     epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[1;32m   1380\u001b[0m     step_num\u001b[38;5;241m=\u001b[39mstep,\n\u001b[1;32m   1381\u001b[0m     batch_size\u001b[38;5;241m=\u001b[39mbatch_size,\n\u001b[1;32m   1382\u001b[0m     _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m   1383\u001b[0m   callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1384\u001b[0m   tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1385\u001b[0m   \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m   1386\u001b[0m     context\u001b[38;5;241m.\u001b[39masync_wait()\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    152\u001b[0m   filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/def_function.py:915\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    912\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    914\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 915\u001b[0m   result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    917\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m    918\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/def_function.py:947\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    944\u001b[0m   \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m    945\u001b[0m   \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m    946\u001b[0m   \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 947\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_stateless_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m  \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m    948\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stateful_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    949\u001b[0m   \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m    950\u001b[0m   \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m    951\u001b[0m   \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/function.py:2956\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2953\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m   2954\u001b[0m   (graph_function,\n\u001b[1;32m   2955\u001b[0m    filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m-> 2956\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   2957\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/function.py:1853\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1849\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m   1850\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m   1851\u001b[0m     \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m   1852\u001b[0m   \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1853\u001b[0m   \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1854\u001b[0m \u001b[43m      \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m   1855\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m   1856\u001b[0m     args,\n\u001b[1;32m   1857\u001b[0m     possible_gradient_type,\n\u001b[1;32m   1858\u001b[0m     executing_eagerly)\n\u001b[1;32m   1859\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/function.py:499\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    497\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    498\u001b[0m   \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 499\u001b[0m     outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    500\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    501\u001b[0m \u001b[43m        \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    502\u001b[0m \u001b[43m        \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    503\u001b[0m \u001b[43m        \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    504\u001b[0m \u001b[43m        \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    505\u001b[0m   \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    506\u001b[0m     outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m    507\u001b[0m         \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m    508\u001b[0m         num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    511\u001b[0m         ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m    512\u001b[0m         cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
      "File \u001b[0;32m~/SageMaker/setup/miniconda/envs/ucla_deeplearning/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:54\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     53\u001b[0m   ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 54\u001b[0m   tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     55\u001b[0m \u001b[43m                                      \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     56\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     57\u001b[0m   \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "### Training the Ensemble model\n",
    "np.random.seed(11)\n",
    "tf.random.set_seed(11)\n",
    "\n",
    "### Callback to save model log / results after each epoch\n",
    "csv_logger = tf.keras.callbacks.CSVLogger(\"model27_2_history_log.csv\", append=True)\n",
    "\n",
    "### Callback to save weights after every epoch i.e., checkpoint\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1, save_best_only=True, monitor = 'val_loss') \n",
    "\n",
    "### Callback to stop the training if there's no improvement in validation loss for 3 epochs\n",
    "stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n",
    "\n",
    "### Fitting the model\n",
    "ensemble_model.fit(\n",
    "    train_single_dataset,\n",
    "    epochs=128,\n",
    "    shuffle=True,\n",
    "    validation_data=(val_single_dataset),\n",
    "    callbacks=[stop_early, cp_callback, csv_logger]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7fa8abeac220>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Loading the weights from the best model\n",
    "ensemble_model.load_weights(checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "37/37 [==============================] - 230s 6s/step - loss: 8.7377 - dense_2_loss: 7.4917 - dense_3_loss: 1.2460\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[8.737717628479004, 7.491723537445068, 1.2459948062896729]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Evaluating model on Validation\n",
    "ensemble_model.evaluate(val_single_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "37/37 [==============================] - 230s 6s/step - loss: 8.8654 - dense_2_loss: 7.5516 - dense_3_loss: 1.3138\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[8.8654146194458, 7.551567554473877, 1.3138481378555298]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Evaluating model on Test\n",
    "ensemble_model.evaluate(test_single_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "score = ensemble_model.predict(score_single_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_y_hat = pd.DataFrame(np.array(score).reshape(2, -1).transpose(), columns = [\"vehicle\", \"signal\"])\n",
    "score_y_hat = score_y_hat[[\"signal\", \"vehicle\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following code to save the final predictions.\n",
    "import os \n",
    "model_dir = 'carnet_model'\n",
    "os.makedirs(model_dir, exist_ok=True)\n",
    "score_y_hat.to_parquet(f'{model_dir}/score_y_hat.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Saving the Ensemble model\n",
    "with open(f'{model_dir}/keras_model.json', 'w') as f:\n",
    "    ensemble_model_json = json.dumps(json.loads(ensemble_model.to_json()), indent=True)\n",
    "    f.write(ensemble_model_json)\n",
    "ensemble_model.save_weights(f\"{model_dir}/keras_parameters.h5\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ucla_deeplearning",
   "language": "python",
   "name": "ucla_deeplearning"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}