doc/tutorial/pipeline/nn_tutorial/Hetero-NN-Customize-Model.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0d19012c",
   "metadata": {},
   "source": [
    "# Hetero-NN Customize Models & Loss\n",
    "\n",
    "We can customize the top model and the bottom model in the Hetero NN.\n",
    "The model_zoo module was introduced in FATE 1.10, located under federatedml.nn.model_zoo. This module allows you to customize your own PyTorch model, provided that it is developed based on torch.nn.Module and implements the forward interface. For more information, see the PyTorch documentation  [Pytorch Module](https://pytorch.org/docs/stable/notes/modules.html#a-simple-custom-module) on custom modules. To use your custom model in a federated task, simply place it in the federatedml/nn/model_zoo directory and specify the module and model class through the interface when submitting the task. Hetero-NN components will automatically search and import the model you have implemented.\n",
    "\n",
    "You can also define your own loss class in a similar way. The you can place your loss class under the loss module, located under federatedml.nn.loss.\n",
    "We recommend you read these two tutorials before reading this tutorial: [Customize loss function](Homo-NN-Customize-Loss.ipynb), [Customize Model](Homo-NN-Customize-Model.ipynb)\n",
    "\n",
    "As an example, we consider reuse the task of MNIST handwriting recognition of last hetero-nn tutorial."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2122f107",
   "metadata": {},
   "source": [
    "## Prepare MNIST Data\n",
    "\n",
    "Please download the guest/host MNIST dataset from the link below and place it in the project examples/data folder:\n",
    "\n",
    "- guest data: https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate/examples/data/mnist_guest.zip\n",
    "\n",
    "- host data: https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate/examples/data/mnist_host.zip\n",
    "  \n",
    "The mnist_guest is a simplified version of the MNIST dataset, with a total of ten categories, which are classified into 0-9 10 folders according to labels. The mnist_host has the same images as the mnist_guest, but it is not labeled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "87b15585",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  1  2  3  4  5  6  7\t8  9\n"
     ]
    }
   ],
   "source": [
    "! ls ../../../../examples/data/mnist_guest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0eae2710",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not_labeled\n"
     ]
    }
   ],
   "source": [
    "! ls ../../../../examples/data/mnist_host"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a2cc655b",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "In version FATE-1.10, FATE introduces a new base class for datasets called Dataset, which is based on PyTorch's Dataset class. This class allows users to create custom datasets according to their specific needs. The usage is similar to that of PyTorch's Dataset class, with the added requirement of implementing two additional interfaces when using FATE-NN for data reading and training: load() and get_sample_ids().\n",
    "\n",
    "To create a custom dataset in Hetero-NN, users need to:\n",
    "\n",
    "- Develop a new dataset class that inherits from the Dataset class\n",
    "- Implement the \\_\\_len\\_\\_() and \\_\\_getitem\\_\\_() methods, which are consistent with PyTorch's Dataset usage. The \\_\\_len\\_\\_() method should return the length of the dataset, while the \\_\\_getitem\\_\\_() method should return the corresponding data at the specified index. **However, please notice that different \\_\\_getitem\\_\\_() methods may have different behaviors between different parties. In the guest party(party with labels), _\\_getitem\\_\\_() method return features and labels, while in the host parties(parties without label), _\\_getitem\\_\\_() method return features only.** \n",
    "- Implement the load(), get_sample_ids(), get_classes() methods\n",
    "  \n",
    "For those unfamiliar with PyTorch's Dataset class, more information can be found in the PyTorch documentation: [Pytorch Dataset Documentation](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html)\n",
    "\n",
    "## Customize Bottom/Top Model\n",
    "\n",
    "Name the model code bottom_net.py,  you can put it directly under federatedml/nn/model_zoo or use the shortcut interface of jupyter notebook: save_to_fate, to save it directly to federatedml/nn/model_zoo. This is the bottom model structure we define for feature extraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d2446d7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pipeline.component.nn import save_to_fate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7d4d6291",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%save_to_fate model bottom_net.py\n",
    "import torch as t\n",
    "from torch import nn\n",
    "from torch.nn import Module\n",
    "\n",
    "class BottomNet(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(BottomNet, self).__init__()\n",
    "        self.seq = t.nn.Sequential(\n",
    "            nn.Conv2d(in_channels=3, out_channels=12, kernel_size=5),\n",
    "            nn.MaxPool2d(kernel_size=3),\n",
    "            nn.Conv2d(in_channels=12, out_channels=12, kernel_size=3),\n",
    "            nn.AvgPool2d(kernel_size=3)\n",
    "        )\n",
    "        \n",
    "        self.fc = t.nn.Sequential(   # extracted feature is a 8-dim embedding\n",
    "            nn.Linear(48, 32),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32, 8),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.seq(x)\n",
    "        x = x.flatten(start_dim=1)\n",
    "        x = self.fc(x)\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1a75f275",
   "metadata": {},
   "source": [
    "And this is the top model we define for classification, we named it as top_model.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "12831d25",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%save_to_fate model top_net.py\n",
    "import torch as t\n",
    "from torch import nn\n",
    "from torch.nn import Module\n",
    "\n",
    "class TopNet(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(TopNet, self).__init__()\n",
    "        self.fc = t.nn.Sequential(   \n",
    "            nn.Linear(8, 10)\n",
    "        )\n",
    "        self.softmax = t.nn.Softmax(dim=1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc(x)\n",
    "        return self.softmax(x)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9267b633",
   "metadata": {},
   "source": [
    "## Use Cust Loss\n",
    "\n",
    "Using Cust Loss is exactly the same as Homo-NN, see: [Customize loss function](Homo-NN-Customize-Loss.ipynb). Here we use a new CrossEntropyLoss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e4abf940",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%save_to_fate loss ce.py\n",
    "import torch as t\n",
    "from federatedml.util import consts\n",
    "from torch.nn.functional import one_hot\n",
    "\n",
    "\n",
    "def cross_entropy(p2, p1, reduction='mean'):\n",
    "    p2 = p2 + consts.FLOAT_ZERO  # to avoid nan\n",
    "    assert p2.shape == p1.shape\n",
    "    if reduction == 'sum':\n",
    "        return -t.sum(p1 * t.log(p2))\n",
    "    elif reduction == 'mean':\n",
    "        return -t.mean(t.sum(p1 * t.log(p2), dim=1))\n",
    "    elif reduction == 'none':\n",
    "        return -t.sum(p1 * t.log(p2), dim=1)\n",
    "    else:\n",
    "        raise ValueError('unknown reduction')\n",
    "\n",
    "\n",
    "class CrossEntropyLoss(t.nn.Module):\n",
    "\n",
    "    \"\"\"\n",
    "    A CrossEntropy Loss that will not compute Softmax\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, reduction='mean'):\n",
    "        super(CrossEntropyLoss, self).__init__()\n",
    "        self.reduction = reduction\n",
    "\n",
    "    def forward(self, pred, label):\n",
    "\n",
    "        one_hot_label = one_hot(label.flatten())\n",
    "        loss_ = cross_entropy(pred, one_hot_label, self.reduction)\n",
    "\n",
    "        return loss_"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "11e272d8",
   "metadata": {},
   "source": [
    "Then, we can use our models & loss in the Hetero-NN MNIST task! The usage is the same as Homo-NN: we specify our model and loss by nn.CustModel and nn.CustLoss interfaces.\n",
    "\n",
    "## pipeline initialization\n",
    "\n",
    "Here we define the pipeline to run a hetero task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7184ed53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'namespace': 'experiment', 'table_name': 'mnist_host'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import torch as t\n",
    "from torch import nn\n",
    "from pipeline import fate_torch_hook\n",
    "from pipeline.component import HeteroNN\n",
    "from pipeline.component.hetero_nn import DatasetParam\n",
    "from pipeline.backend.pipeline import PipeLine\n",
    "from pipeline.component import Reader, Evaluation, DataTransform\n",
    "from pipeline.interface import Data, Model\n",
    "from pipeline.component.nn import save_to_fate\n",
    "\n",
    "fate_torch_hook(t)\n",
    "\n",
    "# bind path to fate name&namespace\n",
    "fate_project_path = os.path.abspath('../../../../')\n",
    "guest = 10000\n",
    "host = 9999\n",
    "\n",
    "pipeline_img = PipeLine().set_initiator(role='guest', party_id=guest).set_roles(guest=guest, host=host)\n",
    "\n",
    "guest_data = {\"name\": \"mnist_guest\", \"namespace\": \"experiment\"}\n",
    "host_data = {\"name\": \"mnist_host\", \"namespace\": \"experiment\"}\n",
    "\n",
    "guest_data_path = fate_project_path + '/examples/data/mnist_guest/'\n",
    "host_data_path = fate_project_path + '/examples/data/mnist_host/'\n",
    "pipeline_img.bind_table(name='mnist_guest', namespace='experiment', path=guest_data_path)\n",
    "pipeline_img.bind_table(name='mnist_host', namespace='experiment', path=host_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "94f480ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "guest_data = {\"name\": \"mnist_guest\", \"namespace\": \"experiment\"}\n",
    "host_data = {\"name\": \"mnist_host\", \"namespace\": \"experiment\"}\n",
    "reader_0 = Reader(name=\"reader_0\")\n",
    "reader_0.get_party_instance(role='guest', party_id=guest).component_param(table=guest_data)\n",
    "reader_0.get_party_instance(role='host', party_id=host).component_param(table=host_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "38f41dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "hetero_nn_0 = HeteroNN(name=\"hetero_nn_0\", epochs=3,\n",
    "                       interactive_layer_lr=0.01, batch_size=512, task_type='classification', seed=100\n",
    "                       )\n",
    "guest_nn_0 = hetero_nn_0.get_party_instance(role='guest', party_id=guest)\n",
    "host_nn_0 = hetero_nn_0.get_party_instance(role='host', party_id=host)\n",
    "\n",
    "# define model\n",
    "# use cust model here\n",
    "# our simple classification model:\n",
    "guest_bottom = t.nn.CustModel(module_name='bottom_net.py', class_name='BottomNet')\n",
    "\n",
    "# use cust model here\n",
    "host_bottom = t.nn.CustModel(module_name='bottom_net.py', class_name='BottomNet')\n",
    "\n",
    "# use new top model here\n",
    "guest_top = t.nn.CustModel(module_name='top_net.py', class_name='TopNet')\n",
    "\n",
    "# interactive layer define\n",
    "interactive_layer = t.nn.InteractiveLayer(out_dim=8, guest_dim=8, host_dim=8)\n",
    "\n",
    "# add models\n",
    "guest_nn_0.add_top_model(guest_top)\n",
    "guest_nn_0.add_bottom_model(guest_bottom)\n",
    "host_nn_0.add_bottom_model(host_bottom)\n",
    "\n",
    "# opt, loss\n",
    "optimizer = t.optim.Adam(lr=0.01) \n",
    "loss = t.nn.CustLoss(loss_module_name='ce', class_name='CrossEntropyLoss')\n",
    "\n",
    "# use DatasetParam to specify dataset and pass parameters\n",
    "guest_nn_0.add_dataset(DatasetParam(dataset_name='image', return_label=True))\n",
    "host_nn_0.add_dataset(DatasetParam(dataset_name='image', return_label=False))\n",
    "\n",
    "hetero_nn_0.set_interactive_layer(interactive_layer)\n",
    "hetero_nn_0.compile(optimizer=optimizer, loss=loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "cb09b949",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pipeline.backend.pipeline.PipeLine at 0x7f3f363f5970>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_img.add_component(reader_0)\n",
    "pipeline_img.add_component(hetero_nn_0, data=Data(train_data=reader_0.output.data))\n",
    "pipeline_img.add_component(Evaluation(name='eval_0', eval_type='multi'), data=Data(data=hetero_nn_0.output.data))\n",
    "pipeline_img.compile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "0987a621",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2022-12-24 21:26:06.571\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m83\u001b[0m - \u001b[1mJob id is 202212242126060352040\n",
      "\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:06.584\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m98\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KJob is still waiting, time elapse: 0:00:00\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:07.598\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m98\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KJob is still waiting, time elapse: 0:00:01\u001b[0m\n",
      "\u001b[0mm2022-12-24 21:26:08.622\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m125\u001b[0m - \u001b[1m\n",
      "\u001b[32m2022-12-24 21:26:08.624\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:02\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:09.648\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:03\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:10.671\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:04\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:11.697\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:05\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:12.720\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:06\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:13.738\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component reader_0, time elapse: 0:00:07\u001b[0m\n",
      "\u001b[0mm2022-12-24 21:26:15.888\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m125\u001b[0m - \u001b[1m\n",
      "\u001b[32m2022-12-24 21:26:15.890\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:09\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:17.144\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:10\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:18.180\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:11\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:19.238\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:12\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:20.274\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:13\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:21.303\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:14\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:22.334\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:15\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:23.365\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:16\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:24.389\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:17\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:25.420\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:18\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:26.460\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:19\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:27.501\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:20\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:28.587\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:22\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:29.624\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:23\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:30.651\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:24\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:31.674\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:25\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:32.721\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:26\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:33.741\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:27\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:34.774\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:28\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:35.799\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:29\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:36.825\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:30\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:37.847\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:31\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:38.870\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:32\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:39.902\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:33\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:40.935\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:34\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:41.970\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:35\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:42.998\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:36\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:44.044\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:37\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:45.076\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:38\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:46.100\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:39\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:47.123\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:40\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:48.142\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:41\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:49.202\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:42\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:50.223\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:43\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:51.245\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:44\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:52.269\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:45\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:53.289\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:46\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:54.327\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:47\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:55.348\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:48\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:56.372\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:49\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:57.393\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:50\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:58.415\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:51\u001b[0m\n",
      "\u001b[32m2022-12-24 21:26:59.437\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:52\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:00.460\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:53\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:01.480\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:54\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:02.502\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:55\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:03.568\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:56\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:04.591\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:58\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:05.618\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:00:59\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:06.638\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:00\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:07.659\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:01\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:08.678\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:02\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:09.700\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:03\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:10.723\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:04\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:11.746\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:05\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:12.768\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:06\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:13.788\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:07\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:14.853\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:08\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:15.883\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:09\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:16.935\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:10\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:17.973\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:11\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:19.000\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:12\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:20.022\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:13\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:21.043\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:14\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:22.071\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:15\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:23.107\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:16\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:24.144\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:17\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:25.163\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:18\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:26.187\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:19\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:27.210\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:20\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:28.234\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:21\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:29.258\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:22\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:30.324\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:23\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:31.355\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:24\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:32.388\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:25\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:33.408\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:26\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:34.431\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:27\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:35.454\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:28\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:36.567\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:29\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:37.609\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:31\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:38.641\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:32\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:39.689\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:33\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:40.716\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:34\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:41.741\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:35\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:42.764\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:36\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:43.785\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:37\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:44.805\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:38\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:45.826\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:39\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:46.844\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component hetero_nn_0, time elapse: 0:01:40\u001b[0m\n",
      "\u001b[0mm2022-12-24 21:27:47.923\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m125\u001b[0m - \u001b[1m\n",
      "\u001b[32m2022-12-24 21:27:47.925\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:41\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:48.950\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:42\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:50.007\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:43\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:51.033\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:44\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:52.053\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:45\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:53.076\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:46\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:54.122\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m127\u001b[0m - \u001b[1m\u001b[80D\u001b[1A\u001b[KRunning component eval_0, time elapse: 0:01:47\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:56.202\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m89\u001b[0m - \u001b[1mJob is success!!! Job id is 202212242126060352040\u001b[0m\n",
      "\u001b[32m2022-12-24 21:27:56.204\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpipeline.utils.invoker.job_submitter\u001b[0m:\u001b[36mmonitor_job_status\u001b[0m:\u001b[36m90\u001b[0m - \u001b[1mTotal time: 0:01:49\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "pipeline_img.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "08f7c75c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>label</th>\n",
       "      <th>predict_result</th>\n",
       "      <th>predict_score</th>\n",
       "      <th>predict_detail</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>img_1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>img_3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>img_4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>img_5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>img_6</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1304</th>\n",
       "      <td>img_32537</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1305</th>\n",
       "      <td>img_32558</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306</th>\n",
       "      <td>img_32563</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1307</th>\n",
       "      <td>img_32565</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1308</th>\n",
       "      <td>img_32573</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.12622389197349548</td>\n",
       "      <td>{'0': 0.07662956416606903, '1': 0.126223891973...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1309 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             id label predict_result        predict_score  \\\n",
       "0         img_1     0              1  0.12622389197349548   \n",
       "1         img_3     4              1  0.12622389197349548   \n",
       "2         img_4     0              1  0.12622389197349548   \n",
       "3         img_5     0              1  0.12622389197349548   \n",
       "4         img_6     7              1  0.12622389197349548   \n",
       "...         ...   ...            ...                  ...   \n",
       "1304  img_32537     1              1  0.12622389197349548   \n",
       "1305  img_32558     1              1  0.12622389197349548   \n",
       "1306  img_32563     1              1  0.12622389197349548   \n",
       "1307  img_32565     1              1  0.12622389197349548   \n",
       "1308  img_32573     1              1  0.12622389197349548   \n",
       "\n",
       "                                         predict_detail   type  \n",
       "0     {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "1     {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "2     {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "3     {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "4     {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "...                                                 ...    ...  \n",
       "1304  {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "1305  {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "1306  {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "1307  {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "1308  {'0': 0.07662956416606903, '1': 0.126223891973...  train  \n",
       "\n",
       "[1309 rows x 6 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_img.get_component('hetero_nn_0').get_output_data()  # get result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86810b0e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13 (default, Mar 28 2022, 11:38:47) \n[GCC 7.5.0]"
  },
  "vscode": {
   "interpreter": {
    "hash": "d29574a2ab71ec988cdcd4d29c58400bd2037cad632b9528d973466f7fb6f853"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}