vectorization05b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# introduction to the vectorizing principle\n",
    "\n",
    "## part 5b: generating SQL and dataframe transformation code with one syntax - an example pipeline\n",
    "\n",
    "While pandas is the most popular library for describing data transformations in python, there are more which you will also see in following parts. Relational databases are also a means of highly efficient processing of tabular data. They are typically instructed with a declarative language called SQL which allows for highly sophisticated query engines to first plan how to execute the query. From a large search space of possible execution plans they chose the supposedly best one by using computation cost estimators.\n",
    "\n",
    "For having a quick look at the data or for searching bugs in previously executed transformation code, it is highly efficient to write ad-hoc SQL scripts by hand. However, for better software engineering practices such as testing, programmatically creating SQL is the preferred option. In fact, it would be ideal to have one syntax from which both dataframe transformation code and SQL can be generated. This is what we show in this part.\n",
    "\n",
    "The library [pydiverse.transform](https://github.com/pydiverse/pydiverse.transform/) is used here even though it is early stage and still lacks essential features. It is intended as a proof of concept for this idea.\n",
    "\n",
    "Pandas uses the fluent interface technique chaining functional methods with multiple function calls: `df.assign(x=lambda df: df.int_col*2).loc[lambda df: df.x == 4]`\n",
    "\n",
    "pydiverse.transform uses the pipe syntax instead, similar to its precursors [dplyr from R](https://dplyr.tidyverse.org/) and [siuba](https://siuba.org/): `df >> mutate(x=λ.int_col*2) >> filter(λ.x == 4)`\n",
    "\n",
    "pydiverse.transform also supports creating custom verbs with the `@verb` decorator which then can be used within a pipe just like builtin verbs such as `mutate` or `filter`.\n",
    "\n",
    "Detail: In SQL, joining a table does not add any new columns to the select statement. It rather brings a new table within the namespace that can be used for creating select columns. pydiverse.transform supports this feature by dropping columns with an empty `select()` statement which does not remove them from the namespace for further use in `mutate()` statements.\n",
    "\n",
    "#### define tasks and helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import pydiverse.transform as pdt\n",
    "import xgboost\n",
    "import xgboost as xgb\n",
    "from pydiverse.pipedag import Blob, Flow, Stage, Table, materialize\n",
    "from pydiverse.transform import aligned, λ\n",
    "from pydiverse.transform.core.dtypes import String\n",
    "from pydiverse.transform.core.verbs import (\n",
    "    alias,\n",
    "    collect,\n",
    "    filter,\n",
    "    left_join,\n",
    "    mutate,\n",
    "    select,\n",
    ")\n",
    "from pydiverse.transform.eager import PandasTableImpl\n",
    "from pydiverse.transform.lazy import SQLTableImpl\n",
    "\n",
    "\n",
    "@pdt.verb\n",
    "def transmute(tbl, **kwargs):\n",
    "    return tbl >> select() >> mutate(**kwargs)\n",
    "\n",
    "\n",
    "@pdt.verb\n",
    "def trim_all_str(tbl):\n",
    "    for col in tbl:\n",
    "        if isinstance(col._.dtype, String):\n",
    "            tbl[col] = col.strip()\n",
    "    return tbl\n",
    "\n",
    "\n",
    "def pk(x: pdt.Table):\n",
    "    # This is just a placeholder.\n",
    "    # Ideally there would be a global function in pydiverse transform to\n",
    "    # get the primary key (and another one to get the table / col name)\n",
    "    return x.pk\n",
    "\n",
    "\n",
    "def pk_match(x: pdt.Table, y: pdt.Table):\n",
    "    return pk(x) == pk(y)\n",
    "\n",
    "\n",
    "def get_named_tables(tables: list[pdt.Table]) -> dict[str, pdt.Table]:\n",
    "    return {tbl._impl.name: tbl for tbl in tables}\n",
    "\n",
    "\n",
    "@materialize(version=\"1.0.0\")\n",
    "def read_input_data(src_dir=\"data/pipedag_example_data\"):\n",
    "    return [\n",
    "        Table(pd.read_csv(os.path.join(src_dir, file)), name=file.removesuffix(\".csv.gz\"))\n",
    "        for file in os.listdir(src_dir)\n",
    "        if file.endswith(\".csv.gz\")\n",
    "    ]\n",
    "\n",
    "\n",
    "@materialize(input_type=SQLTableImpl, lazy=True)\n",
    "def clean(src_tbls: list[pdt.Table]):\n",
    "    return [tbl >> trim_all_str() for tbl in src_tbls]\n",
    "\n",
    "\n",
    "@materialize(input_type=SQLTableImpl, lazy=True, nout=3)\n",
    "def transform(src_tbls: list[pdt.Table]):\n",
    "    named_tbls = get_named_tables(src_tbls)\n",
    "    a = named_tbls[\"a\"]\n",
    "    b = named_tbls[\"b\"]\n",
    "    c = named_tbls[\"c\"]\n",
    "\n",
    "    def join_b(a):\n",
    "        return a >> left_join(b >> select(), pk_match(a, b))\n",
    "\n",
    "    new_a = join_b(a) >> mutate(x=b.x)\n",
    "    new_b = b\n",
    "    new_c = c\n",
    "\n",
    "    return new_a, new_b, new_c\n",
    "\n",
    "\n",
    "@materialize(input_type=SQLTableImpl, lazy=True)\n",
    "def lazy_features(a: pdt.Table, src_tbls: list[pdt.Table]):\n",
    "    named_tbls = get_named_tables(src_tbls)\n",
    "    b = named_tbls[\"b\"]\n",
    "    return (\n",
    "        a\n",
    "        >> left_join(b, pk_match(a, b))\n",
    "        >> transmute(pk=pk(a), aiige=a.age, y=b.y, z=b.z * 2)\n",
    "        >> alias(\"lazy_features\")\n",
    "    )\n",
    "\n",
    "\n",
    "@materialize(input_type=PandasTableImpl, version=\"2.3.5\")\n",
    "def eager_features(a: pdt.Table, src_tbls: list[pdt.Table]):\n",
    "    named_tbls = get_named_tables(src_tbls)\n",
    "    c = named_tbls[\"c\"]\n",
    "    return (\n",
    "        a\n",
    "        >> left_join(c, pk_match(a, c))\n",
    "        >> transmute(pk=pk(a), xx=c.x, yy=c.y * 2, zz=c.z + 3)\n",
    "        >> alias(\"eager_features\")\n",
    "    )\n",
    "\n",
    "\n",
    "@materialize(input_type=SQLTableImpl, lazy=True)\n",
    "def combine_features(features1: pdt.Table, features2: pdt.Table):\n",
    "    return (\n",
    "        features1\n",
    "        >> left_join(\n",
    "            features2 >> select(-pk(features2)), pk_match(features1, features2)\n",
    "        )\n",
    "        >> alias(\"features\")\n",
    "    )\n",
    "\n",
    "\n",
    "@materialize(input_type=SQLTableImpl, lazy=True, nout=2)\n",
    "def train_and_test_set(flat_table: pdt.Table, features: pdt.Table):\n",
    "    tbl = (\n",
    "        flat_table\n",
    "        >> left_join(features, pk_match(flat_table, features))\n",
    "        >> mutate(row_num=pdt.functions.row_number(arrange=[pk(flat_table)]))\n",
    "        >> select(-pk(flat_table), -pk(features))\n",
    "    )\n",
    "\n",
    "    training_set = (\n",
    "        tbl\n",
    "        >> filter(λ.row_num % 10 != 0)\n",
    "        >> select(-λ.row_num)\n",
    "        >> alias(\"training_set\")\n",
    "    )\n",
    "    test_set = (\n",
    "        tbl >> filter(λ.row_num % 10 == 0) >> select(-λ.row_num) >> alias(\"test_set\")\n",
    "    )\n",
    "\n",
    "    return (training_set, test_set)\n",
    "\n",
    "\n",
    "@materialize(input_type=pd.DataFrame, version=\"4.5.8\")\n",
    "def model_training(train_set: pd.DataFrame):\n",
    "    x = train_set.drop(\"target\", axis=1)\n",
    "    y = train_set[\"target\"]\n",
    "    dtrain = xgb.DMatrix(x, label=y)\n",
    "\n",
    "    params = {\"max_depth\": 2, \"eta\": 1, \"objective\": \"binary:logistic\"}\n",
    "    model = xgb.train(params, dtrain)\n",
    "\n",
    "    return Blob(model, \"model\")\n",
    "\n",
    "\n",
    "@aligned(with_=\"test_set\")\n",
    "def predict(model: xgboost.Booster, test_set: pdt.Table):\n",
    "    x = test_set >> select(-λ.target) >> collect()\n",
    "\n",
    "    # Ugly hack to convert new pandas dtypes to numpy dtypes, because xgboost\n",
    "    # requires numpy dtypes.\n",
    "    x = x.astype(\n",
    "        x.dtypes.map(lambda d: d.numpy_dtype if hasattr(d, \"numpy_dtype\") else d)\n",
    "    )\n",
    "\n",
    "    dx = xgb.DMatrix(x)\n",
    "    predict_col = model.predict(dx)\n",
    "\n",
    "    return pdt.Table(\n",
    "        PandasTableImpl(\"prediction\", pd.DataFrame({\"prediction\": predict_col}))\n",
    "    ).prediction\n",
    "\n",
    "\n",
    "@materialize(input_type=PandasTableImpl, version=\"3.4.5\")\n",
    "def model_evaluation(model: xgboost.Booster, test_set: pdt.Table):\n",
    "    prediction = predict(model, test_set)  # produces an aligned vector with input\n",
    "    return (\n",
    "        test_set\n",
    "        >> select(λ.target)\n",
    "        >> mutate(prediction=prediction)\n",
    "        >> mutate(abs_error=abs(λ.target - λ.prediction))\n",
    "        >> alias(\"evaluation\")\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### define pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def get_pipeline():\n",
    "    with Flow(\"flow\") as flow:\n",
    "        with Stage(\"1_raw_input\"):\n",
    "            raw_tbls = read_input_data()\n",
    "\n",
    "        with Stage(\"2_clean_input\"):\n",
    "            clean_tbls = clean(raw_tbls)\n",
    "\n",
    "        with Stage(\"3_transformed_data\"):\n",
    "            a, b, c = transform(clean_tbls)\n",
    "\n",
    "        with Stage(\"4_features\"):\n",
    "            features1 = lazy_features(a, [a, b, c])  # s3.tbls\n",
    "            features2 = eager_features(a, [a, b, c])  # s3.tbls\n",
    "            features = combine_features(features1, features2)\n",
    "\n",
    "        with Stage(\"5_model\"):\n",
    "            train_set, test_set = train_and_test_set(a, features)\n",
    "            model = model_training(train_set)\n",
    "\n",
    "        with Stage(\"6_evaluation\"):\n",
    "            _ = model_evaluation(model, test_set)\n",
    "\n",
    "    return flow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### setup logging:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "from pydiverse.pipedag.util.structlog import setup_logging\n",
    "setup_logging(log_level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### run pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m2024-08-06 09:34:52.001795\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mInitialized SQL Table Store   \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mengine_url\u001b[0m=\u001b[35mduckdb:////tmp/pipedag/vectorization/db.duckdb\u001b[0m \u001b[36mschema_prefix\u001b[0m=\u001b[35m\u001b[0m \u001b[36mschema_suffix\u001b[0m=\u001b[35m\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.007518\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mInitialized SQL Table Store   \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mengine_url\u001b[0m=\u001b[35mduckdb:////tmp/pipedag/vectorization/db.duckdb\u001b[0m \u001b[36mschema_prefix\u001b[0m=\u001b[35m\u001b[0m \u001b[36mschema_suffix\u001b[0m=\u001b[35m\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.007984\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mStarting IPCServer            \u001b[0m [\u001b[0m\u001b[1m\u001b[34mRunContextServer\u001b[0m]\u001b[0m \u001b[36maddress\u001b[0m=\u001b[35mtcp://127.0.0.1:52807\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.030584\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS pipedag_metadata\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.036494\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"1_raw_input\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.037299\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"1_raw_input__even\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.038302\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"1_raw_input__even\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.054449\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"1_raw_input__even\".c AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"1_raw_input\".c\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.059408\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"1_raw_input__even\".a AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"1_raw_input\".a\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.064163\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"1_raw_input__even\".b AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"1_raw_input\".b\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.067606\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFound task in cache. Using cached result.\u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'read_input_data'\u001b[0m]\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'read_input_data' 0x108381c50 (id: 0)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.068244\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'read_input_data'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.CACHE_VALID: 2>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'read_input_data' 0x108381c50 (id: 0)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.068894\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 1_raw_input>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.070168\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mStage is cache valid          \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 1_raw_input>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.073781\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 1_raw_input>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.075844\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"2_clean_input\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.076366\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"2_clean_input__odd\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.076974\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"2_clean_input__odd\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.079888\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve table from local table cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mParquetTableCache\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCan't retrieve Table as type <class 'pydiverse.transform.lazy.sql_table.sql_table.SQLTableImpl'>. This is either because no TableHook has been registered for this type, or because not all requirements have been met for the corresponding hook.\n",
      "Hooks with unmet requirements: pydiverse.pipedag.backend.table.sql.hooks.TidyPolarsTableHook\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35m<Table 'c' (1_raw_input)>\u001b[0m\n",
      "/Users/martin/micromamba/envs/vectorization/lib/python3.11/site-packages/duckdb_engine/__init__.py:173: DuckDBEngineWarning: duckdb-engine doesn't yet support reflection on indices\n",
      "  warnings.warn(\n",
      "\u001b[2m2024-08-06 09:34:52.212301\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve table from local table cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mParquetTableCache\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCan't retrieve Table as type <class 'pydiverse.transform.lazy.sql_table.sql_table.SQLTableImpl'>. This is either because no TableHook has been registered for this type, or because not all requirements have been met for the corresponding hook.\n",
      "Hooks with unmet requirements: pydiverse.pipedag.backend.table.sql.hooks.TidyPolarsTableHook\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35m<Table 'a' (1_raw_input)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.336923\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve table from local table cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mParquetTableCache\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCan't retrieve Table as type <class 'pydiverse.transform.lazy.sql_table.sql_table.SQLTableImpl'>. This is either because no TableHook has been registered for this type, or because not all requirements have been met for the corresponding hook.\n",
      "Hooks with unmet requirements: pydiverse.pipedag.backend.table.sql.hooks.TidyPolarsTableHook\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35m<Table 'b' (1_raw_input)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.466560\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"2_clean_input__odd\".c AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"2_clean_input\".c\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.468745\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'c' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.473845\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"2_clean_input__odd\".a AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"2_clean_input\".a\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.475332\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'a' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.480277\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"2_clean_input__odd\".b AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"2_clean_input\".b\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.481899\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'b' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.484958\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'clean'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.CACHE_VALID: 2>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'clean' 0x108377a90 (id: 2)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.485658\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 2_clean_input>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.486852\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mStage is cache valid          \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 2_clean_input>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.490079\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 2_clean_input>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.492050\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"3_transformed_data\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.492570\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"3_transformed_data__odd\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.493208\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"3_transformed_data__odd\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.883055\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"3_transformed_data__odd\".a AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"3_transformed_data\".a\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.885275\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'a' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.890715\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"3_transformed_data__odd\".b AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"3_transformed_data\".b\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.892473\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'b' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.898463\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"3_transformed_data__odd\".c AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"3_transformed_data\".c\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.900129\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'c' found \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.903698\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'transform'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.CACHE_VALID: 2>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'transform' 0x108377f50 (id: 4)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.904446\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 3_transformed_data>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.905829\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mStage is cache valid          \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 3_transformed_data>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.908995\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 3_transformed_data>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.911505\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"4_features\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.912216\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"4_features__odd\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:52.912962\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"4_features__odd\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.282021\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"4_features__odd\".lazy_features AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features\".lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.284371\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mLazy cache of table 'lazy_features' found\u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.288256\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'lazy_features'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.CACHE_VALID: 2>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'lazy_features' 0x108396ed0 (id: 6)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.294813\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"4_features__odd\".eager_features AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features\".eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.298568\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFound task in cache. Using cached result.\u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'eager_features'\u001b[0m]\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'eager_features' 0x108397390 (id: 7)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.299138\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'eager_features'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.CACHE_VALID: 2>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'eager_features' 0x108397390 (id: 7)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.433440\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve table from local table cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mParquetTableCache\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCan't retrieve Table as type <class 'pydiverse.transform.lazy.sql_table.sql_table.SQLTableImpl'>. This is either because no TableHook has been registered for this type, or because not all requirements have been met for the corresponding hook.\n",
      "Hooks with unmet requirements: pydiverse.pipedag.backend.table.sql.hooks.TidyPolarsTableHook\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35m<Table 'eager_features' (4_features)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.565774\u001b[0m [\u001b[33m\u001b[1mwarning  \u001b[0m] \u001b[1mCache miss                    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mNo result found for lazy table cache key\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m4_features\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35mfeatures\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.568662\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE TABLE \"4_features__odd\".features AS\n",
      "    \u001b[35mSELECT features.pk AS pk, features.aiige AS aiige, features.y AS y, features.z AS z, features.xx_eager_features AS xx_eager_features, features.yy_eager_features AS yy_eager_features, features.zz_eager_features AS zz_eager_features \n",
      "    \u001b[35mFROM (SELECT lazy_features.pk AS pk, lazy_features.aiige AS aiige, lazy_features.y AS y, lazy_features.z AS z, eager_features.xx AS xx_eager_features, eager_features.yy AS yy_eager_features, eager_features.zz AS zz_eager_features \n",
      "    \u001b[35mFROM \"4_features__odd\".lazy_features AS lazy_features LEFT OUTER JOIN \"4_features__odd\".eager_features AS eager_features ON lazy_features.pk = eager_features.pk) AS features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.570406\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE TABLE \"4_features__odd\".lazy_features__copy AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features\".lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.573599\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE TABLE \"4_features__odd\".eager_features__copy AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features\".eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.576833\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'combine_features'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'combine_features' 0x108396550 (id: 8)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.577586\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 4_features>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.578352\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"4_features__odd\".lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.578561\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"4_features__odd\".eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.579593\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mALTER TABLE \"4_features__odd\".lazy_features__copy RENAME TO lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.580084\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mALTER TABLE \"4_features__odd\".eager_features__copy RENAME TO eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.588104\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"4_features\".eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.588532\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"4_features\".features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.588865\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"4_features\".lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.593518\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"4_features\".eager_features AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features__odd\".eager_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.594411\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"4_features\".features AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features__odd\".features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.595115\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"4_features\".lazy_features AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"4_features__odd\".lazy_features\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.599453\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 4_features>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.601598\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"5_model\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.602102\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"5_model__odd\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.602692\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"5_model__odd\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.867166\u001b[0m [\u001b[33m\u001b[1mwarning  \u001b[0m] \u001b[1mCache miss                    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mNo result found for lazy table cache key\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m5_model\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35mtraining_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.868869\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE TABLE \"5_model__odd\".training_set AS\n",
      "    \u001b[35mSELECT training_set.age AS age, training_set.target AS target, training_set.x AS x, training_set.aiige_features AS aiige_features, training_set.y_features AS y_features, training_set.z_features AS z_features, training_set.xx_eager_features_features AS xx_eager_features_features, training_set.yy_eager_features_features AS yy_eager_features_features, training_set.zz_eager_features_features AS zz_eager_features_features \n",
      "    \u001b[35mFROM (SELECT anon_1.age AS age, anon_1.target AS target, anon_1.x AS x, anon_1.aiige_features AS aiige_features, anon_1.y_features AS y_features, anon_1.z_features AS z_features, anon_1.xx_eager_features_features AS xx_eager_features_features, anon_1.yy_eager_features_features AS yy_eager_features_features, anon_1.zz_eager_features_features AS zz_eager_features_features \n",
      "    \u001b[35mFROM (SELECT a.age AS age, a.target AS target, a.x AS x, features.aiige AS aiige_features, features.y AS y_features, features.z AS z_features, features.xx_eager_features AS xx_eager_features_features, features.yy_eager_features AS yy_eager_features_features, features.zz_eager_features AS zz_eager_features_features, ROW_NUMBER() OVER (ORDER BY a.pk ASC NULLS LAST) AS row_num, a.pk AS pk, features.pk AS pk_features \n",
      "    \u001b[35mFROM \"3_transformed_data\".a AS a LEFT OUTER JOIN \"4_features\".features AS features ON a.pk = features.pk) AS anon_1 \n",
      "    \u001b[35mWHERE anon_1.row_num % 10 != 0) AS training_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.874299\u001b[0m [\u001b[33m\u001b[1mwarning  \u001b[0m] \u001b[1mCache miss                    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mNo result found for lazy table cache key\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m5_model\u001b[0m \u001b[36mtable\u001b[0m=\u001b[35mtest_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.876002\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE TABLE \"5_model__odd\".test_set AS\n",
      "    \u001b[35mSELECT test_set.age AS age, test_set.target AS target, test_set.x AS x, test_set.aiige_features AS aiige_features, test_set.y_features AS y_features, test_set.z_features AS z_features, test_set.xx_eager_features_features AS xx_eager_features_features, test_set.yy_eager_features_features AS yy_eager_features_features, test_set.zz_eager_features_features AS zz_eager_features_features \n",
      "    \u001b[35mFROM (SELECT anon_1.age AS age, anon_1.target AS target, anon_1.x AS x, anon_1.aiige_features AS aiige_features, anon_1.y_features AS y_features, anon_1.z_features AS z_features, anon_1.xx_eager_features_features AS xx_eager_features_features, anon_1.yy_eager_features_features AS yy_eager_features_features, anon_1.zz_eager_features_features AS zz_eager_features_features \n",
      "    \u001b[35mFROM (SELECT a.age AS age, a.target AS target, a.x AS x, features.aiige AS aiige_features, features.y AS y_features, features.z AS z_features, features.xx_eager_features AS xx_eager_features_features, features.yy_eager_features AS yy_eager_features_features, features.zz_eager_features AS zz_eager_features_features, ROW_NUMBER() OVER (ORDER BY a.pk ASC NULLS LAST) AS row_num, a.pk AS pk, features.pk AS pk_features \n",
      "    \u001b[35mFROM \"3_transformed_data\".a AS a LEFT OUTER JOIN \"4_features\".features AS features ON a.pk = features.pk) AS anon_1 \n",
      "    \u001b[35mWHERE anon_1.row_num % 10 = 0) AS test_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.881355\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'train_and_test_set'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'train_and_test_set' 0x108396690 (id: 10)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:53.884207\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve task from cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'model_training'\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCouldn't retrieve task from cache: <Task 'model_training' 0x108385250 (id: 11)>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'model_training' 0x108385250 (id: 11)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.028576\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'model_training'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'model_training' 0x108385250 (id: 11)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.029252\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 5_model>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.036865\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"5_model\".test_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.037354\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"5_model\".training_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.044409\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"5_model\".test_set AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"5_model__odd\".test_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.045322\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"5_model\".training_set AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"5_model__odd\".training_set\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.049624\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 5_model>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.051499\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA IF NOT EXISTS \"6_evaluation\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.051990\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP SCHEMA IF EXISTS \"6_evaluation__odd\" CASCADE\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.052477\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE SCHEMA \"6_evaluation__odd\"\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.056263\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFailed to retrieve task from cache\u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'model_evaluation'\u001b[0m]\u001b[0m \u001b[36mcause\u001b[0m=\u001b[35mCouldn't retrieve task from cache: <Task 'model_evaluation' 0x108384f50 (id: 13)>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'model_evaluation' 0x108384f50 (id: 13)>\u001b[0m\n",
      "/Users/martin/micromamba/envs/vectorization/lib/python3.11/site-packages/duckdb_engine/__init__.py:173: DuckDBEngineWarning: duckdb-engine doesn't yet support reflection on indices\n",
      "  warnings.warn(\n",
      "\u001b[2m2024-08-06 09:34:54.200576\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mWriting table '6_evaluation__odd.evaluation'\u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mtable_obj\u001b[0m]\n",
      "    \u001b[35m    target  prediction  abs_error\n",
      "    \u001b[35m0        1    0.999585   0.000415\n",
      "    \u001b[35m1        0    0.000732   0.000732\n",
      "    \u001b[35m2        0    0.000278   0.000278\n",
      "    \u001b[35m3        1    0.997495   0.002505\n",
      "    \u001b[35m4        0    0.029311   0.029311\n",
      "    \u001b[35m..     ...         ...        ...\n",
      "    \u001b[35m95       0    0.003391   0.003391\n",
      "    \u001b[35m96       0    0.006547   0.006547\n",
      "    \u001b[35m97       1    0.999585   0.000415\n",
      "    \u001b[35m98       1    0.999837   0.000163\n",
      "    \u001b[35m99       0    0.000732   0.000732\n",
      "\n",
      "    \u001b[35m[100 rows x 3 columns]\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.223037\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mTask 'model_evaluation'\u001b[0m]\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m \u001b[36mtask\u001b[0m=\u001b[35m<Task 'model_evaluation' 0x108384f50 (id: 13)>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.223798\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mCommitting stage              \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 6_evaluation>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.231446\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mDROP VIEW \"6_evaluation\".evaluation\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.236283\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mExecuting sql                 \u001b[0m [\u001b[0m\u001b[1m\u001b[34mDuckDBTableStore\u001b[0m]\u001b[0m\n",
      "    [\u001b[36mquery\u001b[0m]\n",
      "    \u001b[35mCREATE VIEW \"6_evaluation\".evaluation AS\n",
      "    \u001b[35mSELECT * \n",
      "    \u001b[35mFROM \"6_evaluation__odd\".evaluation\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.240581\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mTask finished successfully    \u001b[0m [\u001b[0m\u001b[1m\u001b[34mCommit Stage\u001b[0m]\u001b[0m \u001b[36mstage\u001b[0m=\u001b[35m<Stage: 6_evaluation>\u001b[0m \u001b[36mstate\u001b[0m=\u001b[35m<FinalTaskState.COMPLETED: 1>\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.242086\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mFlow visualization            \u001b[0m [\u001b[0m\u001b[1m\u001b[34mFlow\u001b[0m]\u001b[0m \u001b[36murl\u001b[0m=\u001b[35mhttps://kroki.io/graphviz/svg/eNqtk8tugzAQRfd8BaLbVsIkJVQo3fYjqsoa8ECtOHZkm1ZplX-vMa88VClRYDPjO_hyPIwZrzXsPsO38DcwTdEtStEYi5oSquGbcrlrrCsLKFCsoyMxygNj9wLXRgnO8qCoSyWUXkcPsX-S2L3RSYWAcpMHlZL2RIjD995XI7DOljKwED2GFRdiMMSsqsrUid0Ho7aGLPrIg0NwCZ7QUiDIc_QTeQb4ZID3tvcAL6jVIE2l9BaZP_9EfVmbAX05oI_e9-AvaYVgG41mwp60GXDTcUwE_Own52uhV-N-hNoB32yQjQal2hZc4n8WwLB6Ka5t3DPdKoZi6lovzNAyQkZkb9lOEZdc1lcDk-ly-r0UJKMWjaUG7V3HTil-gWjAciWnsx-rczRgcdaAI_cb4OPw6TVM2svu4rK9OS6mfVz1kcTtjLoka2eti9lQIN6CkDFb-L_TZ4fgDxaGzsQ=\u001b[0m\n",
      "\u001b[2m2024-08-06 09:34:54.447068\u001b[0m [\u001b[32m\u001b[1minfo     \u001b[0m] \u001b[1mStopped IPCServer             \u001b[0m [\u001b[0m\u001b[1m\u001b[34mRunContextServer\u001b[0m]\u001b[0m \u001b[36maddress\u001b[0m=\u001b[35mtcp://127.0.0.1:52807\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"330pt\" height=\"667pt\" viewBox=\"0.00 0.00 329.51 667.00\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 663)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-663 325.51,-663 325.51,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster_1_raw_input</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"141,-574.5 141,-651 297,-651 297,-574.5 141,-574.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-633.7\" font-family=\"Times,serif\" font-size=\"14.00\">1_raw_input</text>\n",
       "</g>\n",
       "<g id=\"clust2\" class=\"cluster\">\n",
       "<title>cluster_2_clean_input</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"173,-490 173,-566.5 265,-566.5 265,-490 173,-490\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-549.2\" font-family=\"Times,serif\" font-size=\"14.00\">2_clean_input</text>\n",
       "</g>\n",
       "<g id=\"clust3\" class=\"cluster\">\n",
       "<title>cluster_3_transformed_data</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"157,-405.5 157,-482 281,-482 281,-405.5 157,-405.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-464.7\" font-family=\"Times,serif\" font-size=\"14.00\">3_transformed_data</text>\n",
       "</g>\n",
       "<g id=\"clust4\" class=\"cluster\">\n",
       "<title>cluster_4_features</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"8,-249 8,-397.5 292,-397.5 292,-249 8,-249\"/>\n",
       "<text text-anchor=\"middle\" x=\"150\" y=\"-380.2\" font-family=\"Times,serif\" font-size=\"14.00\">4_features</text>\n",
       "</g>\n",
       "<g id=\"clust5\" class=\"cluster\">\n",
       "<title>cluster_5_model</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"123,-92.5 123,-241 295,-241 295,-92.5 123,-92.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"209\" y=\"-223.7\" font-family=\"Times,serif\" font-size=\"14.00\">5_model</text>\n",
       "</g>\n",
       "<g id=\"clust6\" class=\"cluster\">\n",
       "<title>cluster_6_evaluation</title>\n",
       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"black\" points=\"125,-8 125,-84.5 295,-84.5 295,-8 125,-8\"/>\n",
       "<text text-anchor=\"middle\" x=\"210\" y=\"-67.2\" font-family=\"Times,serif\" font-size=\"14.00\">6_evaluation</text>\n",
       "</g>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<ellipse fill=\"#e8ffc6\" stroke=\"black\" cx=\"219\" cy=\"-600.5\" rx=\"70.29\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-595.45\" font-family=\"Times,serif\" font-size=\"14.00\">read_input_data</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>2</title>\n",
       "<ellipse fill=\"#e8ffc6\" stroke=\"black\" cx=\"219\" cy=\"-516\" rx=\"30.37\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-510.95\" font-family=\"Times,serif\" font-size=\"14.00\">clean</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0-&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M219,-582C219,-571.5 219,-557.89 219,-545.83\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.5,-545.98 219,-535.98 215.5,-545.98 222.5,-545.98\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>4</title>\n",
       "<ellipse fill=\"#e8ffc6\" stroke=\"black\" cx=\"219\" cy=\"-431.5\" rx=\"47.26\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-426.45\" font-family=\"Times,serif\" font-size=\"14.00\">transform</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>2-&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M219,-497.5C219,-487 219,-473.39 219,-461.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.5,-461.48 219,-451.48 215.5,-461.48 222.5,-461.48\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>6</title>\n",
       "<ellipse fill=\"#e8ffc6\" stroke=\"black\" cx=\"76\" cy=\"-347\" rx=\"60.05\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"76\" y=\"-341.95\" font-family=\"Times,serif\" font-size=\"14.00\">lazy_features</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4-&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M186.98,-417.9C173.69,-412.25 158.3,-405.13 145,-397.5 131.48,-389.74 117.36,-379.95 105.44,-371.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"107.7,-368.45 97.61,-365.22 103.49,-374.04 107.7,-368.45\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>7</title>\n",
       "<ellipse fill=\"#e8ffc6\" stroke=\"black\" cx=\"219\" cy=\"-347\" rx=\"64.66\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"219\" y=\"-341.95\" font-family=\"Times,serif\" font-size=\"14.00\">eager_features</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;7 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>4-&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M219,-413C219,-402.5 219,-388.89 219,-376.83\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.5,-376.98 219,-366.98 215.5,-376.98 222.5,-376.98\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>10</title>\n",
       "<ellipse fill=\"#adef9b\" stroke=\"black\" cx=\"209\" cy=\"-190.5\" rx=\"77.97\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"209\" y=\"-185.45\" font-family=\"Times,serif\" font-size=\"14.00\">train_and_test_set</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;10 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4-&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M260.2,-422.49C273.95,-417.64 287.83,-409.88 296,-397.5 332.33,-342.4 327.26,-307.13 296,-249 287.49,-233.17 272.5,-220.97 257.33,-211.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"259.18,-208.99 248.73,-207.24 255.81,-215.13 259.18,-208.99\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>8</title>\n",
       "<ellipse fill=\"#adef9b\" stroke=\"black\" cx=\"207\" cy=\"-275\" rx=\"76.94\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"207\" y=\"-269.95\" font-family=\"Times,serif\" font-size=\"14.00\">combine_features</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;8 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>6-&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M104.43,-330.81C122.84,-320.97 147.07,-308.02 167.37,-297.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"168.76,-300.4 175.93,-292.6 165.46,-294.23 168.76,-300.4\"/>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;8 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>7-&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M216.03,-328.7C214.77,-321.32 213.26,-312.52 211.84,-304.25\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"215.33,-303.86 210.19,-294.6 208.43,-305.04 215.33,-303.86\"/>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;10 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>8-&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M207.42,-256.5C207.68,-246 208.01,-232.39 208.3,-220.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"211.8,-220.57 208.54,-210.48 204.8,-220.4 211.8,-220.57\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>11</title>\n",
       "<ellipse fill=\"#adef9b\" stroke=\"black\" cx=\"203\" cy=\"-118.5\" rx=\"67.73\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"203\" y=\"-113.45\" font-family=\"Times,serif\" font-size=\"14.00\">model_training</text>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>13</title>\n",
       "<ellipse fill=\"#adef9b\" stroke=\"black\" cx=\"210\" cy=\"-34\" rx=\"77.45\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"210\" y=\"-28.95\" font-family=\"Times,serif\" font-size=\"14.00\">model_evaluation</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;13 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>11-&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M204.45,-100.41C205.35,-89.75 206.54,-75.77 207.58,-63.47\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"211.04,-64.12 208.4,-53.86 204.07,-63.53 211.04,-64.12\"/>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;11 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>10-&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M207.52,-172.2C206.89,-164.91 206.15,-156.23 205.45,-148.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"208.94,-147.78 204.6,-138.11 201.96,-148.37 208.94,-147.78\"/>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;13 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>10-&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M241.93,-173.82C256.41,-165.04 271.93,-152.59 280,-136.5 288.77,-119.02 288.36,-110.18 280,-92.5 273.36,-78.45 261.42,-66.68 249.25,-57.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"251.43,-54.77 241.23,-51.9 247.41,-60.51 251.43,-54.77\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow = get_pipeline()\n",
    "result = flow.run()\n",
    "result.visualize()\n",
    "assert result.successful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "Next: [vectorization06.ipynb](vectorization06.ipynb): many ways to describe data transformations in python"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}