Privacy Engineering workshop with STRM Privacy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "e87cc156-9f28-4331-b197-6f58b3f1b37e",
    "deepnote_cell_type": "text-cell-h1",
    "formattedRanges": [
     {
      "fromCodePoint": 0,
      "marks": {
       "bold": true
      },
      "toCodePoint": 10,
      "type": "marks"
     }
    ],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "# Welcome to the STRM Privacy engineering + 🍕 event! 👋"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "7ece32bc-b679-4cde-a9a8-d0015c02385e",
    "deepnote_cell_type": "text-cell-p",
    "formattedRanges": [
     {
      "fromCodePoint": 95,
      "marks": {
       "italic": true
      },
      "toCodePoint": 105,
      "type": "marks"
     }
    ],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "In this notebook, we'll take you through our workshop. The sections marked as Optional are, you guessed it, optional. They will allow you to explore more of STRM Privacy, or apply our platform to a dataset of your own choice. Feel free to revisit them after completing the remainder of the sections!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "2854851b-0194-49e3-9a89-62f43a98db4d",
    "deepnote_cell_type": "text-cell-p",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "To follow along, make sure you have:\n",
    "\n",
    "* Created a [STRM Privacy account](https://console.strmprivacy.io)\n",
    "* Installed the [STRM CLI](https://github.com/strmprivacy/cli)\n",
    "* Access to a bucket/blob storage in GCP/AWS/Azure (optional)\n",
    "  * If you don't have a ready-to-use bucket, we can provide you with a Google Cloud Storage bucket (can be viewed unauthenticated, and can be written to with a service account that we provide)\n",
    "* Installed [Jupyter](https://jupyter.org/install) to execute or modify some of the code examples yourself (optional)\n",
    "* A belly full of pizza 😉"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "51da5d0f89af46beb33d47e659aaf8cd",
    "deepnote_cell_type": "text-cell-h1",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "## Workshop outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "b85f65c5-e90d-47db-ac69-23afd83f2055",
    "deepnote_cell_type": "text-cell-p",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "During the workshop, you will:\n",
    "\n",
    "1. Create a STRM Pipeline\n",
    "1. Send data to the pipeline\n",
    "   1. Using the CLI to simulate random events\n",
    "   1. Read the published events via the CLI web socket\n",
    "   1. _(Optional) Publish the example webshop data using the (Python) driver_\n",
    "   1. _(Optional) Create your own data contract and use it to publish your own data_\n",
    "1. Create one or more privacy streams for the pipeline\n",
    "1. Create one or more batch exporters\n",
    "1. _(Optional) set up an external table with AWS Athena or Google Cloud BigQuery_\n",
    "1. Explore the opportunities of privacy-safe data\n",
    "1. Try out (some of) the optional sections and/or explore more of our product\n",
    "\n",
    "💡 If at any time you would like more information about a specific concept or feature, feel free to ask us, or have a look at [our docs](https://docs.strmprivacy.io/docs/latest/overview). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Create a STRM Pipeline 🔀\n",
    "\n",
    "When you login for the first time, a new project is generated for you. You can either use that project, or you can create a new one, either [through the console](https://console.strmprivacy.io/projects), or [the CLI](https://docs.strmprivacy.io/docs/latest/reference/cli-reference/strm/create/project/).\n",
    "\n",
    "In the console, navigate to the Pipelines tab of your project to create a pipeline:\n",
    "\n",
    "![create pipeline](images/create-pipeline.png)\n",
    "\n",
    "Alternatively, create it [through the CLI](https://docs.strmprivacy.io/docs/latest/reference/cli-reference/strm/create/stream/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Send data to the pipeline 🚀\n",
    "\n",
    "When sending a data record / event to a pipeline, all sensitive (PII) fields defined in the event's data contract will be encrypted. In this section you will see the effect this has in real-time.\n",
    "\n",
    "⚠️ **Note**: you can skip ahead to section 3 after having tried the CLI's web socket command. Or stick around and see how we generate some fake but real-looking sensitive data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Simulate random events via the CLI\n",
    "\n",
    "Our CLI provides a [simulate command](https://docs.strmprivacy.io/docs/latest/reference/cli-reference/strm/simulate/random-events/) that publishes events containing random data using the [`strmprivacy/example/1.5.0`](https://console.strmprivacy.io/data-contracts/a43fd098-30b4-4910-b6fb-3fe4334dfd64) data contract:\n",
    "![example data contract](images/example-dc.png)\n",
    "\n",
    "This data contract contains one non-sensitive field and three sensitive fields, each for a different *purpose* (e.g. analytics, marketing, ...). For more on data contracts and how to create them, see section 2.4.\n",
    "\n",
    "💡 **Note:** if this is your first time using the CLI, make sure to authenticate with:\n",
    "\n",
    "```bash\n",
    "strm auth login\n",
    "```\n",
    "\n",
    "To start sending events, simply execute the following command:\n",
    "\n",
    "```bash\n",
    "strm simulate random-events <your-pipeline-name-here> --interval 5000\n",
    "```\n",
    "\n",
    "By default, an event is published every second, but as you can see, a different interval can be used instead (5 seconds here)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 See the published events via the CLI\n",
    "\n",
    "To see the events as they get published, execute the [web socket command](https://docs.strmprivacy.io/docs/latest/reference/cli-reference/strm/listen/web-socket/) in a separate shell:\n",
    "\n",
    "```bash\n",
    "strm listen web-socket <your-pipeline-name-here>\n",
    "```\n",
    "\n",
    "This will print the received events as JSON, one line per event. You could use the `jq` tool for a more readable output:\n",
    "\n",
    "```bash\n",
    "strm listen web-socket <your-pipeline-name-here> | jq\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "da097822-a2cd-48c9-99b9-e4a910db2589",
    "deepnote_cell_type": "text-cell-h1",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "## 2.3 (Optional) Let's use some fake webshop data\n",
    "\n",
    "### 2.3.1 Generating data with Faker\n",
    "\n",
    "💡 **Note:** alternatively, you can use the `example_input.csv` file that we already generated. But feel free to generate your own data. If you change the structure of the data, you will also need to create your own data contract!\n",
    "\n",
    "First, let's import the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cell_id": "e16a279eb8d64367b98ea66327e40d97",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 2157,
    "execution_start": 1669891211464,
    "source_hash": "2a5bf08c",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: faker in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (16.4.0)\n",
      "Requirement already satisfied: pandas>=1.4.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (1.5.2)\n",
      "Requirement already satisfied: strmprivacy-driver==3.0.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (3.0.0)\n",
      "Requirement already satisfied: arrow in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (0.14.7)\n",
      "Requirement already satisfied: python-dateutil>=2.4 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from faker->-r requirements.txt (line 1)) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from pandas>=1.4.0->-r requirements.txt (line 2)) (2022.2.1)\n",
      "Requirement already satisfied: numpy>=1.20.3; python_version < \"3.10\" in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from pandas>=1.4.0->-r requirements.txt (line 2)) (1.24.1)\n",
      "Requirement already satisfied: aiohttp>=3.6.2 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (3.8.3)\n",
      "Requirement already satisfied: setuptools>=47.3.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (49.2.1)\n",
      "Requirement already satisfied: avro-python3>=1.10.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.10.2)\n",
      "Requirement already satisfied: responses>=0.10.15 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (0.22.0)\n",
      "Requirement already satisfied: janus>=0.5.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.0.0)\n",
      "Requirement already satisfied: pbr>=5.5.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (5.11.1)\n",
      "Requirement already satisfied: jsonschema>=3.2.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (4.17.3)\n",
      "Requirement already satisfied: avro-json-serializer==1.0.3 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.0.3)\n",
      "Requirement already satisfied: requests>=2.24.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (2.28.1)\n",
      "Requirement already satisfied: pydantic>=1.5.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.10.4)\n",
      "Requirement already satisfied: six>=1.15.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.16.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (22.2.0)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (4.0.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.3.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.8.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.3.3)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (6.0.4)\n",
      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from aiohttp>=3.6.2->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (2.1.0)\n",
      "Requirement already satisfied: urllib3>=1.25.10 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from responses>=0.10.15->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (1.26.10)\n",
      "Requirement already satisfied: types-toml in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from responses>=0.10.15->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (0.10.8.1)\n",
      "Requirement already satisfied: toml in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from responses>=0.10.15->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (0.10.2)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from janus>=0.5.0->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (4.4.0)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from jsonschema>=3.2.0->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (0.19.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests>=2.24.0->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (2022.6.15)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests>=2.24.0->strmprivacy-driver==3.0.0->-r requirements.txt (line 3)) (3.3)\n",
      "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 22.3.1 is available.\n",
      "You should consider upgrading via the '/Users/trietsch/.pyenv/versions/3.9.4/bin/python3.9 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip3 install -r requirements.txt\n",
    "\n",
    "import pandas as pd\n",
    "import datetime, time, arrow\n",
    "import random\n",
    "from random import choice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "9163a18b4d5e4fea882b255491da900a",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Faker is pretty great at making up data! 🥸\n",
    "\n",
    "See for yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cell_id": "c2c11a2a985642b999ad158e087a53e6",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 1318,
    "execution_start": 1669650208895,
    "source_hash": "1ae3d239",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "David Combs 887.574.3306 307 Larry Tunnel Apt. 843\n",
      "Andrea Brown +1-698-231-6782x74652 97729 Tina Meadows\n",
      "Dawn Patel 001-448-827-7445x360 526 Eric Vista Suite 460\n",
      "Jeremy Espinoza +1-608-801-6969 776 Manuel Plaza\n",
      "Martin Lowery 639.161.8726x53053 57339 Jack Shoal Suite 819\n",
      "Tracy Sutton +1-063-418-1869x14377 7196 James Stravenue Apt. 811\n",
      "Nancy Glenn 176-474-2423 7642 Melinda Haven Apt. 178\n",
      "Henry Mitchell DVM 073-058-4162x5618 153 Haley Drives\n",
      "Gloria Rich 123-764-9857x0102 020 Christopher Turnpike Suite 079\n",
      "Richard Foster 634-381-5107 6265 Smith Avenue Apt. 147\n"
     ]
    }
   ],
   "source": [
    "from faker import Faker\n",
    "fake = Faker('en_US')\n",
    "for _ in range(10):\n",
    "    print(fake.name(), fake.phone_number(), fake.street_address())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "e439fcb3586f4b8f9e2854b1f32063a9",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Let's assume we are in e-commerce, a webshop, if you will 🏪"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "1dc160c5-fa24-4d89-9fd9-516c8a1a8113",
    "deepnote_cell_type": "text-cell-p",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "Let's generate an e-commerce inspired dataset, which matches with the [`strmprivacy/PrivacyEngineeringWorkshop/1.0.0`](https://console.strmprivacy.io/data-contracts/324b2163-239f-440d-a5f7-1551ebf97fbc) data contract:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cell_id": "508d54e15d3248498901fd484944e5cb",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 6254,
    "execution_start": 1669650283566,
    "source_hash": "2f4b9e2e",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def fake_data_generation(total_records, total_customers, total_products):\n",
    "    transactions = []\n",
    "    fake = Faker()\n",
    "\n",
    "    customer_ids = random.sample(range(100000, 999999), total_customers)\n",
    "    emails = [fake.free_email() for _ in range(total_customers)]\n",
    "    ages = [random.randrange(99) for _ in range(total_customers)]\n",
    "    sizes = [random.choice(['XS', 'S', 'M', 'L', 'XL']) for _ in range(total_customers)]\n",
    "    names = [fake.name() for _ in range(total_customers)]\n",
    "    phone_numbers = [fake.phone_number() for _ in range(total_customers)]\n",
    "    addresses = [fake.street_address() for _ in range(total_customers)]\n",
    "\n",
    "    personal_data = list(zip(customer_ids, emails, ages, sizes, names, phone_numbers, addresses))\n",
    "    products = random.sample(range(10000, 19999), total_products)\n",
    "\n",
    "    for i in range(total_records):\n",
    "        item_ids = random.choices(products, k=random.randrange(1,7))\n",
    "        customer = random.choice(personal_data)\n",
    "        transactions.append({\n",
    "                \"transactionId\":  random.randrange(100000000, 999999999),\n",
    "                \"userId\": customer[0],\n",
    "                \"email\":  customer[1],\n",
    "                \"age\" : customer[2],\n",
    "                \"tshirtSize\": customer[3],\n",
    "                \"fullName\": customer[4],\n",
    "                \"phoneNumber\": customer[5],\n",
    "                \"address\": customer[6],\n",
    "                \"transactionAmount\": random.randrange(323),\n",
    "                \"items\": item_ids,\n",
    "                \"totalItems\" : len(item_ids),\n",
    "                \"date\" : arrow.utcnow().shift(days=-1*random.randrange(180)).format('YYYY-MM-DD HH:mm:ss'),\n",
    "                \"purposeConsent\" : random.randint(0, 2)\n",
    "                })\n",
    "        \n",
    "    return transactions\n",
    "\n",
    "\n",
    "df = pd.DataFrame(fake_data_generation(10000, 500, 50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "f749581dbda8478e98295ccc20867645",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [
     {
      "fromCodePoint": 12,
      "marks": {
       "strike": true
      },
      "toCodePoint": 17,
      "type": "marks"
     }
    ],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Show me the ~money~ data 🧐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "cell_id": "7eaa94cc5950419cb347439136e2c8a5",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 98,
    "execution_start": 1669650290635,
    "source_hash": "f804c160",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transactionId</th>\n",
       "      <th>userId</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "      <th>tshirtSize</th>\n",
       "      <th>fullName</th>\n",
       "      <th>phoneNumber</th>\n",
       "      <th>address</th>\n",
       "      <th>transactionAmount</th>\n",
       "      <th>items</th>\n",
       "      <th>totalItems</th>\n",
       "      <th>date</th>\n",
       "      <th>purposeConsent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>849938247</td>\n",
       "      <td>477049</td>\n",
       "      <td>mistykirby@hotmail.com</td>\n",
       "      <td>31</td>\n",
       "      <td>M</td>\n",
       "      <td>William Robles</td>\n",
       "      <td>001-652-752-8469x68291</td>\n",
       "      <td>076 Mandy Corners Suite 751</td>\n",
       "      <td>12</td>\n",
       "      <td>[11217, 18090]</td>\n",
       "      <td>2</td>\n",
       "      <td>2022-09-16 16:02:32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>986081859</td>\n",
       "      <td>107296</td>\n",
       "      <td>johnsmith@hotmail.com</td>\n",
       "      <td>73</td>\n",
       "      <td>L</td>\n",
       "      <td>Bruce Patton</td>\n",
       "      <td>(311)182-4594</td>\n",
       "      <td>61283 Paul Gateway Apt. 026</td>\n",
       "      <td>153</td>\n",
       "      <td>[18347, 17344, 19829, 13463, 18720]</td>\n",
       "      <td>5</td>\n",
       "      <td>2022-11-01 16:02:32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>534542356</td>\n",
       "      <td>270039</td>\n",
       "      <td>rhowell@gmail.com</td>\n",
       "      <td>83</td>\n",
       "      <td>M</td>\n",
       "      <td>Crystal Bruce</td>\n",
       "      <td>001-096-429-9884x2572</td>\n",
       "      <td>5359 Hernandez Union Apt. 222</td>\n",
       "      <td>160</td>\n",
       "      <td>[17453, 11685, 17985, 10845, 19366, 17451]</td>\n",
       "      <td>6</td>\n",
       "      <td>2022-09-14 16:02:32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>991419130</td>\n",
       "      <td>975416</td>\n",
       "      <td>allenbetty@yahoo.com</td>\n",
       "      <td>30</td>\n",
       "      <td>S</td>\n",
       "      <td>Julie Martin</td>\n",
       "      <td>887.367.4267</td>\n",
       "      <td>021 Weber Underpass</td>\n",
       "      <td>110</td>\n",
       "      <td>[14137]</td>\n",
       "      <td>1</td>\n",
       "      <td>2022-09-09 16:02:32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>488932444</td>\n",
       "      <td>376203</td>\n",
       "      <td>owenspatricia@gmail.com</td>\n",
       "      <td>26</td>\n",
       "      <td>M</td>\n",
       "      <td>Meagan Willis</td>\n",
       "      <td>960.762.9771x96025</td>\n",
       "      <td>0641 Robert Views</td>\n",
       "      <td>34</td>\n",
       "      <td>[11217, 12285, 11685, 19829]</td>\n",
       "      <td>4</td>\n",
       "      <td>2023-01-09 16:02:32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>772769402</td>\n",
       "      <td>547509</td>\n",
       "      <td>kimberly39@yahoo.com</td>\n",
       "      <td>1</td>\n",
       "      <td>XL</td>\n",
       "      <td>Carrie Powers</td>\n",
       "      <td>001-554-692-2447x4680</td>\n",
       "      <td>909 Cox Underpass Apt. 704</td>\n",
       "      <td>157</td>\n",
       "      <td>[19829]</td>\n",
       "      <td>1</td>\n",
       "      <td>2022-10-01 16:02:32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>145924309</td>\n",
       "      <td>303846</td>\n",
       "      <td>kathy99@yahoo.com</td>\n",
       "      <td>0</td>\n",
       "      <td>XL</td>\n",
       "      <td>Carl Bush</td>\n",
       "      <td>(396)957-8756x53943</td>\n",
       "      <td>518 Gabriel Meadow Apt. 515</td>\n",
       "      <td>90</td>\n",
       "      <td>[10534]</td>\n",
       "      <td>1</td>\n",
       "      <td>2023-01-01 16:02:32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>408577849</td>\n",
       "      <td>751655</td>\n",
       "      <td>gibsonrobert@gmail.com</td>\n",
       "      <td>20</td>\n",
       "      <td>L</td>\n",
       "      <td>Wendy Jackson</td>\n",
       "      <td>(206)463-0028</td>\n",
       "      <td>141 Patrick Avenue Apt. 906</td>\n",
       "      <td>319</td>\n",
       "      <td>[18720, 16880, 19227]</td>\n",
       "      <td>3</td>\n",
       "      <td>2022-08-29 16:02:32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>523334445</td>\n",
       "      <td>280664</td>\n",
       "      <td>nmurray@yahoo.com</td>\n",
       "      <td>23</td>\n",
       "      <td>S</td>\n",
       "      <td>Shannon Oconnor</td>\n",
       "      <td>001-727-065-1972</td>\n",
       "      <td>54307 Amber Motorway</td>\n",
       "      <td>272</td>\n",
       "      <td>[12877, 13476, 17344, 15521, 11283]</td>\n",
       "      <td>5</td>\n",
       "      <td>2022-12-27 16:02:32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>636085093</td>\n",
       "      <td>531000</td>\n",
       "      <td>gutierrezlori@gmail.com</td>\n",
       "      <td>28</td>\n",
       "      <td>L</td>\n",
       "      <td>Darren Hernandez</td>\n",
       "      <td>779-449-1290x2272</td>\n",
       "      <td>85112 Rebekah Trafficway Apt. 629</td>\n",
       "      <td>222</td>\n",
       "      <td>[14137, 19366]</td>\n",
       "      <td>2</td>\n",
       "      <td>2022-10-15 16:02:32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      transactionId  userId                    email  age tshirtSize  \\\n",
       "0         849938247  477049   mistykirby@hotmail.com   31          M   \n",
       "1         986081859  107296    johnsmith@hotmail.com   73          L   \n",
       "2         534542356  270039        rhowell@gmail.com   83          M   \n",
       "3         991419130  975416     allenbetty@yahoo.com   30          S   \n",
       "4         488932444  376203  owenspatricia@gmail.com   26          M   \n",
       "...             ...     ...                      ...  ...        ...   \n",
       "9995      772769402  547509     kimberly39@yahoo.com    1         XL   \n",
       "9996      145924309  303846        kathy99@yahoo.com    0         XL   \n",
       "9997      408577849  751655   gibsonrobert@gmail.com   20          L   \n",
       "9998      523334445  280664        nmurray@yahoo.com   23          S   \n",
       "9999      636085093  531000  gutierrezlori@gmail.com   28          L   \n",
       "\n",
       "              fullName             phoneNumber  \\\n",
       "0       William Robles  001-652-752-8469x68291   \n",
       "1         Bruce Patton           (311)182-4594   \n",
       "2        Crystal Bruce   001-096-429-9884x2572   \n",
       "3         Julie Martin            887.367.4267   \n",
       "4        Meagan Willis      960.762.9771x96025   \n",
       "...                ...                     ...   \n",
       "9995     Carrie Powers   001-554-692-2447x4680   \n",
       "9996         Carl Bush     (396)957-8756x53943   \n",
       "9997     Wendy Jackson           (206)463-0028   \n",
       "9998   Shannon Oconnor        001-727-065-1972   \n",
       "9999  Darren Hernandez       779-449-1290x2272   \n",
       "\n",
       "                                address  transactionAmount  \\\n",
       "0           076 Mandy Corners Suite 751                 12   \n",
       "1           61283 Paul Gateway Apt. 026                153   \n",
       "2         5359 Hernandez Union Apt. 222                160   \n",
       "3                   021 Weber Underpass                110   \n",
       "4                     0641 Robert Views                 34   \n",
       "...                                 ...                ...   \n",
       "9995         909 Cox Underpass Apt. 704                157   \n",
       "9996        518 Gabriel Meadow Apt. 515                 90   \n",
       "9997        141 Patrick Avenue Apt. 906                319   \n",
       "9998               54307 Amber Motorway                272   \n",
       "9999  85112 Rebekah Trafficway Apt. 629                222   \n",
       "\n",
       "                                           items  totalItems  \\\n",
       "0                                 [11217, 18090]           2   \n",
       "1            [18347, 17344, 19829, 13463, 18720]           5   \n",
       "2     [17453, 11685, 17985, 10845, 19366, 17451]           6   \n",
       "3                                        [14137]           1   \n",
       "4                   [11217, 12285, 11685, 19829]           4   \n",
       "...                                          ...         ...   \n",
       "9995                                     [19829]           1   \n",
       "9996                                     [10534]           1   \n",
       "9997                       [18720, 16880, 19227]           3   \n",
       "9998         [12877, 13476, 17344, 15521, 11283]           5   \n",
       "9999                              [14137, 19366]           2   \n",
       "\n",
       "                     date  purposeConsent  \n",
       "0     2022-09-16 16:02:32               0  \n",
       "1     2022-11-01 16:02:32               2  \n",
       "2     2022-09-14 16:02:32               0  \n",
       "3     2022-09-09 16:02:32               2  \n",
       "4     2023-01-09 16:02:32               2  \n",
       "...                   ...             ...  \n",
       "9995  2022-10-01 16:02:32               0  \n",
       "9996  2023-01-01 16:02:32               2  \n",
       "9997  2022-08-29 16:02:32               0  \n",
       "9998  2022-12-27 16:02:32               1  \n",
       "9999  2022-10-15 16:02:32               2  \n",
       "\n",
       "[10000 rows x 13 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "9d1ce25ab9554cf4be21a66305022c9e",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Persist data 🪣\n",
    "\n",
    "For convenience, let's save the data as a csv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "cell_id": "9c1d7d96f43c41ac82c7d9aa4d994022",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 6039,
    "execution_start": 1669650294004,
    "source_hash": "ea50b173",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transactionId,userId,email,age,tshirtSize,fullName,phoneNumber,address,transactionAmount,items,totalItems,date,purposeConsent\r\n",
      "849938247,477049,mistykirby@hotmail.com,31,M,William Robles,001-652-752-8469x68291,076 Mandy Corners Suite 751,12,\"[11217, 18090]\",2,2022-09-16 16:02:32,0\r\n",
      "986081859,107296,johnsmith@hotmail.com,73,L,Bruce Patton,(311)182-4594,61283 Paul Gateway Apt. 026,153,\"[18347, 17344, 19829, 13463, 18720]\",5,2022-11-01 16:02:32,2\r\n",
      "534542356,270039,rhowell@gmail.com,83,M,Crystal Bruce,001-096-429-9884x2572,5359 Hernandez Union Apt. 222,160,\"[17453, 11685, 17985, 10845, 19366, 17451]\",6,2022-09-14 16:02:32,0\r\n",
      "991419130,975416,allenbetty@yahoo.com,30,S,Julie Martin,887.367.4267,021 Weber Underpass,110,[14137],1,2022-09-09 16:02:32,2\r\n",
      "488932444,376203,owenspatricia@gmail.com,26,M,Meagan Willis,960.762.9771x96025,0641 Robert Views,34,\"[11217, 12285, 11685, 19829]\",4,2023-01-09 16:02:32,2\r\n",
      "832564492,531000,gutierrezlori@gmail.com,28,L,Darren Hernandez,779-449-1290x2272,85112 Rebekah Trafficway Apt. 629,258,\"[10407, 12764, 13573, 17586, 19227]\",5,2022-09-26 16:02:32,1\r\n",
      "672899663,399447,katrinalang@gmail.com,23,XS,Cheryl Reyes,547.020.6176x2754,1664 Horton Stravenue,221,\"[16559, 11485, 17451, 15521, 13463]\",5,2022-10-13 16:02:32,0\r\n",
      "103398044,622222,timothyjohnson@gmail.com,13,M,Andrea Green,365-366-6379x329,835 Edwards Rapids Apt. 812,125,\"[19366, 19227, 17453]\",3,2022-11-02 16:02:32,2\r\n",
      "302246621,420431,breannapayne@hotmail.com,46,XS,Dr. Kimberly Sampson,(714)852-3707x18115,60761 Nancy Motorway Apt. 284,297,\"[17586, 16605, 10407, 13476, 16559]\",5,2022-08-03 16:02:32,1\r\n"
     ]
    }
   ],
   "source": [
    "timestamp = time.time()\n",
    "df.to_csv(f'example_input.csv', index=False)  \n",
    "\n",
    "!head example_input.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "37b5e478a0b84a5a935f476f3cfdece3",
    "deepnote_cell_type": "text-cell-h1",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "### 2.3.2 Streaming the data using the Python driver ⚡️\n",
    "\n",
    "The next step is to use STRM to its full potential: with event-driven, real-time data.\n",
    "\n",
    "For the purpose of this workshop, we'll simulate a real-time anonymisation pipeline by replaying our generated data as if they were produced in real time. You will use the Python driver to send the events one-by-one into the STRM Privacy \"event gateway\", similar to how the CLI's `simulate` command does.\n",
    "\n",
    "In sections 3 and later, we will build upon this pipeline by adding privacy streams, batch exporters, and more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "358caa86a48841dd9181535664bd32cd",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "#### Installing the generated schema code\n",
    "\n",
    "Execute the following commands to download a zip of generated code for our specific data contract and have it made available through python's pip package installer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!strm get schema-code strmprivacy/PrivacyEngineeringWorkshop/1.0.0 --language python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  python-avro-PrivacyEngineeringWorkshop-1.0.0.zip\n",
      "   creating: python-avro-PrivacyEngineeringWorkshop-1.0.0/\n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/README.md  \n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/requirements-dev.txt  \n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/Makefile  \n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/setup.py  \n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/requirements.txt  \n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/build.py  \n",
      "   creating: python-avro-PrivacyEngineeringWorkshop-1.0.0/schema/\n",
      "  inflating: python-avro-PrivacyEngineeringWorkshop-1.0.0/schema/schema.avsc  \n",
      "rm -fr build/\n",
      "rm -fr dist/\n",
      "rm -fr .eggs/\n",
      "find . -name '*.egg-info' -exec rm -fr {} +\n",
      "find . -name '*.egg' -exec rm -f {} +\n",
      "find . -name '*.pyc' -exec rm -f {} +\n",
      "find . -name '*.pyo' -exec rm -f {} +\n",
      "find . -name '*~' -exec rm -f {} +\n",
      "find . -name '__pycache__' -exec rm -fr {} +\n",
      "python3 -m pip install -r requirements.txt\n",
      "Requirement already satisfied: avro-gen==0.3.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (0.3.0)\n",
      "Requirement already satisfied: tzlocal==2.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (2.1)\n",
      "Requirement already satisfied: frozendict in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->-r requirements.txt (line 1)) (2.3.4)\n",
      "Requirement already satisfied: avro-python3>=1.8.0; python_version > \"3.0\" in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->-r requirements.txt (line 1)) (1.10.2)\n",
      "Requirement already satisfied: pytz in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->-r requirements.txt (line 1)) (2022.2.1)\n",
      "Requirement already satisfied: six in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->-r requirements.txt (line 1)) (1.16.0)\n",
      "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 22.3.1 is available.\n",
      "You should consider upgrading via the '/Users/trietsch/.pyenv/versions/3.9.4/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n",
      "python3 -m pip install -r requirements-dev.txt\n",
      "Requirement already satisfied: twine==1.14.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements-dev.txt (line 1)) (1.14.0)\n",
      "Requirement already satisfied: wheel in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from -r requirements-dev.txt (line 2)) (0.37.1)\n",
      "Requirement already satisfied: setuptools>=0.7.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (49.2.1)\n",
      "Requirement already satisfied: requests-toolbelt!=0.9.0,>=0.8.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (0.9.1)\n",
      "Requirement already satisfied: readme-renderer>=21.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (37.2)\n",
      "Requirement already satisfied: tqdm>=4.14 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (4.64.0)\n",
      "Requirement already satisfied: pkginfo>=1.4.2 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (1.8.3)\n",
      "Requirement already satisfied: requests!=2.15,!=2.16,>=2.5.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from twine==1.14.0->-r requirements-dev.txt (line 1)) (2.28.1)\n",
      "Requirement already satisfied: bleach>=2.1.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from readme-renderer>=21.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (5.0.1)\n",
      "Requirement already satisfied: Pygments>=2.5.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from readme-renderer>=21.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (2.12.0)\n",
      "Requirement already satisfied: docutils>=0.13.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from readme-renderer>=21.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (0.19)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests!=2.15,!=2.16,>=2.5.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (2022.6.15)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests!=2.15,!=2.16,>=2.5.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (2.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests!=2.15,!=2.16,>=2.5.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from requests!=2.15,!=2.16,>=2.5.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (1.26.10)\n",
      "Requirement already satisfied: webencodings in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (0.5.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine==1.14.0->-r requirements-dev.txt (line 1)) (1.16.0)\n",
      "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 22.3.1 is available.\n",
      "You should consider upgrading via the '/Users/trietsch/.pyenv/versions/3.9.4/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n",
      "python3 build.py clean\n",
      "python3 build.py build\n",
      "python3 setup.py sdist\n",
      "running sdist\n",
      "running egg_info\n",
      "creating strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "writing strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/PKG-INFO\n",
      "writing dependency_links to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/dependency_links.txt\n",
      "writing requirements to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/requires.txt\n",
      "writing top-level names to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/top_level.txt\n",
      "writing manifest file 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt'\n",
      "reading manifest file 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt'\n",
      "reading manifest template 'MANIFEST.in'\n",
      "writing manifest file 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt'\n",
      "running check\n",
      "warning: check: missing required meta-data: url\n",
      "\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A/v\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "creating strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying files to strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0...\n",
      "copying MANIFEST.in -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying Makefile -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying README.md -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying build.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying requirements-dev.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying requirements.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying setup.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying top_package -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "copying strmprivacy_A_A_v/__init__.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v\n",
      "copying strmprivacy_A_A_v/schema.avsc -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v\n",
      "copying strmprivacy_A_A_v/schema_classes.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v\n",
      "copying strmprivacy_A_A_v/A/__init__.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A\n",
      "copying strmprivacy_A_A_v/A/A/__init__.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A\n",
      "copying strmprivacy_A_A_v/A/A/v/__init__.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A/v\n",
      "copying strmprivacy_A_A_v/A/A/v/strmmeta/__init__.py -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/PKG-INFO -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/dependency_links.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/not-zip-safe -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/requires.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/top_level.txt -> strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info\n",
      "Writing strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0/setup.cfg\n",
      "creating dist\n",
      "Creating tar archive\n",
      "removing 'strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0' (and everything under it)\n",
      "python3 setup.py bdist_wheel\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running bdist_wheel\n",
      "running build\n",
      "running build_py\n",
      "creating build\n",
      "creating build/lib\n",
      "creating build/lib/strmprivacy_A_A_v\n",
      "copying strmprivacy_A_A_v/__init__.py -> build/lib/strmprivacy_A_A_v\n",
      "copying strmprivacy_A_A_v/schema_classes.py -> build/lib/strmprivacy_A_A_v\n",
      "creating build/lib/strmprivacy_A_A_v/A\n",
      "copying strmprivacy_A_A_v/A/__init__.py -> build/lib/strmprivacy_A_A_v/A\n",
      "creating build/lib/strmprivacy_A_A_v/A/A\n",
      "copying strmprivacy_A_A_v/A/A/__init__.py -> build/lib/strmprivacy_A_A_v/A/A\n",
      "creating build/lib/strmprivacy_A_A_v/A/A/v\n",
      "copying strmprivacy_A_A_v/A/A/v/__init__.py -> build/lib/strmprivacy_A_A_v/A/A/v\n",
      "creating build/lib/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "copying strmprivacy_A_A_v/A/A/v/strmmeta/__init__.py -> build/lib/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "running egg_info\n",
      "writing strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/PKG-INFO\n",
      "writing dependency_links to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/dependency_links.txt\n",
      "writing requirements to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/requires.txt\n",
      "writing top-level names to strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/top_level.txt\n",
      "reading manifest file 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt'\n",
      "reading manifest template 'MANIFEST.in'\n",
      "writing manifest file 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info/SOURCES.txt'\n",
      "copying strmprivacy_A_A_v/schema.avsc -> build/lib/strmprivacy_A_A_v\n",
      "installing to build/bdist.macosx-12.3-arm64/wheel\n",
      "running install\n",
      "running install_lib\n",
      "creating build/bdist.macosx-12.3-arm64\n",
      "creating build/bdist.macosx-12.3-arm64/wheel\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v\n",
      "copying build/lib/strmprivacy_A_A_v/__init__.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A\n",
      "copying build/lib/strmprivacy_A_A_v/A/__init__.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A\n",
      "copying build/lib/strmprivacy_A_A_v/A/A/__init__.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A/v\n",
      "copying build/lib/strmprivacy_A_A_v/A/A/v/__init__.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A/v\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "copying build/lib/strmprivacy_A_A_v/A/A/v/strmmeta/__init__.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v/A/A/v/strmmeta\n",
      "copying build/lib/strmprivacy_A_A_v/schema.avsc -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v\n",
      "copying build/lib/strmprivacy_A_A_v/schema_classes.py -> build/bdist.macosx-12.3-arm64/wheel/strmprivacy_A_A_v\n",
      "running install_egg_info\n",
      "Copying strmprivacy_schemas_PrivacyEngineeringWorkshop_avro.egg-info to build/bdist.macosx-12.3-arm64/wheel/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0-py3.9.egg-info\n",
      "running install_scripts\n",
      "creating build/bdist.macosx-12.3-arm64/wheel/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0.dist-info/WHEEL\n",
      "creating 'dist/strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0-py3-none-any.whl' and adding 'build/bdist.macosx-12.3-arm64/wheel' to it\n",
      "adding 'strmprivacy_A_A_v/__init__.py'\n",
      "adding 'strmprivacy_A_A_v/schema.avsc'\n",
      "adding 'strmprivacy_A_A_v/schema_classes.py'\n",
      "adding 'strmprivacy_A_A_v/A/__init__.py'\n",
      "adding 'strmprivacy_A_A_v/A/A/__init__.py'\n",
      "adding 'strmprivacy_A_A_v/A/A/v/__init__.py'\n",
      "adding 'strmprivacy_A_A_v/A/A/v/strmmeta/__init__.py'\n",
      "adding 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0.dist-info/METADATA'\n",
      "adding 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0.dist-info/WHEEL'\n",
      "adding 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0.dist-info/top_level.txt'\n",
      "adding 'strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0.dist-info/RECORD'\n",
      "removing build/bdist.macosx-12.3-arm64/wheel\n",
      "ls -l dist\n",
      "total 32\n",
      "-rw-r--r--  1 trietsch  staff  6337 Jan 17 16:56 strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0.tar.gz\n",
      "-rw-r--r--  1 trietsch  staff  5702 Jan 17 16:56 strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0-py3-none-any.whl\n",
      "python3 -m pip install .\n",
      "Processing /Users/trietsch/workspace/strmprivacy/gitlab/workshops/privacy-engineering-workshop/python-avro-PrivacyEngineeringWorkshop-1.0.0\n",
      "Requirement already satisfied: avro-gen==0.3.0 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (0.3.0)\n",
      "Requirement already satisfied: tzlocal==2.1 in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (2.1)\n",
      "Requirement already satisfied: six in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (1.16.0)\n",
      "Requirement already satisfied: pytz in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (2022.2.1)\n",
      "Requirement already satisfied: frozendict in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (2.3.4)\n",
      "Requirement already satisfied: avro-python3>=1.8.0; python_version > \"3.0\" in /Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages (from avro-gen==0.3.0->strmprivacy-schemas-PrivacyEngineeringWorkshop-avro==1.0.0) (1.10.2)\n",
      "Building wheels for collected packages: strmprivacy-schemas-PrivacyEngineeringWorkshop-avro\n",
      "  Building wheel for strmprivacy-schemas-PrivacyEngineeringWorkshop-avro (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for strmprivacy-schemas-PrivacyEngineeringWorkshop-avro: filename=strmprivacy_schemas_PrivacyEngineeringWorkshop_avro-1.0.0-py3-none-any.whl size=5702 sha256=2e466a7dc182a5ae03a71581d79e857b7e7e0a997c36f08c67a827aee2d04ee2\n",
      "  Stored in directory: /Users/trietsch/Library/Caches/pip/wheels/e1/24/f4/7ac6bc1204a312bdfb43233359c6c53cc53185f3046e1eea22\n",
      "Successfully built strmprivacy-schemas-PrivacyEngineeringWorkshop-avro\n",
      "Installing collected packages: strmprivacy-schemas-PrivacyEngineeringWorkshop-avro\n",
      "Successfully installed strmprivacy-schemas-PrivacyEngineeringWorkshop-avro-1.0.0\n",
      "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 22.3.1 is available.\n",
      "You should consider upgrading via the '/Users/trietsch/.pyenv/versions/3.9.4/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!unzip python-avro-PrivacyEngineeringWorkshop-1.0.0.zip\n",
    "!cd python-avro-PrivacyEngineeringWorkshop-1.0.0 && make install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "503c06a90a47429391fe94e913b9687c",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Creating and sending the events\n",
    "\n",
    "Now, you can use the generated schema code to construct each event and use the python driver (SyncSender) to send them to your pipeline.\n",
    "\n",
    "⚠️ **Note:** you will need to insert your Pipeline's credentials in the code snippet below, otherwise the SyncSender will fail on authentication. You can retrieve your pipeline's options by requesting json output on the CLI:\n",
    "\n",
    "```bash\n",
    "strm get stream <your-pipeline-name-here> -o json\n",
    "```\n",
    "\n",
    "Or alternatively, you can click on *Pipeline options* => *View Credentials* in the console:\n",
    "![pipeline credentials](images/pipeline-credentials.png)\n",
    "\n",
    "💡 The code below only sends the first few records. Remove the `.head()` call to send all records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cell_id": "a1dddc1396904c3ea6f2a1a4124036ec",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DEBUG:strmprivacy.driver.client.auth:_initialize_auth_provider\n",
      "DEBUG:strmprivacy.driver.client.auth:Initializing a new Auth Provider for SenderService\n",
      "DEBUG:strmprivacy.driver.client.auth:authenticate\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread Thread-6:\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/threading.py\", line 954, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/syncsender.py\", line 35, in run\n",
      "    asyncio.run(self.async_start(*self._props), debug=self.async_debug)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/asyncio/runners.py\", line 44, in run\n",
      "    return loop.run_until_complete(main)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/asyncio/base_events.py\", line 642, in run_until_complete\n",
      "    return future.result()\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/syncsender.py\", line 39, in async_start\n",
      "    client = StrmPrivacyClient(client_id, client_secret, self._config)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/client.py\", line 19, in __init__\n",
      "    self._sender_service = SenderService(client_id, client_secret, config)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/sender.py\", line 12, in __init__\n",
      "    self.auth_service = AuthService(\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/auth.py\", line 31, in __init__\n",
      "    self.timer_task = TimerTask(self._initialize_auth_provider, config.auth_refresh_interval)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/util/__init__.py\", line 23, in __init__\n",
      "    self.method()\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/auth.py\", line 80, in _initialize_auth_provider\n",
      "    self._authenticate(self._client_id, self._client_secret)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/auth.py\", line 46, in _authenticate\n",
      "    self._do_post(self._config.auth_auth_uri, payload)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/auth.py\", line 69, in _do_post\n",
      "    self.auth_provider = AuthProvider.from_json(response.json())\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/client/auth.py\", line 97, in from_json\n",
      "    return JsonSerializable._from_json(json_dict, AuthProvider)\n",
      "  File \"/Users/trietsch/.pyenv/versions/3.9.4/lib/python3.9/site-packages/strmprivacy/driver/domain/base.py\", line 8, in _from_json\n",
      "    return clazz(**JsonSerializable._to_snake_case(json_dict))\n",
      "TypeError: __init__() missing 3 required positional arguments: 'access_token', 'refresh_token', and 'expires_in'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished sending events\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "import time\n",
    "import pandas as pd\n",
    "\n",
    "from strmprivacy.driver.client.syncsender import SyncSender\n",
    "from strmprivacy.driver.serializer import SerializationType\n",
    "# Note: if you want to use your own data contract, adapt the schema code installation commands accordingly, \n",
    "# and replace \"PrivacyEngineeringWorkshop\" by your contract's schema name.\n",
    "from strmprivacy_A_A_v.A.A.v import PrivacyEngineeringWorkshop\n",
    "\n",
    "def create_avro_event(row):\n",
    "    # Here we convert each row of generated data into the event objects matching our schema.\n",
    "    # If you use your own contract, you will need to make the appropriate changes below.\n",
    "    event = PrivacyEngineeringWorkshop()\n",
    "    \n",
    "    event.strmMeta.eventContractRef = \"strmprivacy/PrivacyEngineeringWorkshop/1.0.0\"\n",
    "    event.strmMeta.consentLevels = [int(row['purposeConsent'])]\n",
    "    \n",
    "    event.transactionId = str(row['transactionId'])\n",
    "    event.userId = str(row['userId'])\n",
    "    event.email = str(row['email'])\n",
    "    event.age = int(row['age'])\n",
    "    event.tshirtSize = str(row['tshirtSize'])\n",
    "    event.fullName = str(row['fullName'])\n",
    "    event.phoneNumber = str(row['phoneNumber'])\n",
    "    event.address = str(row['address'])\n",
    "    event.transactionAmount = int(row['transactionAmount'])\n",
    "    event.items = row['items']\n",
    "    event.date = str(row['date'])\n",
    "    event.totalItems = int(row['totalItems'])\n",
    "    event.purposeConsent = int(row['purposeConsent'])\n",
    "\n",
    "    return event\n",
    "\n",
    "def main():\n",
    "    logging.basicConfig(stream=sys.stdout)\n",
    "    logger = logging.getLogger(__name__)\n",
    "    \n",
    "    # To send all records, remove the .head() call\n",
    "    records = pd.read_csv(\"example_input.csv\").head()\n",
    "\n",
    "    # Use your Pipeline's credentials here.\n",
    "    sender = SyncSender(\"<your_client_id>\", \"<your_client_secret>\")\n",
    "    sender.start()\n",
    "\n",
    "    sender.wait_ready()\n",
    "\n",
    "    for row in records.iterrows():\n",
    "        event = create_avro_event(row[1])\n",
    "        response = sender.send_event(event)\n",
    "\n",
    "        if response != None:\n",
    "            logger.error(response)\n",
    "#         print(event)\n",
    "        time.sleep(0.002)\n",
    "    \n",
    "    time.sleep(2)\n",
    "    print(\"Finished sending events\")\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "c3ba3190a91543bfb5271b16fd8843cd",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "#### Check your data 🧐\n",
    "\n",
    "You can again use the CLI's `listen web-socket` command to verify that the events are being sent as expected, or your batch export(s) if you've already set those up.\n",
    "\n",
    "Congratulations on sending your first \"real-time\" events with STRM Privacy! If you want to try it out with your own (fake) data, see the next section!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "cce1b9063d894b6daa704c4c138c6485",
    "deepnote_cell_type": "text-cell-h1",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "## 2.4 (Optional) Create your own data contract 📑\n",
    "\n",
    "To try STRM out with your own data, you will need to create a data contract that matches the shape (schema) of your data, and specify the privacy implications. STRM uses Simple Schema for creating data contracts, which is written in YAML.\n",
    "\n",
    "For example, this is the (slightly simplified) schema for the `PrivacyEngineeringWorkshop` contract used before:\n",
    "\n",
    "```yaml\n",
    "name: PrivacyEngineeringWorkshop\n",
    "nodes:\n",
    "  - type: STRING\n",
    "    name: transactionId\n",
    "    repeated: false\n",
    "    required: true\n",
    "  - type: STRING\n",
    "    name: userId\n",
    "  - type: STRING\n",
    "    name: email\n",
    "  - type: INTEGER\n",
    "    name: age\n",
    "  - type: STRING\n",
    "    name: tshirtSize\n",
    "  - type: INTEGER\n",
    "    name: transactionAmount\n",
    "  - type: STRING\n",
    "    name: items\n",
    "  - type: INTEGER\n",
    "    name: totalItems\n",
    "  - type: STRING\n",
    "    name: date\n",
    "  - type: INTEGER\n",
    "    name: purposeConsent\n",
    "  - type: STRING\n",
    "    name: fullName\n",
    "  - type: STRING\n",
    "    name: phoneNumber\n",
    "  - type: STRING\n",
    "    name: address\n",
    "```\n",
    "\n",
    "Simple Schema supports repeated and nested data structures. For a full reference of the format, see [our docs](https://docs.strmprivacy.io/docs/latest/quickstart/data-contracts/simple-schema/#simple-schema-reference).\n",
    "\n",
    "To create your own contract, go to the *Data Contracts* tab of your project and click *Create data contract*. After defining the schema, click the *Generate fields* button to proceed with the next step:\n",
    "\n",
    "![create contract](images/create-contract.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "11b90dd4-f4e7-488d-a1fa-520b6a5c1ede",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "### Classify the privacy implications ⚖️\n",
    "\n",
    "With the structure of your data defined, it is time to classify the privacy implications. This basically consists of answering two questions:\n",
    "\n",
    "1. Which fields in my data are sensitive?\n",
    "1. For which purposes can we use this data without additional privacy transformations?\n",
    "\n",
    "First, specify the *Key field*. This must be a required field of type `STRING` and determines which events share their encryption keys. All events with the same value for the key field will contain the same encrypted string for any identical values. (This allows for some analysis on encrypted data: keys rotate after 24 hours, so within that window, some patterns and trends are preserved)\n",
    "\n",
    "![classify fields](images/classify-fields-2.png)\n",
    "\n",
    "Under *PII fields*, specify for each sensitive field to which purpose it belongs. For example, 1 could denote analytics, and 2 marketing. We are currently working on a more descriptive way that uses a purpose mapping defined for your organization. Note that only (repeated) `STRING` fields can be marked as PII.\n",
    "\n",
    "You can also include field validations. Currently, we support regex patterns as validations. These are simple (but powerful) validations that will ensure only events with valid data will be accepted and further processed. In the example above, the `tshirtSize` must be one of the allowed options. Sending an event that does not match the pattern, will be rejected with HTTP status code 400.\n",
    "\n",
    "In the example above, `userId` is linked to purpose 1 and `email` to purpose 2. Both fields will be encrypted in the pipeline and only (potentially) decrypted in *privacy streams* based on this pipeline. More on privacy streams in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send your data using the (Python) driver 🚀\n",
    "\n",
    "Once you have created the contract and _**activated**_ it (through the three dots menu on the data contracts tab), you can send your own events by adapting the steps from section 2.3.2. Let us know if you need any help!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "60a62eff-aaa0-437f-b572-612535a590bd",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "# 3 Create one or more privacy streams\n",
    "\n",
    "Now that you know how to send data to a pipeline, and how to peek at the encrypted stream using the CLI's `web-socket`, it's time to create a couple *privacy streams*. A privacy stream essentially decrypts (a part of) the encrypted data, but *only* if the data subject (the owner of the data, e.g. a user/customer) has consented.\n",
    "\n",
    "You can create a privacy stream for your pipeline through the *Add privacy stream* button on the pipelines tab:\n",
    "\n",
    "![create privacy stream](images/create-privacy-stream.png)\n",
    "\n",
    "Alternatively, you can create one [through the CLI](https://docs.strmprivacy.io/docs/latest/reference/cli-reference/strm/create/stream/).\n",
    "\n",
    "In this example, the purpose type is *Cumulative* with a highest purpose of 1. This means that fields with a purpose up to and including 1 will be decrypted, but again, only when the data subject has consented. By choosing *Granular*, you can specify the exact purposes that should be decrypted. I.e. a cumulative configuration with purpose 2 would decrypt fields with purpose level 1 and 2, while a granular configuration with purposes 1 and 3 would leave any fields with a different purpose level encrypted.\n",
    "\n",
    "💡 In the example, field masking is applied to the `userId` field. Field masking will hash the decrypted value. This can be useful when the original value should remain unknown, but deterministic processing of the data is allowed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've mentioned consent a few times now, but how does that work? Let's look at this example event from the CLI's random events simulator:\n",
    "```json\n",
    "{\n",
    "  \"strmMeta\": {\n",
    "    \"eventContractRef\": \"strmprivacy/example/1.3.0\",\n",
    "    \"nonce\": 485472365,\n",
    "    \"timestamp\": 1673610238506,\n",
    "    \"keyLink\": \"75e08aeb-131d-4f84-8070-108f8806fbe2\",\n",
    "    \"consentLevels\": [\n",
    "      0,\n",
    "      1,\n",
    "      2\n",
    "    ]\n",
    "  },\n",
    "  \"uniqueIdentifier\": \"AXpgWnZ6thGw6v8pED36Z/5c536WoikG3BWJ7S5I\",\n",
    "  \"consistentValue\": \"AXpgWnb4O+NsrSy8/9CmwzrXHKtrwGILHUSEfnAhadA=\",\n",
    "  \"someSensitiveValue\": \"AXpgWnY8oDprlwtmdAO1hAbp417Ea2OLp1HaRxkQylj5\",\n",
    "  \"notSensitiveValue\": \"not-sensitive-41\"\n",
    "}\n",
    "```\n",
    "\n",
    "In the `strmMeta` data of the event, the `consentLevels` field contains all levels the user consented to. This has to be specified by the sender. In this example, the user has consented to purposes 0, 1 and 2.\n",
    "\n",
    "Now, looking at the exact same message, but processed by privacy stream with a purpose type of cumulative 1, we see:\n",
    "```json\n",
    "{\n",
    "  \"strmMeta\": {\n",
    "    \"eventContractRef\": \"strmprivacy/example/1.3.0\",\n",
    "    \"nonce\": 485472365,\n",
    "    \"timestamp\": 1673610238506,\n",
    "    \"keyLink\": \"75e08aeb-131d-4f84-8070-108f8806fbe2\",\n",
    "    \"consentLevels\": [\n",
    "      0,\n",
    "      1,\n",
    "      2\n",
    "    ]\n",
    "  },\n",
    "  \"uniqueIdentifier\": \"unique-37\",\n",
    "  \"consistentValue\": \"AXpgWnb4O+NsrSy8/9CmwzrXHKtrwGILHUSEfnAhadA=\",\n",
    "  \"someSensitiveValue\": \"AXpgWnY8oDprlwtmdAO1hAbp417Ea2OLp1HaRxkQylj5\",\n",
    "  \"notSensitiveValue\": \"not-sensitive-41\"\n",
    "}\n",
    "```\n",
    "\n",
    "Everything is identical, except the value of the `uniqueIdentifier` field, which has purpose level 1, and got decrypted because the user has consented.\n",
    "\n",
    "And after processing by a privacy stream of type cumulative 2, the `consistentValue` field is also decrypted:\n",
    "```json\n",
    "{\n",
    "  \"strmMeta\": {\n",
    "    \"eventContractRef\": \"strmprivacy/example/1.3.0\",\n",
    "    \"nonce\": 485472365,\n",
    "    \"timestamp\": 1673610238506,\n",
    "    \"keyLink\": \"75e08aeb-131d-4f84-8070-108f8806fbe2\",\n",
    "    \"consentLevels\": [\n",
    "      0,\n",
    "      1,\n",
    "      2\n",
    "    ]\n",
    "  },\n",
    "  \"uniqueIdentifier\": \"unique-37\",\n",
    "  \"consistentValue\": \"session-623\",\n",
    "  \"someSensitiveValue\": \"AXpgWnY8oDprlwtmdAO1hAbp417Ea2OLp1HaRxkQylj5\",\n",
    "  \"notSensitiveValue\": \"not-sensitive-41\"\n",
    "}\n",
    "```\n",
    "\n",
    "💡 **Note:** events of which the data subject didn't consent to the level required by a privacy stream, will be absent from the privacy stream's output! In other words, privacy streams will likely process fewer events than originally delivered to the pipeline.\n",
    "\n",
    "### Create your privacy streams\n",
    "\n",
    "Now that you know how privacy streams are created, go ahead and create one or a few with configurations of your own liking!\n",
    "\n",
    "💡 Use the CLI's random events simulator and one or more web socket commands to see this behaviour for yourself.\n",
    "\n",
    "💡 Are you running into any issues? Try the `strm monitor stream <your-stream-name>` and `strm logs stream <your-stream-name>` commands to check for unexpected errors. You can also add `--follow` to the commands to get a continuous stream of logs and state updates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "aab082a1c7774d8a9e4afd1dccb75b94",
    "deepnote_cell_type": "text-cell-h2",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "# 4 Create one or more batch exporters 🚰\n",
    "\n",
    "Although the CLI's web socket command is useful to debug or experiment, it is not meant for production workloads. Instead, privacy streams can be exported to common file/blob stores through the use of *batch exporters*. If you have access to an AWS S3 bucket, GCS bucket, or Azure Blob Storage Container, you can use it in this section. If you don't have one readily available, we will provide you with a [Google Cloud Storage bucket](https://console.cloud.google.com/storage/browser/privacy-engineering-workshop) (the writer credentials can be found in the bucket).\n",
    "\n",
    "⚠️ **Note:** to avoid data from different participants (or privacy streams) to mix when using our provided bucket, make sure to set a unique *Path prefix* as explained further below.\n",
    "\n",
    "To create a data connector, go to the *Data connectors* tab of your project, or follow the respective [CLI quickstart](https://docs.strmprivacy.io/docs/latest/quickstart/data-connectors/).\n",
    "\n",
    "💡 A data connector can be used both for reading or writing data, and reused for different purposes.\n",
    "\n",
    "With your data connector ready for use, go to the *Exporters* tab to create a batch exporter. Specify the pipeline or privacy stream you want to export and the data connector to use. You can also give it a custom name and modify the default export interval. Make sure to set a unique path prefix! If no prefix is set, data will be exported to the \"root\" of your data connector. If you select *include existing events*, recent events will be immediately exported. An example:\n",
    "![create batch exporter](images/create-batch-exporter.png)\n",
    "\n",
    "Shortly after creating your first batch exporter(s), you should see `.jsonl` files getting added to your data connector's target storage. Each line in these files consists of a STRM event in JSON format, like you saw with the web socket before.\n",
    "\n",
    "💡 Are you running into any issues, or do you want to see what your batch exporter is doing? Try the `strm logs batch-exporter <your-batch-exporter-name> --follow` to follow its progress. Or the `strm monitor batch-exporter <your-batch-exporter-name>` to check its latest state.\n",
    "\n",
    "With the JSON lines format being well-supported by many (cloud) data tools, you can build upon batch exporters in various interesting ways. See the next section for an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 (Optional) set up an external table with AWS Athena or Google Cloud BigQuery ☁️\n",
    "\n",
    "## 5.1 Creating an external table with AWS Athena\n",
    "\n",
    "If you used a batch exporter to export data to an S3 bucket, you could use AWS Athena to create an external table on the newline-delimited json files. This will allow you to query the data using SQL. We were a bit rusty on how the `CREATE TABLE` statement should look like, so we thought \"why not just ask Chat GPT?\". Well, here's what we asked for the `PrivacyEngineeringWorkshop` data contract:\n",
    "\n",
    "```\n",
    "write an AWS Athena create table statement for querying this JSON data:\n",
    "{\n",
    "  \"strmMeta\": {\n",
    "    \"eventContractRef\": \"strmprivacy/PrivacyEngineeringWorkshop/1.0.0\",\n",
    "    \"nonce\": 942989520,\n",
    "    \"timestamp\": 1673808342405,\n",
    "    \"keyLink\": \"c4b90723-08ff-4e7b-a21b-0cd6b6a9d400\",\n",
    "    \"billingId\": null,\n",
    "    \"consentLevels\": [\n",
    "      2\n",
    "    ]\n",
    "  },\n",
    "  \"transactionId\": \"820683509\",\n",
    "  \"userId\": \"AWcF7Cl5mCYJ+QCYRNjTclRhC1KCMJd1PQ1b\",\n",
    "  \"email\": \"AWcF7Cl5W2ONhCpfBGK4eAaKs+2uUmEaUynVl/iTZyzTSvar5ZdG0b5K3Sc=\",\n",
    "  \"age\": 26,\n",
    "  \"tshirtSize\": \"L\",\n",
    "  \"transactionAmount\": 62,\n",
    "  \"items\": \"[15081, 14707, 13197, 16778, 15546]\",\n",
    "  \"totalItems\": 5,\n",
    "  \"date\": \"2022-08-13 18:43:59\",\n",
    "  \"purposeConsent\": 2,\n",
    "  \"fullName\": \"AWcF7CldDr3B3R6y341aMh4h8Q/ke8o7a8JPRTbomg==\",\n",
    "  \"phoneNumber\": \"AWcF7Ck85wX/V5wrJbCDHxoZOmHdgosHzbUZNpWuiU5LAQggXG+VRg==\",\n",
    "  \"address\": \"AWcF7Cm95K6wXGAosawf0sV2w6ZBOfZHjxHqjuYkCc4D1rISyaHb32f3tRUQ\"\n",
    "}\n",
    "```\n",
    "\n",
    "To which it replied:\n",
    "\n",
    "```\n",
    "CREATE EXTERNAL TABLE IF NOT EXISTS events (\n",
    "`strmMeta` struct<eventContractRef:string,nonce:bigint,timestamp:bigint,keyLink:string,consentLevels:array<int>>,\n",
    "`transactionId` string,\n",
    "`userId` string,\n",
    "`email` string,\n",
    "`age` int,\n",
    "`tshirtSize` string,\n",
    "`transactionAmount` int,\n",
    "`items` array<int>,\n",
    "`totalItems` int,\n",
    "`date` timestamp,\n",
    "`purposeConsent` int,\n",
    "`fullName` string,\n",
    "`phoneNumber` string,\n",
    "`address` string\n",
    ")\n",
    "ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'\n",
    "LOCATION 's3://<your-bucket-name>/path/to/data/';\n",
    "```\n",
    "\n",
    "Pretty convenient, don't you think! 🔥 However, trying the same for Google Cloud BigQuery resulted in an invalid query, at least for the current version of it. So don't get your hopes up just yet.. Or maybe this is one more example AI won't ever fully replace us, right?\n",
    "\n",
    "## 5.2 Creating an external table with Google Cloud BigQuery\n",
    "\n",
    "For BigQuery, the easiest way to create an external table is through the Cloud Console. See below example:\n",
    "\n",
    "![BigQuery create external table](images/bq-create-table.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 Explore the opportunities of privacy-safe data 🔍💡\n",
    "\n",
    "Now that you have had some practice applying STRM privacy transformations to data, what opportunities do you see for your own use cases, or in general? Can you see how data contracts could help your organization bridge the gap between legal and tech, and balance risk/compliance with utility?\n",
    "\n",
    "If you haven't already done so, you may want to return to the optional sections and give the STRM (Python) driver a try to send \"real-time\" events, or even create a data contract for one of your own data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "2736eb38-e836-499a-8141-5c8ce7404587",
    "deepnote_cell_type": "text-cell-p",
    "formattedRanges": [],
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "# 7 The End 🏁\n",
    "\n",
    "<div class=\"row\">\n",
    "  <div style=\"float: left; width: 50%; padding: 5px\">\n",
    "\n",
    "        \n",
    "Thank you for joining this workshop, and we hope you enjoyed it. If you'd like to explore more of our platform, try out one of these tutorials:\n",
    "* **On the fly decryption**  \n",
    "  Why persist decrypted/raw data when you can decrypt on the fly ([Google BigQuery](https://strmprivacy.io/posts/on-the-fly-decryption-bigquery/), [AWS Redshift](https://strmprivacy.io/posts/on-the-fly-decryption-redshift/), [Apache Spark](https://strmprivacy.io/posts/on-the-fly-decryption-spark/), [Databricks](https://strmprivacy.io/posts/on-the-fly-decryption-spark/), [Snowflake](https://strmprivacy.io/posts/on-the-fly-decryption-snowflake/))?\n",
    "* **[Data Subject Management](https://strmprivacy.io/posts/batchjobs-and-datasubjects/)**  \n",
    "  Know which encryption keys to delete for RTBF (Right To Be Forgotten) requests or know get a hold of which data records belong to a specific data subject for DSAR (Data Subject Access Requests), without needing to query many data.\n",
    "* **[Batch Job Data Processing](https://strmprivacy.io/posts/batchjobs-and-datasubjects/)**  \n",
    "  Not all data is streaming, we know, and therefore STRM Privacy also supports processing batches of data.\n",
    "\n",
    "Currently, we're working on:\n",
    "* Data contract wizard incl review/approve flow\n",
    "* Purpose mapping\n",
    "\n",
    "Thanks again, and feel free to reach out to any of us for more details or if you have questions, for example about a use case within your company. Want to [stay in touch](https://share-eu1.hsforms.com/1pW7xWg-pRia9cCK-N_2dzAfw8w8)? Scan the QR code (or click the image)!\n",
    "\n",
    "Feedback is welcome 🪞👀! \n",
    "\n",
    "      \n",
    "  </div>\n",
    "  <div style=\"float: left; width: 50%; padding: 5px\">\n",
    "    <a href=\"https://share-eu1.hsforms.com/1pW7xWg-pRia9cCK-N_2dzAfw8w8\" target=\"_blank\">\n",
    "      <img src=\"images/stay-in-touch.png\" style=\"width:100%\">\n",
    "    </a>\n",
    "  </div>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "deepnote": {},
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "681be7708cf844589c24db36e0a5d2d9",
  "deepnote_persisted_session": {
   "createdAt": "2022-12-01T11:31:14.277Z"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}