Merge branch 'main' of https://github.com/SEMICeu/STR-AP

prototype/images/file1

@@ -0,0 +1 @@
+

prototype/technical_prototype/Activity_Data_Sharing.md

@@ -0,0 +1,129 @@
+| Disclaimer  |
+|-----------|
+| This Readme was prepared for DG Grow by PwC EU Services. The views expressed in this report are purely those of the authors and may not, in any circumstances, be interpreted as stating an official position of the European Commission. The European Commission does not guarantee the accuracy of the information included in this report, nor does it accept any responsibility for any use thereof. Reference herein to any specific products, specifications, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favouring by the European Commission. All care has been taken by the author to ensure that s/he has obtained, where necessary, permission to use any parts of manuscripts including illustrations, maps, and graphs, on which intellectual property rights already exist from the titular holder(s) of such rights or from her/his or their legal representative.|
+
+# 1. Introduction  
+This document outlines the architecture and implementation details for sharing activity data from STR platforms to the SDEP. It covers the ability to receive and process activity data, ensuring secure and efficient data transmission using Kafka. The detailed Swagger documentation for the POST /api/v0/str/activity-data endpoint is included to provide a comprehensive guide for developers.
+
+# 2. Architecture Overview
+
+## 2.1. Components
+- API Gateway: Routes incoming requests to appropriate microservices.
+- SDEP: Handles the processing and retrieval of activity data.
+- Kafka: Message broker for reliable data transmission.
+
+
+## 2.2 Technologies 
+
+- Programming Language: Go
+- API Documentation: Swagger/OpenAPI
+- Messaging: Kafka
+- Deployment: Kubernetes, HELM
+
+
+## 2.3 FLOW
+**1. Data Submission by STR Platforms:**
+- The STR platform sends a POST request to the SDEP endpoint /api/v0/str/activity-data with the activity data.
+
+**2. Decompose data object**
+
+**3. Data Validation:**
+
+Example Validation Checks:
+- Ensure unitId is present and valid.
+- Check if numberOfGuests is a positive integer.
+- Validate the date format in stayDuration.
+- Ensure countryOfResidence matches a recognized country code.
+- Ensure aAreaID matches a known areaIid.
+
+**4. Kafka Topic for Activity Data:**
+A dedicated Kafka topic, say activity-data, is created to handle the activity data submissions.
+Each record in this topic represents an entry of activity data submitted by an STR platform.
+
+## 2.4 Endpoint Documentation 
+
+**Endpoint:** POST /api/v0/str/activity-data
+
+**Description:**  Allows an STR platform to submit activity data, which includes details like the number of guests, the period of stay, and the location of the unit.
+
+**Request:**
+- URL: /api/v0/str/activity-data
+- Method: POST
+- Content-Type: application/json
+- Authentication: Bearer Token (JWT)
+
+**Request Body:**
+
+```json
+{
+  "data": [
+    {
+      "numberOfGuests": 3,
+      "countryOfGuests": [
+        "ITA",
+        "NLD"
+      ],
+      "temporal": {
+        "startDateTime": "2024-07-21T17:32:28Z",
+        "endDateTime": "2024-07-25T17:32:28Z"
+      },
+      "address": {
+        "street": "123 Main St",
+        "city": "Brussels",
+        "postalCode": "1000",
+        "country": "BEL"
+      },
+      "hostId": "placeholder-host-id",
+      "unitId": "placeholder-unit-id",
+      "areaId": "placeholder-area-id"
+    }
+  ],
+  "metadata": {
+    "platform": "booking.com",
+    "submissionDate": "2024-07-21T17:32:28Z",
+    "additionalProp1": {}
+  }
+}
+```
+
+**Request Parameters:**
+- hostId: The ID of the host providing the accommodation.
+- unitId: The ID of the unit being rented.
+- numberOfGuests: The number of guests staying at the unit.
+- temporal: The period during which the unit is rented, with startDate and endDate.
+- address: The physical address of the unit.
+- areaId: The ID of the Area of the relevant CA 
+
+**Responses:**
+
+**201 OK**
+```json
+{
+  "status": "success",
+  "message": "Activity data submitted successfully."
+}
+```
+**400 Bad Request**
+```json
+{
+  "status": "error",
+  "message": "Invalid input data."
+}
+```
+**401 Unauthorized**
+```json
+{
+  "status": "error",
+  "message": "Unauthorized access."
+}
+```
+**500 Internal Server Error**
+```json
+{
+  "status": "error",
+  "message": "An unexpected error occurred."
+}
+```
+
+
+

prototype/technical_prototype/EU_STR_connection.md

@@ -0,0 +1,237 @@
+# EU STR - Single Digital Entry Point
+Base URL: eu-str.sdep-pilot.eu/api/v0
+
+Swagger: https://eu-str.sdep-pilot.eu/swagger/index.html#/
+
+A gateway for teh electronic transmission of data between online short-term rental platforms and competent authorities, ensuring priority of development is: 1. listings, 2. activity data, 3. area. 
+
+To obtain API credentials, please contact: wouter.travers@pwc.com via email. 
+
+## Setup
+This notebook is a collection of `curl` commands that can be executed in the terminal to validate the API endpoints\.
+Besides `curl` also `jq` is used for formatting purposes\. 
+### client credentials
+The `bw` command is the Bitwarden password manager CLI command\.
+Replace the CLIENT\_ID value\, with your personal CLIENT\_ID\. And provison your CLIENT\_SECRET
+```warp-runnable-command
+CLIENT_ID=plQ8P4mNxEoKFLY0NnYvfkO1Ak9HozjG \
+CLIENT_SECRET=$(bw get password plQ8P4mNxEoKFLY0NnYvfkO1Ak9HozjG)
+```
+The client SECRET can also be captured via CLI read\:
+```warp-runnable-command
+read CLIENT_SECRET
+```
+Get an OAUTH token \( from the `/token` endpoint\)
+```warp-runnable-command
+# Compose the JSON string using jq
+DATA=$(jq -n \
+          --arg client_id "$CLIENT_ID" \
+          --arg client_secret "$CLIENT_SECRET" \
+          --arg audience "https://str.eu" \
+          --arg grant_type "client_credentials" \
+          '{client_id: $client_id, client_secret: $client_secret, audience: $audience, grant_type: $grant_type}')
+
+# Get the token          
+TOKEN=$(curl -s --request POST \
+  --url https://tt-dp-dev.eu.auth0.com/oauth/token \
+  --header 'content-type: application/json' \
+  --data $DATA | jq -r .access_token )
+```
+### API host
+```warp-runnable-command
+HOST=eu-str.sdep-pilot.eu
+
+```
+## Test API
+### Unauthenticated health endpoint
+```warp-runnable-command
+curl -s https://$HOST/api/v0/ping | jq -r .
+
+```
+This should return\:
+```json
+{
+  "status": "ok"
+}
+```
+## Activity data
+### Submit of activity data
+Multiple records can be posted at ounce\, but will be stored as individual records\.
+For demo purposes\, we can set an invalid country code\.
+```warp-runnable-command
+curl -s -X POST https://$HOST/api/v0/str/activity-data \
+--header "Authorization: Bearer $TOKEN" \
+--header "Content-Type: application/json" \
+--data '{
+  "data": [
+    {
+      "numberOfGuests": 10,
+      "countryOfGuests": [
+        "ITZ",
+        "NLD"
+      ],
+      "temporal": {
+        "startDateTime": "2024-07-21T17:32:28Z",
+        "endDateTime": "2024-07-25T17:32:28Z"
+      },
+      "address": {
+        "street": "123 Main St",
+        "city": "Brussels",
+        "postalCode": "1000",
+        "country": "BEL"
+      },
+      "hostId": "placeholder-host-id",
+      "unitId": "placeholder-unit-id",
+      "areaId": "placeholder-area-id"
+    },
+    {
+      "numberOfGuests": 2,
+      "countryOfGuests": [
+        "BEL"
+      ],
+      "temporal": {
+        "startDateTime": "2024-07-21T17:32:28Z",
+        "endDateTime": "2024-07-25T17:32:28Z"
+      },
+      "address": {
+        "street": "123 Main St",
+        "city": "Brussels",
+        "postalCode": "1000",
+        "country": "BEL"
+      },
+      "hostId": "placeholder-host-id",
+      "unitId": "placeholder-unit-id",
+      "areaId": "placeholder-area-id"
+    },
+    {
+      "numberOfGuests": 3,
+      "countryOfGuests": [
+        "BEL"
+      ],
+      "temporal": {
+        "startDateTime": "2024-07-21T17:32:28Z",
+        "endDateTime": "2024-07-25T17:32:28Z"
+      },
+      "address": {
+        "street": "123 Main St",
+        "city": "Brussels",
+        "postalCode": "1000",
+        "country": "BEL"
+      },
+      "hostId": "placeholder-host-id",
+      "unitId": "placeholder-unit-id",
+      "areaId": "placeholder-area-id"
+    }          
+  ],
+  "metadata": {
+    "platform": "booking.com",
+    "submissionDate": "2024-07-21T17:32:28Z",
+    "additionalProp1": {}
+  }
+}
+' \
+| jq .
+```
+Data has been pushed to the Kafka topic\: `activity-data`
+
+### Only for demonstration purposes\, and if you have an own deployment 
+We can use `kcat` to list kafka topics and kafka topics content
+```warp-runnable-command
+kcat -L
+```
+```warp-runnable-command
+kcat -C -t activity-data -c 20 -f 'Headers: %h || Message value: %s\n'
+```
+Confluent Kafka also allows to views data in the topics\: [https\:\/\/confluent\.cloud\/environments\/env\-d19ry\/clusters\/lkc\-1dm6dj\/topics\/activity\-data\/message\-viewer](https://confluent.cloud/environments/env-d19ry/clusters/lkc-1dm6dj/topics/activity-data/message-viewer)
+Note also the header data contains the OAUTH2 app name\.
+
+### Retrieve activity data submitted to the SDEP
+```warp-runnable-command
+curl -s https://$HOST/api/v0/ca/activity-data \
+--header "Authorization: Bearer $TOKEN" \
+| jq .
+```
+Note each consumer has it\'s offset and will only get \"new\" published data\.
+
+## Listings
+### Submit of listings data
+```warp-runnable-command
+curl -s -X POST https://$HOST/api/v0/str/listings \
+--header "Authorization: Bearer $TOKEN" \
+--header "Content-Type: application/json" \
+--data '
+{
+  "data": [
+    {
+      "registrationNumber": "1234",
+      "Unit": {
+        "description": "string",
+        "floorLevel": "string",
+        "address": {
+          "street": "Culliganlaan 5",
+          "city": "Diegem",
+          "postalCode": "1831",
+          "country": "BEL"
+        },
+        "obtainedAuth": true,
+        "subjectToAuth": true,
+        "numberOfRooms": 0,
+        "occupancy": 0,
+        "purpose": "string",
+        "type": "string",
+        "url": "STR-Platform.com/1234"
+      }
+    }
+  ],
+  "metadata": {
+    "platform": "STR-Platform"
+  }
+}
+'\
+| jq .
+```
+### Only for demonstration purposes\, and if you have an own deployment
+Consuming data from the kafka topic `listings`
+```warp-runnable-command
+kcat -C -t listings -c 20 -f 'Headers: %h || Message value: %s\n'
+```
+### Retrieving listing data
+The limit parameter is optional\.
+This data retrieval is also incremental and will only return \"new\" data\.
+```warp-runnable-command
+curl -s https://$HOST/api/v0/ca/listings \
+--header "Authorization: Bearer $TOKEN" \
+| jq .
+```
+## Area
+### Shape file upload
+```warp-runnable-command
+curl -s -X POST https://$HOST/api/v0/ca/area \
+--header "Authorization: Bearer $TOKEN" \
+-F "file=@/Users/thierryturpin/GolandProjects/str-ap-internal/BELGIUM_-_Regions.shp"
+```
+### List shape files
+```warp-runnable-command
+curl -s https://$HOST/api/v0/str/area \
+--header "Authorization: Bearer $TOKEN" \
+| jq .
+```
+### Download of a shape file
+Set the ID returned by the previous command
+```warp-runnable-command
+curl -s https://$HOST/api/v0/str/area/01J0R0GC700000000000000000 \
+--header "Authorization: Bearer $TOKEN" \
+-o downloaded_shape_file.shp
+```
+Verify the downloaded file
+```warp-runnable-command
+file downloaded_shape_file.shp
+
+```
+### Verify SSL certificate
+Only relevant for own deployments\:
+```warp-runnable-command
+export SERVER_NAME=$HOST
+export PORT=443 
+openssl s_client -servername $SERVER_NAME -connect $SERVER_NAME:$PORT | openssl x509 -noout -dates
+```