From 7b2cbc30614e01ef3bdb4cb2ce7b85a35a2251ac Mon Sep 17 00:00:00 2001 From: TomJKing Date: Tue, 7 May 2024 07:50:45 +0100 Subject: [PATCH 1/3] ADR for hosting service integration endpoints --- ...1-hosting-service-integration-endpoints.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 architecture-decision-records/0031-hosting-service-integration-endpoints.md diff --git a/architecture-decision-records/0031-hosting-service-integration-endpoints.md b/architecture-decision-records/0031-hosting-service-integration-endpoints.md new file mode 100644 index 0000000..9739512 --- /dev/null +++ b/architecture-decision-records/0031-hosting-service-integration-endpoints.md @@ -0,0 +1,56 @@ +# 31. Hosting Service Integration Endpoints + +**Date:** 2024-MM-DD + +## Context + +TDR needs to support transfers from additional sources other than network drives, for example Sharepoint, Google Drive, hard drives etc. + +This requires the exposure of endpoints so other services can integrate with TDR to allow the transfers from such sources. + +These endpoints will need to be hosted by the TDR infrastructure. + +Such transfers are likely to involve much larger volume transfers in terms of number of records and overall size. + +## Decision + +Decision to host the endpoints on a new AWS ECS Task within TDR's VPC. + +This provides the following advantages: +* Service integrations always available +* Flexibility to extend with future requirements +* Potential to handle high volume transfers +* Take advantage of TDR's existing access controls to the VPC (for example WAF rules) + +### Disadvantages to hosting on new ECS Task + +* Higher maintenance burden as less code re-use - potential mitigation with abstracting existing TDR frontend code to re-use +* Potential higher AWS cost running additional ECS task - potential mitigation with closing down service over weekends etc + +## Other Options Considered + +### Hosting on existing TDR Frontend ECS Task + +#### Advantages + +* Re-use existing code, such as authorisation and authentication, connections with other TDR services +* Lower maintenance burden +* Lower potential AWS cost burden + +#### Disadvantages + +* Code was not designed for supporting endpoints externally exposed +* Capacity issues with larger transfers, ECS task already at capacity at peak usage supporting the TDR frontend + +### Hosting on AWS API Gateway and Lambda + +#### Advantages + +* Serverless architecture with potential cost savings +* AWS API Gateway features + +#### Disadvantages + +* Large amount of additional AWS infrastructure to create and maintain +* Lambda "cold start" causing lag on transfers +* Securing endpoints whilst allowing external access will be difficult From 0f726ad9ee02ee281c7e7433c259fcad685944e9 Mon Sep 17 00:00:00 2001 From: TomJKing Date: Thu, 23 May 2024 08:19:25 +0100 Subject: [PATCH 2/3] Reword to include additional processing --- ....md => 0031-hosting-service-integrations.md} | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) rename architecture-decision-records/{0031-hosting-service-integration-endpoints.md => 0031-hosting-service-integrations.md} (65%) diff --git a/architecture-decision-records/0031-hosting-service-integration-endpoints.md b/architecture-decision-records/0031-hosting-service-integrations.md similarity index 65% rename from architecture-decision-records/0031-hosting-service-integration-endpoints.md rename to architecture-decision-records/0031-hosting-service-integrations.md index 9739512..26d808a 100644 --- a/architecture-decision-records/0031-hosting-service-integration-endpoints.md +++ b/architecture-decision-records/0031-hosting-service-integrations.md @@ -1,25 +1,26 @@ -# 31. Hosting Service Integration Endpoints +# 31. Hosting Service Integrations **Date:** 2024-MM-DD ## Context -TDR needs to support transfers from additional sources other than network drives, for example Sharepoint, Google Drive, hard drives etc. +TDR needs to support transfers from additional sources other than network drives, for example SharePoint, Google Drive, hard drives etc. -This requires the exposure of endpoints so other services can integrate with TDR to allow the transfers from such sources. +This will require the exposure of endpoints so other services can integrate with TDR to allow the transfers from such sources, along with additional processing of metadata and records. -These endpoints will need to be hosted by the TDR infrastructure. +The additional processing and endpoints will need to be hosted by the TDR infrastructure. -Such transfers are likely to involve much larger volume transfers in terms of number of records and overall size. +Such transfers are likely to involve much larger volume transfers in terms of number of records and overall size. In addition bespoke processing of the data is likely to be needed. ## Decision -Decision to host the endpoints on a new AWS ECS Task within TDR's VPC. +Decision to host the necessary code for supporting transfer from additional sources on a new AWS ECS Task within existing TDR's VPC. This provides the following advantages: * Service integrations always available * Flexibility to extend with future requirements -* Potential to handle high volume transfers +* Potential to handle higher volume transfers +* Less burden on existing TDR infrastructure used to support network transfers * Take advantage of TDR's existing access controls to the VPC (for example WAF rules) ### Disadvantages to hosting on new ECS Task @@ -47,7 +48,7 @@ This provides the following advantages: #### Advantages * Serverless architecture with potential cost savings -* AWS API Gateway features +* Use AWS API Gateway features #### Disadvantages From 27f9aad861d0eda67fa45d1dbefd995ce186ab42 Mon Sep 17 00:00:00 2001 From: TomJKing Date: Fri, 7 Jun 2024 08:29:27 +0100 Subject: [PATCH 3/3] Add date and renumber --- ...e-integrations.md => 0032-hosting-service-integrations.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename architecture-decision-records/{0031-hosting-service-integrations.md => 0032-hosting-service-integrations.md} (97%) diff --git a/architecture-decision-records/0031-hosting-service-integrations.md b/architecture-decision-records/0032-hosting-service-integrations.md similarity index 97% rename from architecture-decision-records/0031-hosting-service-integrations.md rename to architecture-decision-records/0032-hosting-service-integrations.md index 26d808a..75bbf1f 100644 --- a/architecture-decision-records/0031-hosting-service-integrations.md +++ b/architecture-decision-records/0032-hosting-service-integrations.md @@ -1,6 +1,6 @@ -# 31. Hosting Service Integrations +# 32. Hosting Service Integrations -**Date:** 2024-MM-DD +**Date:** 2024-06-07 ## Context