Merge branch 'master' into TDRD-89-Excel-Metadata-Download-POC

beta-architecture/beta-architecture.md

@@ -13,16 +13,16 @@ This is not the final architecture, and it will change during the course of the
 TDR uses five AWS accounts:
 
 * Management (mgmt): the top-level account, which is used for running
-  cross-environment services like Jenkins CI and Grafana.
+  cross-environment services like central logging, Terraform state tracking, ECR and Grafana.
 * Sandbox (sbox): used for technical spikes
 * Integration (intg): the first environment that code is deployed to. Most
   services are deployed automatically to intg when code is merged to the
-  main/master branch. Most deployments are run programatically through Jenkins,
+  main/master branch. Most deployments are run programatically through Github actions,
   but we sometimes make temporary manual changes or deploy a branch when that's
   the easiest way to test something.
 * Staging (staging): a more stable environment used to check changes just before
-  deployment to production. Developers need to manually start a Jenkins job to
-  deploy code to this environment. Used for user research sessions.
+  deployment to production. Developers need to manually start a Githud action to
+  deploy code to this environment. Used for user research sessions and demonstrations to key stakeholders.
 * Production (prod): the environment used by real users. Deployment is the same
   as for staging.
 
@@ -41,42 +41,19 @@ This diagram shows a high level view of the interactions between the different A
 
 ![](./diagrams/tdr-transfer-sequence.svg)
 
-## Transform Engine Service
+## Digital Archiving Event Bus
 
-As part of the Find Case law service TDR accepts judgment transfers.
+TDR communicates with other Digital Archiving services using the Digital Archiving event bus hosted in the Transform Engine AWS accounts.
 
-TDR communicates with the Transfer Engine service to support the Find Case law service.
+TDR sends export messages to the event bus for other services to consume.
 
-The Transfer Engine service sits within a different AWS account. To allow communication between the two accounts, relevant IAM roles are given permission to access the AWS resources across the accounts.
+## Access Your Records (AYR) Service Integration
 
-### Communication Between TDR and Transform Engine
+The Access Your Records (AYR) service shares the same user base as TDR.
 
-* Upon export of a judgment transfer TDR send a message to an SQS queue belonging to the Transform Engine with details of the transfer allowing the Transform Engine to process the judgment.
-* If the Transform Engine requires a retry of any transfer, it sends a message to a TDR SQS queue that triggers a new notification to be sent back from TDR to the Transform Engine SQS queue.
-  * Note: the retry message from the Transform Engine does not trigger a re-export of the transfer.
-
-It is likely in the future that this functionality will be expanded to include all transfers from TDR, and not just judgment transfers.
+AYR uses the TDR Keycloak instance to manage its users and provide authentication and authorisaion for AYR's functionality.
 
-#### Transform Engine Version 2 Architecture
-
-Transform Engine is changing the way it communicates with external systems.
-
-See details of this here:
-* [TRE version 2 Architecture](https://github.com/nationalarchives/da-transform-dev-documentation/blob/master/architecture-decision-records/002-New-messaging-architecture.md)
-* [TDR to TRE Integration](https://github.com/nationalarchives/da-transform-dev-documentation/blob/master/architecture-decision-records/003-New-TDR-TRE-integration.md)
-
-TDR will support this new architecture, in parallel with the original, until TRE fully migrates to version 2.
-
-Once the migration from the original architecture to the new architecture is complete, the TDR resources used to support the original architecture will be removed.
-
-### Initial assumptions
-
-* We don't yet need to integrate TDR with any other systems, though we might
-  reuse the login system to let department users view their own records
-* Series and departments won't need to be updated very often, so for MVP it's OK
-  for this to be a developer task, even though eventually we might build an
-  admin UI or integrate with the planned catalogue API
-* Data stored in the export bucket can be transferred to the preservation system
+AYR makes use of Keycloak's provided APIs.
 
 ## Reference Generator Service
 
@@ -95,4 +72,13 @@ For more details on the service refer to here:
 TNA will be changing where we assign citeable references for individual born-digital records 
 This is currently done by our Digital Preservation Service. Following this change it will be done by our Digital Selection and Transfer Service.
 At the moment, TDR generates a system reference (UUID) which is not readily citeable and does not persist into the preservation system or Discovery.
-The reference generator service will be used to assign citeable references.
+The reference generator service will be used to assign citeable references.
+
+## Initial assumptions (historic)
+
+* We don't yet need to integrate TDR with any other systems, though we might
+  reuse the login system to let department users view their own records
+* Series and departments won't need to be updated very often, so for MVP it's OK
+  for this to be a developer task, even though eventually we might build an
+  admin UI or integrate with the planned catalogue API
+* Data stored in the export bucket can be transferred to the preservation system

manual/create-repo.md

@@ -8,7 +8,14 @@ Create a repo in <https://github.com/nationalarchives> by clicking the "New" but
 * The repo should be public unless you have a [good reason to make it closed][open-code]
 * Click "Create Repository"
 
-## 2. Add Terraform Secrets
+## 2. Add new repo to Synk
+
+* Log into Synk via Github: 
+* Switch to `nationalarchives` Organisation
+* From the Homepage use the `Add projects` dropdown to select Github
+* Find the new repo and add it to Synk (this can take a few moments to add the repo)
+
+## 3. Add Terraform Secrets
 
 If environment secrets are going to be needed in the workflow files (mentioned later), if you haven't already, you are going to have to define them in the `root_github.tf` file(s) of the Terraform projects:
 
@@ -20,7 +27,7 @@ Add a Terraform backend secrets module for your new repo to the `root_github.tf`
 
 Add a Terraform secrets module for your new repo to the `root_github.tf` file of the Terraform environments project [like so][terraform-environments-module-example] and make a PR for this change.
 
-## 3. Add a README
+## 4. Add a README
 
 On the next page, steps should appear for how to create a repo via the command line, if they do follow those
 instructions, if not, follow these instructions:
@@ -34,7 +41,7 @@ instructions, if not, follow these instructions:
 7. git remote add origin `git@github.com:nationalarchives/{name of repo}.git`
 8. git push -u origin main
 
-## 4. Add a licence
+## 5. Add a licence
 
 In the parent directory add an open source licence with the name "licence"
 
@@ -43,17 +50,17 @@ In the parent directory add an open source licence with the name "licence"
 * For documentation, add an MIT licence for the code and an [Open Government Licence][ogl] for the content. See this
   repo's LICENCE file the licence note in the README for an example.
 
-## 5. Add a config file
+## 6. Add a config file
 
 **If** the project contains a [dependabot supported package manager][supported-package-managers] then add
 a [dependabot config file][dependabot-config]
 to the `workflows` folder in the `.github` folder (create a .github folder if one doesn't exist). 
 
 If the project doesn't contain a dependabot supported package manager, then skip this step.
 
-## 6. Choose the correct settings for the repo
+## 7. Choose the correct settings for the repo
 
-### 6a. Add teams
+### 7a. Add teams
 
 Once in the repo, click the "Settings" button (at the top) to update the repo's settings:
 
@@ -66,11 +73,11 @@ Once in the repo, click the "Settings" button (at the top) to update the repo's
         * give **admin** access to this team
         * click the green "Add" button to add it
 
-### 6b. Automatically delete head branches
+### 7b. Automatically delete head branches
 
 Go to "General" settings (as the top), and scroll down to "Automatically delete head branches" and select this
 
-### 6c. Set rules for main branch
+### 7c. Set rules for main branch
 
 * In "Branches", you should see the "main" branch.
 * Under "Branch protection rules" select "Add rule"
@@ -86,13 +93,14 @@ Go to "General" settings (as the top), and scroll down to "Automatically delete
             * **Require branches to be up-to-date before merging** *(sub option)*
                 * **Note**: the GitHub Actions test job will need to be configured first before this status check will
                   appear; see step 7 (below) for more information on how to do this.
+                * **Note**: Snyk should be added to this list if available for the repo.
     * **Require signed commits**
     * **Include administrators**
 * Click the "Save changes" button
 
-### 7. Add file dependencies
+## 8. Add file dependencies
 
-#### 7a. Add workflow files
+### 8a. Add workflow files
 
 We use GiHub Actions in order to test, build and deploy our projects; since step 5 was all about requiring status checks,
 the GitHub Actions files it runs, need to be added.
@@ -112,7 +120,7 @@ the GitHub Actions files it runs, need to be added.
 * Once this is done, commit the files and push them to the repo; you will notice that the test job will start running
     * **Note**: In order to push to the repo, you'll need to create a new branch
 
-#### 7b. Add the files that the workflow files rely on
+### 8b. Add the files that the workflow files rely on
 
 The workflow files might have a rule that runs a command like `sbt assembly`; in order for it to do that it would need
 the `build.sbt` file to be present. Add the files that are necessary for the workflow files to run for example:
@@ -124,7 +132,7 @@ the `build.sbt` file to be present. Add the files that are necessary for the wor
 
 Make sure that each of the files you add, contain all the dependencies/commands necessary
 
-### 8. Select status checks that are required before merging
+## 9. Select status checks that are required before merging
 
 * Go back to the repo and select the "Settings"
 * Select "Branches"
@@ -133,15 +141,15 @@ Make sure that each of the files you add, contain all the dependencies/commands
   bar with the placeholder text "Search for status checks in the last week for this repository"
 * type in "test / test" and select that status check option
 
-### 9. Remove yourself from list of who can manage access
+## 10. Remove yourself from list of who can manage access
 
 * Go to the "Settings"
 * Under the "Manage access" section, you should see your account that has been given admin access
 * Next to your account, Click "remove"
     * *Since you are either a part of the "Transfer Digital Records" or "Transfer Digital Records Admins", you will
       still have access rights to this repo*
 
-### 10. Link other repos to project
+## 11. Link other repos to project
 
 * Add a link from the new project's README to this documentation, to add context for anyone who finds the project
 * Add the new project to the repo list in the README of **this** project

manual/development-process/manually-trigger-consignment-export.md

@@ -0,0 +1,32 @@
+# Manually Trigger Consignment Export Step Function
+
+**NOTE**: Before manually triggering a **judgment** consignment export double check with the Find Case Law (FCL) team that the user has not re-attempted the transfer to prevent duplicate judgments being published
+
+In some circumstances it may be necessary to manually trigger a consignment export, for example:
+* Consignment export failure;
+* Legitimate request from another Digital Archiving service for a re-export
+
+## Use AWS Console to manually execute Consignment Export Step Function
+1. Retrieve the relevant consignment id for the consignment to trigger the export for
+2. Log into the relevant AWS account
+3. Go to the TDRConsignmentExport{env} step function
+4. Click `Start Execution` and add the relevant json into the dialog box:
+    ```
+   {
+      "ConsignmentId": "{consignment id for consignment}"
+   }
+   ```
+5. Enter the a name for the new execution into the dialog box in the following form: `{initials}-{consignment id for consignment}`
+6. Click `Start execution`
+
+## Use AWS CLI to manually execute Consignment Export Step Function
+1. Retrieve the relevant consignment id for the consignment to trigger the export for
+2. Ensure have relevant credentials set to execute the step function
+3. Run the following command:
+    ```
+   $ aws cli start-execution --state-machine-arn {arn of the step function} \
+      --name {initials}-{consignment id for consignment} \
+      --input "input": "{\"ConsignmentId\" : \"{consignment id for consignment}\"}"
+   ```
+
+Full details of the `start-execution` command can be found here: https://docs.aws.amazon.com/cli/latest/reference/stepfunctions/start-execution.html

manual/development-setup/new-starters.md

@@ -1,15 +1,55 @@
 # New starters
 
-## Essential setup
+## Development Laptop Setup
 
-- Install [git secrets](git-secrets.md). You **must** do this before working on
-  TDR code changes, and when you switch to a new computer.
+### Install Ubuntu OS
+
+Ubuntu is the recommended OS to install on the development laptop.
+
+Instruction for installation can be found here: https://ubuntu.com/tutorials/install-ubuntu-desktop#1-overview
+
+**Full hard drive encryption must be implemented**
+
+### Install Virus Scanning / Firewall
+
+#### Virus Scanning
+
+Install ClamAV: https://help.ubuntu.com/community/ClamAV
+
+#### Firewall
+[TBC]
+
+### Software Installation
+
+* *Git*: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git
+* *Git Secrets*: [git secrets](git-secrets.md). You **must** do this before working on TDR code changes
+* *AWS CLI*: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
+* *Java*: https://ubuntu.com/tutorials/install-jre#1-overview
+* *Sbt*: https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html
+* *Intellij*: You should have a licence for the version (https://www.jetbrains.com/help/idea/installation-guide.html)
+* *Docker*: https://docs.docker.com/engine/install/ubuntu/
+  ** Ensure to do follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+* *Terraform*: https://developer.hashicorp.com/terraform/cli/install/apt
+* *Nvm*: https://github.com/nvm-sh/nvm
+* *Npm*: Use NVM to install: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
+* *Python*: https://docs.python-guide.org/starting/install3/linux/
+* *Chrome*: https://support.google.com/chrome/a/answer/9025903?hl=en
+* *Firefox*: https://support.mozilla.org/en-US/kb/install-firefox-linux
+* *MS Edge*: https://answers.microsoft.com/en-us/microsoftedge/forum/all/how-to-install-edge-on-linux-desktop/c6a1575a-8910-431f-b651-b24938da904e
 
 ## Accounts and access
 
-- To access the AWS environments, follow the "Access TDR AWS accounts as a
+* To access the AWS environments, follow the "Access TDR AWS accounts as a
   developer" guide in the [internal developer manual on Sharepoint][sharepoint-dev-manual]
-- To access the TDR GitHub repos, ask an admin of The National Archives' GitHub
+* To access the TDR GitHub repos, ask an admin of The National Archives' GitHub
   organisation to add you to the Transfer Digital Records team
 
+## Familiarisation
+
+### Accessibility
+
+Please read through this presentation for [background on accessibility][accessibility-presentation]
+
 [sharepoint-dev-manual]: https://nationalarchivesuk.sharepoint.com/:f:/r/sites/DA_Proj/Transfer%20Digital%20Records/Documentation/Developer%20manual
+
+[accessibility-presentation]: https://nationalarchivesuk.sharepoint.com/:f:/r/sites/DA_Proj/Transfer%20Digital%20Records/TDR%20UX%20and%20Accessibility/Accessibility/Introduction%20to%20Accessibility%20Testing%20May%202024?csf=1&web=1&e=RYRyi2

tdr-admins/tdr-user-administrator.md

@@ -38,7 +38,7 @@ TDR user administrators have rights and privileges to manage:
 1. Contact TDR team to request set up as a TDR user administrator: tdr@nationalarchives.gov.uk
 2. You will receive an email from the TDR team with:
   * your user name
-  * URL to the Keycloak application: https://auth.tdr.nationalarchives.gov.uk/auth/admin/tdr/console
+  * URL to the Keycloak application: https://auth.tdr.nationalarchives.gov.uk/admin/tdr/console
 3. A separate email will be sent with an URL link for you to set a password
 4. Ensure you have either Google Authenticator (https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en_GB) or Microsoft Authenticator (https://www.microsoft.com/en-us/account/authenticator) available as you will need these to log on to the Keycloak application
 5. Log on to the Keycloak application for the first time: