Skip to content
This repository has been archived by the owner on Jan 19, 2024. It is now read-only.

Commit

Permalink
aws backup|restore support and backup cronjob (#118)
Browse files Browse the repository at this point in the history 
* #backup aws support #100

* #backup secretName option

* #restore aws support for core pods

* #restore aws support for read pods

* #default values updated

* #single secret path for backup

* #restore deployment-scenarios
  • Loading branch information
@srfaytkn
srfaytkn authored Oct 16, 2020
1 parent e592460 commit c126657
Show file tree
Hide file tree
Showing 12 changed files with 554 additions and 359 deletions.
83 changes: 27 additions & 56 deletions deployment-scenarios/cluster-restore.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,60 +1,31 @@
# This scenario creates a cluster
# that automatically restores from a backup with the given
# parameters in the "initContainers" section.
#
# IN ORDER FOR THIS TO WORK - you must have pre-created a secret, and
# you must modify the BUCKET parameter, and make sure backups are in place
# to be restored. See the documentation for more information.
# See the documentation for more information.
###############################################################
restoreSecret: neo4j-service-key

neo4jPassword: test
plugins: "[\"apoc\"]"
core:
standalone: false
numberOfServers: 3
initContainers:
- name: restore-from-backup
image: gcr.io/neo4j-helm/restore:4.1.0-1
imagePullPolicy: Always
volumeMounts:
- name: datadir
mountPath: /data
# The statefulset will already have a volume named "creds" when restoreSecret is set.
# Make sure to mount this to /auth
- name: creds
mountPath: /auth
env:
- name: BUCKET
value: gs://graph-backup-storage
- name: DATABASE
value: neo4j,system
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /auth/credentials.json
- name: FORCE_OVERWRITE
value: "true"

numberOfServers: 3
restore:
enabled: true
image: gcr.io/neo4j-helm/restore
imageTag: 4.1.0-1
secretName: neo4j-gcp-credentials #neo4j-aws-credentials
database: neo4j,system
cloudProvider: gcp #aws
bucket: gs://test-neo4j #s3://test-neo4j
timestamp: "latest"
forceOverwrite: true
purgeOnComplete: true
readReplica:
numberOfServers: 1
initContainers:
- name: restore-from-backup
image: gcr.io/neo4j-helm/restore:4.1.0-1
imagePullPolicy: Always
volumeMounts:
# - name: datadir
# mountPath: /data
# The statefulset will already have a volume named "creds" when restoreSecret is set.
# Make sure to mount this to /auth
- name: creds
mountPath: /auth
env:
- name: BUCKET
value: gs://graph-backup-storage
- name: DATABASE
value: neo4j,system
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /auth/credentials.json
- name: FORCE_OVERWRITE
value: "true"

acceptLicenseAgreement: "yes"
neo4jPassword: other
numberOfServers: 1
restore:
enabled: true
image: gcr.io/neo4j-helm/restore
imageTag: 4.1.0-1
secretName: neo4j-gcp-credentials #neo4j-aws-credentials
database: neo4j,system
cloudProvider: gcp #aws
bucket: gs://test-neo4j #s3://test-neo4j
timestamp: "latest"
forceOverwrite: true
purgeOnComplete: true

91 changes: 75 additions & 16 deletions doc/docs/modules/ROOT/pages/backup.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,34 +50,83 @@ In Neo4j 4.0, the system can be multidatabase; most systems have at least 2 DBs,

First you want to create a kubernetes secret that contains the content of your account service key. This key must have permissions to access the bucket and backup set that you're trying to restore.

**AWS**

- You must create the credential file and this file should look like this:
```aws-credentials
[default]
region=
aws_access_key_id=
aws_secret_access_key=
```
- You have to create a secret for this file
```shell
kubectl create secret generic neo4j-aws-credentials \
--from-file=credentials=aws-credentials
```
**GCP**

- You must create the credential file and this file should look like this:
- You don't actually need to create this file; this is the natural format of a JSON service key from Google
```gcp-credentials.json
{
"type": "",
"project_id": "",
"private_key_id": "",
"private_key": "",
"client_email": "",
"client_id": "",
"auth_uri": "",
"token_uri": "",
"auth_provider_x509_cert_url": "",
"client_x509_cert_url": ""
}
```

- You have to create a secret for this file
```shell
MY_SERVICE_ACCOUNT_KEY=$HOME/.google/my-service-key.json
kubectl create secret generic neo4j-service-key \
--from-file=credentials.json=$MY_SERVICE_ACCOUNT_KEY
kubectl create secret generic neo4j-gcp-credentials \
--from-file=credentials=gcp-credentials.json
```
The backup container is going to take this kubernetes secret
(named `neo4j-service-key`) and is going to mount it as a file
inside of the backup container (`/auth/credentials.json`). That
file will then be used to authenticate the storage client that we
need to upload the backupset to cloud storage when it's complete.
[NOTE]
**'--from-file=credentials=<your-config-path>' here is important**


### Running a Backup

The backup method is itself a mini-helm chart, and so to run a backup, you just
do this as a minimal required example:

**AWS**

```shell
helm install my-neo4j-backup . \
--set neo4jaddr=my-neo4j.default.svc.cluster.local:6362 \
--set bucket=s3://my-bucket \
--set database="neo4j\,system" \
--set cloudProvider=aws \
--set secretName=neo4j-aws-credentials \
--set jobSchedule="0 */12 * * *"
```
**GCP**
```shell
helm install my-backup-deployment . \
helm install my-neo4j-backup . \
--set neo4jaddr=my-neo4j.default.svc.cluster.local:6362 \
--set bucket=gs://my-bucket/ \
--set bucket=gs://my-bucket \
--set database="neo4j\,system" \
--set secretName=neo4j-service-key
```
--set cloudProvider=gcp \
--set secretName=neo4j-gcp-credentials \
--set jobSchedule="0 */12 * * *"
```
[NOTE]
**This command must be run in 'https://github.com/neo4j-contrib/neo4j-helm/tree/master/tools/backup'**

from within the tools/backup directory
where the chart resides. You must have first created a `neo4j-service-key`
secret in the same namespace as your Neo4j is running.

If all goes well, after a period of time when the Kubernetes Job is complete, you
will simply see the backup files appear in the designated bucket.
Expand All @@ -86,13 +135,23 @@ will simply see the backup files appear in the designated bucket.
**If your backup does not appear, consult the job container logs to find out
why**


**If you want to get a hot backup before schedule, you can use this command:**

```shell
kubectl create job --from=cronjob/my-neo4j-backup-job neo4j-hot-backup
```

**Required parameters**

* `neo4jaddr` pointing to an address where your cluster is running, ideally the
discovery address.
* `bucket` where you want the backup copied to. It should be `gs://bucketname`. This parameter may include a relative path (`gs://bucketname/mycluster`)
* `bucket` where you want the backup copied to. It should be `gs://bucketname` or `s3://bucketname`.
* `databases` a comma separated list of databases to back up. The default is
`neo4j,system`. If your DBMS has many individual databases, you should change this.
* `cloudProvider` Which cloud service do you want to keep backups on?(gcp or aws)
* `secretName` the name of the secret you created (neo4j-gcp-credentials|neo4j-aws-credentials)
* `jobSchedule` what intervals do you want to take backup? It should be cron like "0 */12 * * *". You can set your own schedule(https://crontab.guru/#0_*/12_*_*_*)
**Optional environment variables**

Expand Down
110 changes: 72 additions & 38 deletions doc/docs/modules/ROOT/pages/restore.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,29 +27,48 @@ directory structure that come out of archived backups in order to restore proper

### Create a service key secret to access cloud storage

First you want to create a kubernetes secret that contains the content of your account service key. This key must have permissions to access the bucket and backup set that you're trying to restore.
First you want to create a kubernetes secret that contains the content of your account service key. This key must have permissions to access the bucket and backup set that you're trying to restore.

```shell
MY_SERVICE_ACCOUNT_KEY=$HOME/.google/my-service-key.json
kubectl create secret generic neo4j-service-key \
--from-file=credentials.json=$MY_SERVICE_ACCOUNT_KEY
**AWS**

- You must create the credential file and this file should look like this:
```aws-credentials
[default]
region=
aws_access_key_id=
aws_secret_access_key=
```
The restore container is going to take this kubernetes secret
(named `neo4j-service-key`) and is going to mount it as a file
inside of the backup container (`/auth/credentials.json`). That
file will then be used to authenticate the storage client that we
need to upload the backupset to cloud storage when it's complete.
- You have to create a secret for this file
```shell
kubectl create secret generic neo4j-aws-credentials \
--from-file=credentials=aws-credentials
```
In `values.yaml`, then configure the secret you set here like so:
**GCP**

- You must create the credential file and this file should look like this:
```gcp-credentials.json
{
"type": "",
"project_id": "",
"private_key_id": "",
"private_key": "",
"client_email": "",
"client_id": "",
"auth_uri": "",
"token_uri": "",
"auth_provider_x509_cert_url": "",
"client_x509_cert_url": ""
}
```yaml
restoreSecret: neo4j-service-key
```

This allows the core and read replica nodes to access that service key
as a volume. That volume being present within the containers is necessary for the
next step, and will be mounted as `/auth/credentials.json` inside the container.
- You have to create a secret for this file
```shell
kubectl create secret generic neo4j-gcp-credentials \
--from-file=credentials=gcp-credentials.json
```
If this service key secret is not in place, the auth information will not be able to be mounted as
a volume in the initContainer, and your pods may get stuck/hung at "ContainerCreating" phase.
Expand All @@ -61,35 +80,50 @@ Refer to the single instance restore deploy scenario to see how the initContaine
What you will need to customize and ensure:
* Ensure you have created the appropriate secret and set its name
* Ensure that the volume mount to /auth matches the secret name you created above.
* Ensure that your BUCKET, and GOOGLE_APPLICATION_CREDENTIALS are
set correctly given the way you created your secret.
* Ensure that your BUCKET, and credentials are set correctly given the way you created your secret.

The example scenario above creates the initContainer just for core nodes. It's strongly recommended you do the same for `readReplica.initContainers` if you are using read replicas. If you restore only to core nodes and not to read replicas, when they start the core nodes will replicate the data to the read replicas. This will work just fine, but may result in longer startup times and much more bandwidth.

## Restore Environment Variables for the Init Container

### Required

- `GOOGLE_APPLICATION_CREDENTIALS` - path to a file with a JSON service account key (see credentials below). Defaults to /auth/credentials.json
- `BUCKET` - the storage bucket where backups are located, e.g. `gs://bucketname`. This parameter may include a relative path (`gs://bucketname/mycluster`)
- `DATABASE` - comma-separated list of databases to restore, e.g. neo4j,system
* `TIMESTAMP` - this defaults to "latest". See the backup container's documentation
on the latest pointer. But you may set this to a particular timestamp to restore
that exact moment in time. This timestamp must match the filename in storage.
So if you want to restore the backup set at `neo4j-2020-06-16-12:32:57.tar.gz ` then
the TIMESTAMP would be `2020-06-16-12:32:57`.
### Optional
- To restore you need to add the necessary parameters to values.yaml and this file should look like this:
```values.yaml
...
core:
...
restore:
enabled: true
secretName: (neo4j-gcp-credentials|neo4j-aws-credentials) #required
database: neo4j,system #required
cloudProvider: (gcp|aws) #required
bucket: (gs|s3)://test-neo4j #required
timestamp: "latest" #optional #default:"latest"
forceOverwrite: true #optional #default:true
purgeOnComplete: true #optinal #default:true
readReplica:
...
restore:
enabled: true
secretName: (neo4j-gcp-credentials|neo4j-aws-credentials) #required
database: neo4j,system #required
cloudProvider: (gcp|aws) #required
bucket: (gs|s3)://test-neo4j #required
timestamp: "2020-06-16-12:32:57" #optional #default:"latest"
forceOverwrite: true #optional #default:true
purgeOnComplete: true #optinal #default:true
...
```
- `PURGE_ON_COMPLETE` (defaults to true). If this is set to the value "true", the restore process will remove the restore artifacts from disk. With any other
value, they will be left in place. This is useful for debugging restores, to
see what was copied down from cloud storage and how it was expanded.
- `FORCE_OVERWRITE` if this is the value "true", then the restore process will overwrite and
destroy any existing data that is on the volume. Take care when using this in combination with
persistent volumes. The default is false; if data already exists on the drive, the restore operation will fail but preserve your data. **You must set this to true
if you want restore to work over-top of an existing database**.
- standard neo4j installation
```
helm install \
neo4j neo4j/neo4j \
-f values.yaml \
--set acceptLicenseAgreement=yes \
--version 4.1.1-1
```
## Warnings
## Warnings

A common way you might deploy Neo4j would be restore from last backup when a container initializes. This would be good for a cluster, because it would minimize how much catch-up
is needed when a node is launched. Any difference between the last backup and the rest of the
Expand Down
Loading

0 comments on commit c126657

Please sign in to comment.