Merge pull request #96 from Ontotext-AD/GDB-9426-various-improvements…

…-pt3

GDB-9426: Various improvements part 3

CHANGELOG.md

@@ -10,12 +10,13 @@ TODO: short section about the most notable changes (decoupling, naming, plugins,
 
 TODO: decide how detailed we want this to be
 
+- Updated the chart to require Kubernetes version 1.24+
 - Resource names are no longer hardcoded and are using the templates for `nameOverride` and `fullnameOverride`
 - Renamed `extraLabels` to just `labels`
-- Renamed GraphDB storage PVC prefix to `graphdb-storage` and server import folder to `graphdb-server-import`
+- Renamed GraphDB storage PVC template name prefix to `storage` and server import folder to `import`
 - Removed setting FQDN as hostnames in GraphDB and the proxy in favor of dynamically resolving and configuring the hostnames in the init containers
 - Renamed `messageSize` to `messageSizeKB` in the cluster creation configuration
-- Renamed `java_args` to `javaArguments`
+- Renamed `java_args` to `defaultJavaArguments` and added a separate `javaArguments` that can be used for additional configurations
 - Removed the default logback XML configuration and configmap in favor of an [example](examples/custom-logback) and a new configuration options
   under `logging.logback`
 - Removed `global.storageClass` in favor of using by default the default storage class in the cluster. Templates will no longer
@@ -46,6 +47,10 @@ TODO: decide how detailed we want this to be
 - Moved `graphdb.security` configurations to `security`
 - Updated the Service type of the proxy to be ClusterIP by default
 - Renamed `pdb` to `podDisruptionBudget` and renamed `podDisruptionBudget.create` to `podDisruptionBudget.enabled` for consistency
+- Removed configuration overrides from the default `GDB_JAVA_OPTS`: `enable-context-index`, `entity-pool-implementation`
+  and `health.max.query.time.seconds`
+- Removed `default.min.distinct.threshold` from the default `defaultJavaArguments` values
+- Moved `provisioningUsername` and `provisioningPassword` under `security.provisioner`
 
 ### New
 
@@ -87,6 +92,26 @@ TODO: decide how detailed we want this to be
 - Added default Secret objects for GraphDB and the proxy that contain sensitive GraphDB configurations
 - Added `configuration.secretProperties` and `proxy.secretProperties` for appending additional sensitive GraphDB configurations if needed
 - Added `proxy.pdb` for configuring a pod disruption budget for the GraphDB Proxy
+- Added `updateStrategy` and `proxy.updateStrategy` for controlling the strategy when updating pods
+- Added `podManagementPolicy` and `proxy.podManagementPolicy` for configuring how the pods are created and scaled
+- Added `automountServiceAccountToken` with default value `false` effectively ejecting the service account token by default
+- Added `schedulerName` and `proxy.schedulerName` for overriding the default Kubernetes scheduler
+- Added `dnsConfig`, `dnsPolicy`, `proxy.dnsConfig` and `proxy.dnsPolicy` for customizing the DNS resolution if needed
+- Added `proxy.initContainerSecurityContext` and `proxy.initContainerResources` to avoid using the configurations from GraphDB
+- Added `extraContainers` and `proxy.extraContainers` for inserting additional containers into the pods of GraphDB and the GraphDB proxy
+- Added `extraObjects` as a way to insert additional Kubernetes objects into the deployment
+- Added `service.externalTrafficPolicy` and `service.proxy.externalTrafficPolicy` to override the policy to Local if needed
+- Added `service.healthCheckNodePort` and `service.proxy.healthCheckNodePort` to define a specific node port for LB health checks
+- Added `service.loadBalancerClass` and `service.proxy.loadBalancerClass` to select a specific load balancer implementation
+- Added `service.loadBalancerSourceRanges` and `service.proxy.loadBalancerSourceRanges` to restrict the external ingress traffic from the LB
+- Added `service.externalIPs` and `service.proxy.externalIPs` to use existing external IPs
+- Added `service.extraPorts` and `service.proxy.extraPorts` for exposing additional ports
+- Added configurations for extra `labels` and `annotations` for all persistent volume claim templates: `persistence`, `proxy.persistence`
+  and `import.volumeMount`
+- Added `jobs.backoffLimit` for configuring the retry count for all jobs
+- Added `jobs.ttlSecondsAfterFinished` for configuring the time in seconds for all jobs before deleting finished pods
+- Added `jobs.persistence.emptyDir` configurations for the default temporary storage for all jobs
+- Added `security.provisioner.existingSecret` and `security.provisioner.tokenKey` to provide an existing authentication token
 
 ### Updates
 
@@ -101,6 +126,10 @@ TODO: decide how detailed we want this to be
 - Removed `files/config/graphdb.properties` and `files/config/proxy/graphdb.properties` and moved any defined properties directly into the ConfigMap
   declarations
 - Moved GraphDB specific properties from `GDB_JAVA_OPTS` into the properties ConfigMaps
+- Added `-XX:-UseCompressedOops` in the default Java arguments to allow allocating heap sizes larger than 32GBs when the max heap size is based on
+  the `-XX:MaxRAMPercentage` Java option
+- Ejected the default service account token in the proxy pods
+- Overhauled NOTES.txt to be more helpful
 
 ## Version 10.6.0-R2
 

Chart.yaml

@@ -7,7 +7,7 @@ description: GraphDB is a highly efficient, scalable and robust graph database w
 type: application
 version: 11.0.0
 appVersion: 10.6.2
-kubeVersion: ^1.22.0-0
+kubeVersion: ^1.24.0-0
 home: https://graphdb.ontotext.com/
 icon: https://graphdb.ontotext.com/home/images/visual_Logo_GraphDB_02_12_2015.png
 maintainers:

README.md

@@ -473,6 +473,7 @@ about defining resource limits.
 | affinity | object | `{}` |  |
 | annotations | object | `{}` |  |
 | args | list | `[]` |  |
+| automountServiceAccountToken | bool | `false` |  |
 | cluster.clusterCreationTimeout | int | `60` |  |
 | cluster.clusterSecret | string | `"s3cr37"` |  |
 | cluster.config.electionMinTimeout | int | `8000` |  |
@@ -481,28 +482,33 @@ about defining resource limits.
 | cluster.config.messageSizeKB | int | `64` |  |
 | cluster.config.transactionLogMaximumSizeGB | int | `50` |  |
 | cluster.config.verificationTimeout | int | `1500` |  |
-| cluster.existingClusterSecret | string | `nil` |  |
-| cluster.existingClusterSecretKey | string | `nil` |  |
+| cluster.existingClusterSecret | string | `""` |  |
+| cluster.existingClusterSecretKey | string | `""` |  |
 | cluster.existingConfig | string | `""` |  |
 | cluster.existingConfigmapKey | string | `"cluster-config.json"` |  |
 | cluster.jobs.createCluster.enabled | bool | `true` |  |
 | cluster.jobs.patchCluster.enabled | bool | `true` |  |
 | cluster.jobs.scaleCluster.enabled | bool | `true` |  |
 | command | list | `[]` |  |
+| configuration.defaultJavaArguments | string | `"-XX:+UseContainerSupport -XX:MaxRAMPercentage=70 -XX:-UseCompressedOops"` |  |
 | configuration.externalUrl | string | `"http://graphdb.127.0.0.1.nip.io/"` |  |
-| configuration.javaArguments | string | `"-XX:+UseContainerSupport -XX:MaxRAMPercentage=70 -Ddefault.min.distinct.threshold=100m"` |  |
+| configuration.javaArguments | string | `""` |  |
 | configuration.properties | object | `{}` |  |
 | configuration.secretProperties | object | `{}` |  |
 | containerPorts.http | int | `7200` |  |
 | containerPorts.rpc | int | `7300` |  |
+| dnsConfig | object | `{}` |  |
+| dnsPolicy | string | `""` |  |
 | extraConfiguration.properties.configmapKey | string | `"graphdb.properties"` |  |
 | extraConfiguration.properties.existingConfigmap | string | `""` |  |
 | extraConfiguration.properties.existingSecret | string | `""` |  |
 | extraConfiguration.properties.secretKey | string | `"graphdb.properties"` |  |
 | extraContainerPorts | object | `{}` |  |
+| extraContainers | list | `[]` |  |
 | extraEnv | list | `[]` |  |
 | extraEnvFrom | list | `[]` |  |
 | extraInitContainers | list | `[]` |  |
+| extraObjects | list | `[]` |  |
 | extraVolumeMounts | list | `[]` |  |
 | extraVolumes | list | `[]` |  |
 | fullnameOverride | string | `""` |  |
@@ -520,7 +526,9 @@ about defining resource limits.
 | image.repository | string | `"ontotext/graphdb"` |  |
 | image.sha | string | `""` |  |
 | image.tag | string | `""` |  |
+| import.volumeMount.annotations | object | `{}` |  |
 | import.volumeMount.enabled | bool | `false` |  |
+| import.volumeMount.labels | object | `{}` |  |
 | import.volumeMount.volumeClaimTemplateSpec.accessModes[0] | string | `"ReadWriteOnce"` |  |
 | import.volumeMount.volumeClaimTemplateSpec.resources.requests.storage | string | `"10Gi"` |  |
 | ingress.annotations | object | `{}` |  |
@@ -540,9 +548,12 @@ about defining resource limits.
 | initialConfiguration.settings.existingConfigmap | string | `""` |  |
 | initialConfiguration.users.existingSecret | string | `""` |  |
 | initialConfiguration.users.secretKey | string | `"users.js"` |  |
+| jobs.backoffLimit | int | `10` |  |
+| jobs.persistence.emptyDir.sizeLimit | string | `"10Mi"` |  |
 | jobs.podSecurityContext | object | `{}` |  |
 | jobs.resources | object | `{}` |  |
 | jobs.securityContext | object | `{}` |  |
+| jobs.ttlSecondsAfterFinished | int | `300` |  |
 | labels | object | `{}` |  |
 | license.existingSecret | string | `""` |  |
 | license.licenseFilename | string | `"graphdb.license"` |  |
@@ -556,29 +567,36 @@ about defining resource limits.
 | nameOverride | string | `""` |  |
 | namespaceOverride | string | `""` |  |
 | nodeSelector | object | `{}` |  |
+| persistence.annotations | object | `{}` |  |
 | persistence.enabled | bool | `true` |  |
+| persistence.labels | object | `{}` |  |
 | persistence.volumeClaimTemplateSpec.accessModes[0] | string | `"ReadWriteOnce"` |  |
 | persistence.volumeClaimTemplateSpec.resources.requests.storage | string | `"5Gi"` |  |
 | podAnnotations | object | `{}` |  |
 | podDisruptionBudget.enabled | bool | `false` |  |
 | podDisruptionBudget.maxUnavailable | string | `""` |  |
 | podDisruptionBudget.minAvailable | string | `"51%"` |  |
 | podLabels | object | `{}` |  |
+| podManagementPolicy | string | `"Parallel"` |  |
 | podSecurityContext | object | `{}` |  |
 | proxy.affinity | object | `{}` |  |
 | proxy.annotations | object | `{}` |  |
 | proxy.args | list | `[]` |  |
 | proxy.command | list | `[]` |  |
-| proxy.configuration.javaArguments | string | `"-XX:+UseContainerSupport -XX:MaxRAMPercentage=70 -Ddefault.min.distinct.threshold=100m"` |  |
+| proxy.configuration.defaultJavaArguments | string | `"-XX:+UseContainerSupport -XX:MaxRAMPercentage=70"` |  |
+| proxy.configuration.javaArguments | string | `""` |  |
 | proxy.configuration.properties | object | `{}` |  |
 | proxy.configuration.secretProperties | object | `{}` |  |
 | proxy.containerPorts.http | int | `7200` |  |
 | proxy.containerPorts.rpc | int | `7300` |  |
+| proxy.dnsConfig | object | `{}` |  |
+| proxy.dnsPolicy | string | `""` |  |
 | proxy.extraConfiguration.properties.configmapKey | string | `"graphdb.properties"` |  |
 | proxy.extraConfiguration.properties.existingConfigmap | string | `""` |  |
 | proxy.extraConfiguration.properties.existingSecret | string | `""` |  |
 | proxy.extraConfiguration.properties.secretKey | string | `"graphdb.properties"` |  |
 | proxy.extraContainerPorts | object | `{}` |  |
+| proxy.extraContainers | list | `[]` |  |
 | proxy.extraEnv | list | `[]` |  |
 | proxy.extraEnvFrom | list | `[]` |  |
 | proxy.extraInitContainers | list | `[]` |  |
@@ -590,6 +608,8 @@ about defining resource limits.
 | proxy.headlessService.labels | object | `{}` |  |
 | proxy.headlessService.ports.http | int | `7200` |  |
 | proxy.headlessService.ports.rpc | int | `7300` |  |
+| proxy.initContainerResources | object | `{}` |  |
+| proxy.initContainerSecurityContext | object | `{}` |  |
 | proxy.labels | object | `{}` |  |
 | proxy.livenessProbe.httpGet.path | string | `"/proxy/health"` |  |
 | proxy.livenessProbe.httpGet.port | string | `"http"` |  |
@@ -600,14 +620,17 @@ about defining resource limits.
 | proxy.logging.logback.existingConfigmap | string | `""` |  |
 | proxy.nameOverride | string | `""` |  |
 | proxy.nodeSelector | object | `{}` |  |
+| proxy.persistence.annotations | object | `{}` |  |
 | proxy.persistence.enabled | bool | `true` |  |
+| proxy.persistence.labels | object | `{}` |  |
 | proxy.persistence.volumeClaimTemplateSpec.accessModes[0] | string | `"ReadWriteOnce"` |  |
 | proxy.persistence.volumeClaimTemplateSpec.resources.requests.storage | string | `"500Mi"` |  |
 | proxy.podAnnotations | object | `{}` |  |
 | proxy.podDisruptionBudget.enabled | bool | `false` |  |
 | proxy.podDisruptionBudget.maxUnavailable | string | `""` |  |
 | proxy.podDisruptionBudget.minAvailable | string | `"51%"` |  |
 | proxy.podLabels | object | `{}` |  |
+| proxy.podManagementPolicy | string | `"Parallel"` |  |
 | proxy.podSecurityContext | object | `{}` |  |
 | proxy.readinessProbe.httpGet.path | string | `"/proxy/ready"` |  |
 | proxy.readinessProbe.httpGet.port | string | `"http"` |  |
@@ -618,10 +641,17 @@ about defining resource limits.
 | proxy.resources.requests.cpu | string | `"100m"` |  |
 | proxy.resources.requests.memory | string | `"1500Mi"` |  |
 | proxy.revisionHistoryLimit | int | `10` |  |
+| proxy.schedulerName | string | `""` |  |
 | proxy.securityContext | object | `{}` |  |
 | proxy.service.annotations | object | `{}` |  |
 | proxy.service.enabled | bool | `true` |  |
+| proxy.service.externalIPs | list | `[]` |  |
+| proxy.service.externalTrafficPolicy | string | `""` |  |
+| proxy.service.extraPorts | list | `[]` |  |
+| proxy.service.healthCheckNodePort | string | `""` |  |
 | proxy.service.labels | object | `{}` |  |
+| proxy.service.loadBalancerClass | string | `""` |  |
+| proxy.service.loadBalancerSourceRanges | list | `[]` |  |
 | proxy.service.nodePort | string | `""` |  |
 | proxy.service.ports.http | int | `7200` |  |
 | proxy.service.type | string | `"ClusterIP"` |  |
@@ -633,6 +663,7 @@ about defining resource limits.
 | proxy.terminationGracePeriodSeconds | int | `30` |  |
 | proxy.tolerations | list | `[]` |  |
 | proxy.topologySpreadConstraints | list | `[]` |  |
+| proxy.updateStrategy.type | string | `"RollingUpdate"` |  |
 | readinessProbe.httpGet.path | string | `"/protocol"` |  |
 | readinessProbe.httpGet.port | string | `"http"` |  |
 | readinessProbe.initialDelaySeconds | int | `5` |  |
@@ -641,16 +672,25 @@ about defining resource limits.
 | replicas | int | `1` |  |
 | repositories.repositoriesConfigmap | string | `""` |  |
 | resources.limits.memory | string | `"2Gi"` |  |
-| resources.requests.cpu | float | `0.5` |  |
+| resources.requests.cpu | string | `"500m"` |  |
 | resources.requests.memory | string | `"2Gi"` |  |
 | revisionHistoryLimit | int | `10` |  |
+| schedulerName | string | `""` |  |
 | security.enabled | bool | `false` |  |
-| security.provisioningPassword | string | `"iHaveSuperpowers"` |  |
-| security.provisioningUsername | string | `"provisioner"` |  |
+| security.provisioner.existingSecret | string | `""` |  |
+| security.provisioner.password | string | `"iHaveSuperpowers"` |  |
+| security.provisioner.tokenKey | string | `"GRAPHDB_AUTH_TOKEN"` |  |
+| security.provisioner.username | string | `"provisioner"` |  |
 | securityContext | object | `{}` |  |
 | service.annotations | object | `{}` |  |
 | service.enabled | bool | `true` |  |
+| service.externalIPs | list | `[]` |  |
+| service.externalTrafficPolicy | string | `""` |  |
+| service.extraPorts | list | `[]` |  |
+| service.healthCheckNodePort | string | `""` |  |
 | service.labels | object | `{}` |  |
+| service.loadBalancerClass | string | `""` |  |
+| service.loadBalancerSourceRanges | list | `[]` |  |
 | service.nodePort | string | `""` |  |
 | service.ports.http | int | `7200` |  |
 | service.type | string | `"ClusterIP"` |  |
@@ -665,6 +705,7 @@ about defining resource limits.
 | terminationGracePeriodSeconds | int | `120` |  |
 | tolerations | list | `[]` |  |
 | topologySpreadConstraints | list | `[]` |  |
+| updateStrategy.type | string | `"RollingUpdate"` |  |
 
 ## Uninstall
 To remove the deployed GraphDB, use:

files/config/users.js

@@ -13,9 +13,9 @@
       },
       "dateCreated" : 1618403171751
     },
-    "{{ .Values.security.provisioningUsername }}" : {
-      "username" : "{{ .Values.security.provisioningUsername }}",
-      "password" : "{bcrypt}{{ htpasswd .Values.security.provisioningUsername .Values.security.provisioningPassword | trimPrefix (printf "%s:" .Values.security.provisioningUsername) }}",
+    "{{ .Values.security.provisioner.username }}" : {
+      "username" : "{{ .Values.security.provisioner.username }}",
+      "password" : "{bcrypt}{{ htpasswd .Values.security.provisioner.username .Values.security.provisioner.password | trimPrefix (printf "%s:" .Values.security.provisioner.username) }}",
       "grantedAuthorities" : [ "ROLE_ADMIN" ],
       "appSettings" : {
         "DEFAULT_INFERENCE" : true,

files/scripts/graphdb.sh

@@ -1,17 +1,18 @@
 #!/usr/bin/env bash
 
-set -eu
+set -o errexit
+set -o nounset
+set -o pipefail
 
 function createCluster {
   waitAllNodes $1
   local configLocation=$2
-  local authToken=$PROVISION_USER_AUTH_TOKEN
   local timeout=$3
 
   echo "Creating cluster"
-  curl -o response.json -isSL -m $timeout -X POST \
+  curl -o response.json -isSL -m "${timeout}" -X POST \
        -d @"$configLocation" \
-       --header "Authorization: Basic ${authToken}" \
+       --header "Authorization: Basic ${GRAPHDB_AUTH_TOKEN}" \
        --header 'Content-Type: application/json' \
        --header 'Accept: */*' \
        "http://${GRAPHDB_POD_NAME}-0.${GRAPHDB_SERVICE_NAME}:${GRAPHDB_SERVICE_PORT}/rest/cluster/config"
@@ -30,13 +31,12 @@ function createCluster {
 
 function waitService {
   local address=$1
-  local authToken=$PROVISION_USER_AUTH_TOKEN
 
   local attempt_counter=0
   local max_attempts=100
 
   echo "Waiting for ${address}"
-  until $(curl --output /dev/null -fsSL -m 5 -H "Authorization: Basic ${authToken}" --silent --fail ${address}); do
+  until curl --output /dev/null -fsSL -m 5 -H "Authorization: Basic ${GRAPHDB_AUTH_TOKEN}" --silent --fail "${address}"; do
     if [[ ${attempt_counter} -eq ${max_attempts} ]];then
       echo "Max attempts reached"
       exit 1
@@ -51,7 +51,7 @@ function waitService {
 function waitAllNodes {
   local node_count=$1
 
-  for (( c=$node_count; c>0; c ))
+  for (( c=node_count; c>0; c ))
   do
     c=$((c-1))
     waitService "http://${GRAPHDB_POD_NAME}-$c.${GRAPHDB_SERVICE_NAME}:${GRAPHDB_SERVICE_PORT}/rest/repositories"
@@ -61,19 +61,18 @@ function waitAllNodes {
 function createRepositoryFromFile {
   waitAllNodes $1
   local repositoriesConfigsLocation=$2
-  local authToken=$PROVISION_USER_AUTH_TOKEN
   local timeout=60
   local success=true
 
   echo "Creating repositories"
   for filename in ${repositoriesConfigsLocation}/*.ttl; do
-    repositoryName=$(grep "rep:repositoryID" $filename | sed -ne 's/rep:repositoryID "//p' | sed -ne 's/" ;//p' | sed -ne 's/^[[:space:]]*//p')
+    repositoryName=$(grep "rep:repositoryID" "${filename}" | sed -ne 's/rep:repositoryID "//p' | sed -ne 's/" ;//p' | sed -ne 's/^[[:space:]]*//p')
 
     echo "Provisioning repository ${repositoryName}"
     response=$(
       curl -X POST --connect-timeout 60 --retry 3 --retry-all-errors --retry-delay 10 \
-           -F config=@${filename} \
-           -H "Authorization: Basic ${authToken}" \
+           -F config=@"${filename}" \
+           -H "Authorization: Basic ${GRAPHDB_AUTH_TOKEN}" \
            -H 'Content-Type: multipart/form-data' \
            "http://${GRAPHDB_POD_NAME}-0.${GRAPHDB_SERVICE_NAME}:${GRAPHDB_SERVICE_PORT}/rest/repositories"
     )