Improve documentation for the HPA example

camptocamp · Oct 16, 2024 · 6009581 · 6009581
1 parent f5a005c
commit 6009581
Show file tree

Hide file tree

Showing 2 changed files with 151 additions and 42 deletions.
diff --git a/examples/pgconfig-wms-hpa/README.md b/examples/pgconfig-wms-hpa/README.md
@@ -1,42 +1,43 @@
-# HPA example
+# HPA Example
 
-This example allows to create a very simple GeoserverCloud deployment in your local cluster, which allows to evaluate and understand how Horizontal Pod Autoscaling (HPA) works.
+This example demonstrates how to deploy a simple **GeoserverCloud** setup in your local cluster, allowing you to evaluate and understand the behavior of **Horizontal Pod Autoscaling (HPA)**.
 
-The setup includes:
+## Setup Overview
 
-- a unique WebUI instance, (to allow see catalog configuration)
-- the gateway, (access to the GeoserverCloud solution)
-- 2 WMS instances, (the initial 2 instances which serve the WMS OGC protocol)
-- a REST API instance, (used by attached script that allows to create a minimal catalog for testing)
-- a local Postgres and (used along with PgConfig profile, since we want to minimize startup time, so we focus on reducing catalog reading)
-- RabbitMQ (the bus event communication across instances)
+The deployment includes the following components:
 
-Following steps mentioned in next section you will be able to see how HPA works automatically (up to 100 containers created!), when the cluster is stressed by the attached script defined for that.
+- **WebUI instance**: Provides access to the catalog configuration.
+- **Gateway**: Serves as the entry point to the GeoserverCloud solution.
+- **Two WMS instances**: These initial instances serve the WMS OGC protocol.
+- **REST API instance**: Utilized by a script to create a minimal catalog for testing.
+- **Local PostgreSQL**: Used with the PgConfig profile to minimize startup time by focusing on efficient catalog reads.
+- **RabbitMQ**: Manages event communication across instances.
 
-# Considerations
+By following the steps in the next section, you will observe how HPA automatically scales the deployment (up to 100 containers!) in response to stress generated by the provided script.
 
-- Read documentation in ../README.md file (since for running this demo, it's required to have a local cluster installed, along with kubectl)
-- Execution of this kind of tests could freeze your machine, if your hardware/setup is not adequate.
-- If in doubt, change values in the HPA section on the values.yaml file, so you can determine which is the value for maxReplicas (by default = 100)
+## Considerations
 
-# Steps
+- Make sure to review the documentation in the `../README.md` file. A local Kubernetes cluster and `kubectl` are required to run this demo.
+- Be aware that running these tests may overload or freeze your machine if your hardware or configuration is inadequate.
+- If needed, adjust the **maxReplicas** value in the `values.yaml` file under the HPA section. The default value is set to 100.
 
-At repository base folder level, follow next steps:
+## Steps to Run the Example
 
-1. Execute
+At the root level of the repository, follow these steps:
+
+### 1. Execute the example command:
 
 ```shell
-    make example-wms-hpa
+make example-pgconfig-wms-hpa
 ```
 
-2. Use
+### 2. Verify that all pods are running:
 
 ```shell
-   kubectl get po
+kubectl get pods
 ```
 
-in order to check that all the pods are up and running (that is, all of them with values STATUS = Running and READY = 1/1)
-ie.
+Ensure all pods have the status `STATUS = Running` and `READY = 1/1`. Example output:
 
 ```shell
 NAME                                                     READY   STATUS    RESTARTS   AGE
@@ -49,40 +50,153 @@ gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-qs946       1/1     Running   0
 gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-pth59       1/1     Running   0          10m
 ```
 
-3. Define a DNS alias (used in scripts to avoid local references. Note: you can edit scripts if you prefer)
+### 3. Define a DNS alias:
+
+This step prevents the use of local references in the scripts creating an entry like the following in `/etc/hosts`
+
+```shell
+192.168.208.2	gscloud.local
+```
+
+You can modify the scripts if needed.
+
+```shell
+kubectl get ingress --no-headers gs-cloud-pgconfig-wm-geoserver-host1 | \
+awk '{printf("%s\t%s\n", $4, $3)}' | sudo tee -a /etc/hosts
+```
+
+
+### 4. Initialize the catalog:
+
+Run the following command to create a test layer:
+
+```shell
+/examples/pgconfig-wms-hpa/init-catalog.sh
+```
+Should output:
+
+```shell
+------------------------------------
+Preparing initialization process ...
+------------------------------------
+Creating workspace 'hpa-test'...
+hpa-test
+------------------------------------
+Creating WMS datastore 'swisstopo_wms'...
+swisstopo_wms
+------------------------------------
+Publishing layer 'ch.bafu.grundwasserkoerper' from datastore 'swisstopo_wms'...
+ch.bafu.grundwasserkoerper
+------------------------------------
+Catalog initialized successfully.
+```
+
+### 5. Monitor the pods:
+
+Use the following command to continuously monitor the pod status (refreshes every second).
+This will let you see the number of `wms` pods increase during stress testing, and decrease back to one pod after a while.
+
+```shell
+watch -n 1 kubectl top pod -l app.kubernetes.io/component=wms --sort-by cpu
+```
+
+Initially it will output something like this, with the only wms pod running by default:
 
 ```shell
-   kubectl get ingress --no-headers  gs-cloud-pgconfig-wm-geoserver-host1 | awk '{printf("%s\t%s\n",$4,$3 )}' | sudo tee -a /etc/hosts
+Every 1.0s: kubectl top pod -l app.kubernetes.io/component=wms --sort-by cpu
+
+NAME                                                 CPU(cores)   MEMORY(bytes)
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-2frh9   3m           533Mi
 ```
 
-3. Execute
+The `kubectl get hpa` command is also useful to monitor
 
 ```shell
-   ./examples/pgconfig-wms-hpa/init-catalog.sh
+watch -n 1 kubectl get hpa gs-cloud-pgconfig-wms-hpa-gsc-wms
+
+Every 1.0s: kubectl get hpa gs-cloud-pgconfig-wms-hpa-gsc-wms
+NAME                                REFERENCE                                      TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
+gs-cloud-pgconfig-wms-hpa-gsc-wms   Deployment/gs-cloud-pgconfig-wms-hpa-gsc-wms   3%/70%    1         16        16         23m
 ```
 
-4. Execute
+### 6. Stress test the server:
+
+> [!NOTE]
+> The `stress-server.sh` script uses Apache Benchmark (`ab` command). If you don't have it, install it with
+> ```shell
+> sudo apt-get install apache2-utils
+> ```
+> or with your Operating System's package manager.
+
+In a new terminal window, run the stress test script. It'll run for 60 seconds making 100 concurrent WMS requests to the cluster:
 
 ```shell
-   watch kubectl get po
+./examples/pgconfig-wms-hpa/stress-server.sh
 ```
 
-in order to see list of pods (and watch every 2 secs)
+You will observe in the first terminal how the number of pods scales up and down dynamically during the stress test.
 
-5. In a different console, execute
+For example, at some point the number of `wms` pods scaled to 4 here as the CPU utilization increased:
 
 ```shell
-   ./examples/pgconfig-wms-hpa/stress-server.sh
+Every 1.0s: kubectl top pod -l app.kubernetes.io/component=wms --sort-by cpu
+
+NAME                                                 CPU(cores)   MEMORY(bytes)
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-h8nmg   1888m        818Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-xvr2r   1532m        867Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-9kp2g   1318m        845Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-nb2cc   1305m        891Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-9hhlb   1285m        882Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-zpk4d   1267m        860Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-wmdxw   1102m        1398Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-lrlv8   1027m        834Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-6nv2j   968m         826Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-wmvjh   901m         821Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-dtx4k   865m         1301Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-9s4fk   584m         1447Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-gxbj4   460m         985Mi
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-2frh9   367m         953Mi
 ```
 
-so you will trigger 1000 simultaneous request to the cluster.
+And with `kubectl get hpa` will observe the status of the **HorizontalPodAutoscaler**, similar to:
+
+```shell
+watch -n 1 kubectl get hpa gs-cloud-pgconfig-wms-hpa-gsc-wms
 
-At this point you will be able to see (in console 1) how the list of pods are increased and decreased during execution of the stress-server.sh script.
+Every 1.0s: kubectl get hpa gs-cloud-pgconfig-wms-hpa-gsc-wms
+
+NAME                                REFERENCE                                      TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
+gs-cloud-pgconfig-wms-hpa-gsc-wms   Deployment/gs-cloud-pgconfig-wms-hpa-gsc-wms   941%/70%   1         16        14         4m28s
+```
 
-6. Execute
+Here, CPU consumption has increased to 941% out of the 70% average utilization configured. As a result, the Deployment was resized to 14 replicas:
 
 ```shell
-   make examples-clean
+kubectl get deployment gs-cloud-pgconfig-wms-hpa-gsc-wms
+NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
+gs-cloud-pgconfig-wms-hpa-gsc-wms   14/14   14           14          9m23s
 ```
 
-so you get your environment clean, and deployment down.
+And after a couple minutes the replicas start to ramp down until reaching one:
+
+```shell
+kubectl get deployment gs-cloud-pgconfig-wms-hpa-gsc-wms
+NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
+gs-cloud-pgconfig-wms-hpa-gsc-wms   1/1     1            1           9m25s
+
+kubectl top pod -l app.kubernetes.io/component=wms --sort-by cpu
+NAME                                                 CPU(cores)   MEMORY(bytes)
+gs-cloud-pgconfig-wms-hpa-gsc-wms-758dfd8765-2frh9   4m           971Mi
+
+kubectl get hpa gs-cloud-pgconfig-wms-hpa-gsc-wms
+NAME                                REFERENCE                                      TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
+gs-cloud-pgconfig-wms-hpa-gsc-wms   Deployment/gs-cloud-pgconfig-wms-hpa-gsc-wms   3%/70%    1         16        1          15m
+```
+
+### 7. Clean up the environment:
+
+To remove the deployment and restore your environment, run:
+
+```shell
+make examples-clean
+```
diff --git a/examples/pgconfig-wms-hpa/stress-server.sh b/examples/pgconfig-wms-hpa/stress-server.sh
@@ -2,9 +2,4 @@
 
 WMS_URL="http://gscloud.local/geoserver-cloud/wms?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&STYLES&LAYERS=hpa-test%3Ach.bafu.grundwasserkoerper&exceptions=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A2056&WIDTH=769&HEIGHT=359&BBOX=2628297.2396917907%2C1161127.5666655225%2C2745623.985655881%2C1215846.1146757442"
 
-for _ in {1..1000}
-do
-  curl -s "$WMS_URL" > /dev/null &
-done
-
-wait
+ab -Sdl -t 120 -c 64 "$WMS_URL"