Add scalability setup with fake nodes using Fake IPA and FKAS

Signed-off-by: Huy Mai <huy.mai@est.tech>

.yamllint.yaml

@@ -8,3 +8,7 @@ rules:
   key-duplicates: enable
   truthy:
     allowed-values: ['true', 'false', 'yes', 'no', 'on']
+
+ignore: |
+  Support/Multitenancy/Multiple-Ironic-conductors/ironic/templates/**/*.yaml
+  Support/Multitenancy/Multiple-Ironic-conductors/cluster-template.yaml

Support/Multitenancy/Multiple-Ironic-conductors/.gitignore

@@ -0,0 +1,24 @@
+*.json
+*.log
+ironicclient.sh
+_clouds_yaml/*
+kubectl
+minikube-linux-amd64
+macaddrs
+uuids
+sushy-tools-conf/*
+bmc-*.yaml
+ironic.env
+ironic_logs/*
+track.csv
+Metal3/*
+*/ironic-username
+*/ironic-password
+*/ironic-inspector-username
+*/ironic-inspector-password
+*-tpl
+ironic/ironic-auth-config
+ironic/ironic-htpasswd
+ironic/ironic-inspector-auth-config
+ironic/ironic-inspector-htpasswd
+cert/*

Support/Multitenancy/Multiple-Ironic-conductors/Init-environment.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+set -e
+trap 'trap - SIGTERM && kill -- -'$$'' SIGINT SIGTERM EXIT
+# shellcheck disable=SC1091
+. ./config.sh
+__dir__=$(realpath "$(dirname "$0")")
+./start_image_server.sh
+./minikube-setup.sh
+./run-ipa-downloader.sh
+./handle-images.sh
+./install-fkas.sh
+./install-ironic.sh
+./install-bmo.sh
+./generate_unique_nodes.sh
+python create-bmhs.py
+./start_containers.sh
+./apply-bmhs.sh
+
+./clusterctl-init.sh
+
+./create-clusters.sh

Support/Multitenancy/Multiple-Ironic-conductors/README.md

@@ -0,0 +1,79 @@
+# Metal3 Scalability with Fake Nodes and Multiple Ironics
+
+This directory stores the code and configuration for the __Metal3 Scalability with
+Fake Nodes and Multiple Ironics__. Its purpose is to have a way of testing
+[Metal3](https://metal3.io) provisioning on multiple nodes, without needing
+too much hardware resource.
+
+## Requirements
+
+To perform this experiment, you will need:
+
+- A fresh Ubuntu machine with passwordless sudo.
+- `docker` is installed and can be ran without sudo.
+- Some tools, of which most/all could be installed with the script `vm-setup.sh`.
+
+## Quick Start
+
+For a quick start, here are the steps you need:
+
+- Install some additional tools, add libvirt network
+(named `baremetal`) by using `./vm-setup.sh`.
+- Edit the [config.sh](config.sh) file with suitable configuration. We recommend
+starting with small amount of nodes and clusters, just so you know the
+setup works. Provisioning large numbers of nodes (500-1000) was tested
+and worked well, but that will require large machine and longer time to proceed.
+
+For easy comparison, here is the spec of the machine we used to provision 1000
+single-node clusters:
+
+- CPUs: 20c
+- RAM: 64Gb
+- Hard disk: 750Gb
+
+The whole process took around 48h to finish.
+
+- Run the whole process with `./Init-environment.sh`
+
+## More detailed explanation
+
+In this part, we will explain the purpose of steps taken in `Init-environment.sh`:
+
+- `start_image_server.sh`: starts a nginx server running locally on port 8080. This
+server will be used to store the images that ironic uses for provisioning the nodes.
+- `minikube-setup.sh`: This sets up minikube with the configured amounts of CPUs
+and Memory; then attachs the `baremetal` libvirt network to it.
+- `run-ipa-downloader.sh`: This runs the `ipa-downloader` and gets an ipa image to
+the image server. Since we use fake nodes, what images to use do not matter,
+hence step is not crucial.
+- `handle-images.sh`: This script downloads container images and inject into the
+minikube VM. Might help speed up the process a little, if we run this setup
+multiple times.
+- `install-fkas`: This script installs [Metal3-FKAS](https://github.com/metal3-io/cluster-api-provider-metal3/tree/main/hack/fake-apiserver)
+Deployment with the number of replicas specified in `config.sh`.
+Metal3-Fkas will be responsible of generating API servers
+and other kubernetes components for the newly provisioned clusters.
+- `install-ironic`: This script helps installing ironic with the specified number
+of `ironic-conductors`.
+- `install-bmo`: This script installs [Baremetal Operator](https://book.metal3.io/bmo/introduction),
+or BMO for short.
+- `generate_unique_nodes.sh`: This script helps generating the configuration of
+the fake nodes.
+- `create-bmhs.py`: This script generate the BMH manifests based on the fake nodes
+configuration.
+- `start_containers.sh`: This script starts up `sushy-tools` and `fake-ipa` containers.
+Each of them has the number of replicas as specified in `N_FAKE_IPAS` in the config.
+- `apply-bmhs.sh`: This script applies all the BMH manifests into `metal3` namespace.
+After this step, you should be able to see that all the BMHs exist and eventually
+change their states to `available`. Note that the workload clusters will be in the
+same namespace as the BMH objects, so if you want your clusters to be in another
+namespace, you need to change this step.
+- `clusterctl-init.sh`: This script only runs normal `clusterctl init` with `metal3`
+as the infra provider. The result of this step is that we will have CAPI, CAPM3
+and IPAM installed in the bootstrap cluster.
+- `create-clusters.sh`: This step generate the new clusters. The number of clusters,
+as well as how many CP and worker nodes they have, is in the config, too.
+
+## Cleanup
+
+Running `./clean.sh` should help removing most of what created by `Init-environment.sh`

Support/Multitenancy/Multiple-Ironic-conductors/apply-bmhs.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+# shellcheck disable=SC1091
+. ./config.sh
+
+namespace=${1:-"metal3"}
+
+kubectl create ns "${namespace}"
+
+for f in bmc-*.yaml; do
+  kubectl -n "${namespace}" apply -f "$f"
+done
+
+# Wait for the BMHs to be in available state
+check_bmh_status() {
+  local namespace=$1
+  # Get the list of BMH objects and their provisioning states
+  local states
+  states=$(kubectl get bmh -n "$namespace" -o jsonpath='{range .items[*]}{.metadata.name}:{.status.provisioning.state}{"\n"}{end}')
+
+  # Loop through each BMH and check its state
+  local all_available=true
+  while IFS= read -r line; do
+    local name
+    local state
+    name=$(echo "$line" | cut -d':' -f1)
+    state=$(echo "$line" | cut -d':' -f2)
+
+    if [ "$state" != "available" ]; then
+      all_available=false
+      echo "BMH $name is not available (current state: $state)"
+    fi
+  done <<<"$states"
+
+  echo $all_available
+}
+
+echo "Waiting for all BMH objects to become 'available'..."
+while true; do
+  all_available=$(check_bmh_status "${namespace}")
+
+  if [ "$all_available" == "true" ]; then
+    echo "All BMH objects are in 'available' state."
+    break
+  fi
+
+  sleep 60
+done

Support/Multitenancy/Multiple-Ironic-conductors/bmo-config/ironic.env

@@ -0,0 +1,9 @@
+HTTP_PORT=6180
+PROVISIONING_INTERFACE=ironicendpoint
+DHCP_RANGE=192.168.222.100,192.168.222.200
+DEPLOY_KERNEL_URL=http://192.168.222.100:6180/images/ironic-python-agent.kernel
+DEPLOY_RAMDISK_URL=http://192.168.222.100:6180/images/ironic-python-agent.initramfs
+IRONIC_ENDPOINT=https://192.168.222.100:6385/v1/
+IRONIC_INSPECTOR_ENDPOINT=https://192.168.222.100:5050/v1/
+CACHEURL=http://192.168.222.100/images
+IRONIC_FAST_TRACK=true

Support/Multitenancy/Multiple-Ironic-conductors/bmo-config/kustomization.yaml

@@ -0,0 +1,36 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: baremetal-operator-system
+resources:
+- https://github.com/metal3-io/baremetal-operator/config/overlays/basic-auth_tls?ref=release-0.6
+
+configMapGenerator:
+- name: ironic
+  behavior: create
+  envs:
+  - ironic.env
+
+patches:
+- patch: |
+    # Don't try to pull again the pre-loaded image
+    - op: replace
+      path: /spec/template/spec/containers/0/imagePullPolicy
+      value: IfNotPresent
+  target:
+    kind: Deployment
+    name: controller-manager
+
+images:
+- name: quay.io/metal3-io/baremetal-operator
+  newTag: release-0.5
+
+# We cannot use suffix hashes since the kustomizations we build on
+# cannot be aware of what suffixes we add.
+generatorOptions:
+  disableNameSuffixHash: true
+
+secretGenerator:
+  - name: ironic-credentials
+    files:
+    - username=ironic-username
+    - password=ironic-password

Support/Multitenancy/Multiple-Ironic-conductors/build-sushy-tools-image.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+IMAGE_NAME="127.0.0.1:5000/localimages/sushy-tools"
+SUSHYTOOLS_DIR="/tmp/sushy-tools"
+
+if [[ ${1:-""} == "-f" ]]; then
+  rm -rf "${SUSHYTOOLS_DIR}"
+  docker rmi "${IMAGE_NAME}"
+fi
+
+if [[ $(docker images | grep ${IMAGE_NAME}) != "" ]]; then
+  exit 0
+fi
+if [[ ! -d "${SUSHYTOOLS_DIR}" ]]; then
+  git clone https://opendev.org/openstack/sushy-tools.git "$SUSHYTOOLS_DIR"
+fi
+cd "$SUSHYTOOLS_DIR" || exit
+
+pip3 install build
+python3 -m build
+
+cd dist || exit
+WHEEL_FILENAME="*.whl"
+echo "$WHEEL_FILENAME"
+
+cd ..
+
+cat <<EOF >"${SUSHYTOOLS_DIR}/Dockerfile"
+# Use the official ubuntu image as the base image
+FROM ubuntu:22.04
+
+# Install necessary packages
+RUN apt-get update -y && \
+    apt-get install -y python3 python3-pip python3-venv && \
+    apt-get clean all
+
+WORKDIR /opt
+
+# RUN python3 setup.py install
+
+# Copy the application code to the container
+COPY "dist/${WHEEL_FILENAME}" .
+
+RUN pip3 install ${WHEEL_FILENAME}
+
+ENV FLASK_DEBUG=1
+
+RUN mkdir -p /root/sushy
+
+# Set the default command to run when starting the container
+CMD ["sushy-emulator", "-i", "::", "--config", "/root/sushy/conf.py"]
+EOF
+
+docker build -t "${IMAGE_NAME}" .

Support/Multitenancy/Multiple-Ironic-conductors/clean.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# Disable and delete bridge interfaces
+iface=baremetal
+if ip link show $iface &>/dev/null; then
+  sudo ip link set $iface down
+  sudo brctl delbr $iface
+fi
+
+# Delete baremetal network
+net="baremetal"
+if virsh -c qemu:///system net-info $net &>/dev/null; then
+  virsh -c qemu:///system net-destroy $net
+  virsh -c qemu:///system net-undefine $net
+fi
+
+# Delete directories
+sudo rm -rf /opt/metal3-dev-env
+sudo rm -rf "$(dirname "$0")/_clouds_yaml"
+
+# Stop and delete minikube cluster
+minikube stop
+minikube delete --all --purge
+
+# Stop and delete containers
+mapfile -t existing_containers < <(docker ps --all --format json | jq -r '.Names')
+declare -a containers=("ipa-downloader" "ironic" "keepalived" "registry" "ironic-client" "openstack-client" "httpd-infra" "image-server")
+
+for container in "${existing_containers[@]}"; do
+  # shellcheck disable=SC2199
+  if [[ "${containers[@]}" =~ $container || "${container}" =~ "sushy-tools-"* || "${container}" =~ "fake-ipa-"* ]]; then
+    echo "Deleting the container: ${container}"
+    docker rm -f "$container" &>/dev/null
+  fi
+done
+
+rm -rf bmc-*.yaml
+
+rm -rf macaddrs uuids node.json nodes.json batch.json in-memory-development.yaml sushy-tools-conf
+
+rm -rf cert

Support/Multitenancy/Multiple-Ironic-conductors/clusterctl-init.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+export CLUSTER_TOPOLOGY=true
+clusterctl init --infrastructure=metal3
+kubectl -n capi-system wait deploy capi-controller-manager --for=condition=available --timeout=600s
+kubectl -n capm3-system wait deploy capm3-controller-manager --for=condition=available --timeout=600s
+kubectl -n capm3-system wait deploy ipam-controller-manager --for=condition=available --timeout=600s

Support/Multitenancy/Multiple-Ironic-conductors/config.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+export N_NODES=20
+export N_FAKE_IPAS=2
+export N_IRONICS=2
+export N_FKAS=1
+
+# Minikube configuration
+export MINIKUBE_CPUS=4
+export MINIKUBE_MEMORY=10000
+
+# Note: N_CLUSTERS is the number of clusters to provision,
+# while CP_NODE_COUNT and WORKER_NODE_COUNT are the number of
+# CP nodes and worker nodes in each of these clusters.
+# The user is responsible to make sure that
+# N_CLUSTERS*(CP_NODE_COUNT+WORKER_NODE_COUNT) <= N_NODES
+export N_CLUSTERS=10
+export CP_NODE_COUNT=1
+export WORKER_NODE_COUNT=1
+
+# Translating N_IRONICS to IRONIC_ENDPOINTS. Don't change this part
+IRONIC_ENDPOINTS="192.168.222.100"
+for i in $(seq 2 $N_IRONICS); do
+  IRONIC_ENDPOINTS="${IRONIC_ENDPOINTS} 192.168.222.$((100 + i))"
+done
+export IRONIC_ENDPOINTS