Upgrade HTCondor deployment and configuration on Limpopo

[acjbrouwer]

Limpopo still mostly runs HTCondor 8.x. Base containers for creating a submit container are available for HTCondor 10.x. Connecting to Limpopo with an instance thereof does not work. The Limpopo HTCondor deployment needs to be upgraded.

[acjbrouwer]

Preliminary choices

• Deploy central manager (collector and negotiator daemons) on Kubernetes.
• Use HTCondor 23.0.x Long Term Support series.
• Use token-based authentication.
  • Not documented well, but Alexey managed to get it going on official htcondor containers under a Docker-on-Linux runtime. After excluding other auth methods from the configuration also works together with a Windows desktop submit node.

Targeted deployments

For reference see documentation section The Different Roles a Machine Can Play

• Central manager
  • Container on K8s.
  • With a logging sidecar container? (should be possible)
  • With a status page webserver? (probably, Alexey has something like this going for the existing pool)
• Execute Points
  • Windows Sever machines (limpopo1-6)
  • Windows Server VMs (limpopo7-8)
  • Windows desktops as execute points still? (probably not, no ECC, no controlled stack, and very little incremental capability relative to the new hardware)
  • Arbitrary stack Linux containers on K8s (limpopo7-8 and other machines).
• Access Points
  • Windows desktops
    • Todo: refresh I:\Smirnov\CONDOR_SETUP to HTCondor 23.0.x with updated configuration.
  • Container with a scientific stack (trialing installing HTCondor 23.0.x on an Ubuntu stack which gives access to many scientific stacks).
  • Submit-as-a-service container taking in snapshot bundles? (may not be necessary if the former point works out well)

Action points

Alexey:

• Create new setup installer+config+scripting for a Windows Access Point deployment:
  • With Condor 23.0.latest
  • With some README documentation on how to install and get going (e,g, request a token).

Alexey & Albert:

• Managed HTCondor configuration for the various deployment kinds.
  • Supporting IDTOKENS
  • Supporting remote HTCondor administration
  • For now not supporting RUN_AS_OWNER

Albert:

• Build custom containers for:
  • Central Manager
  • Linux Execute Point with scientific compute stack
  • Linux Access Point with interactive scientific compute stack.
• Reproducible Kubernetes orchestration for deploying, redeploying, upgrading the above.

[acjbrouwer]

Kubernetes orchestration for spinning up pods for a Condor Central Manager and one or more Execution Points and Submit nodes (command line access without a scientific stack) was elaborated. IDTOKENS authentication now works. Integration with IIASA user management, storage provisioning is as yet lacking.

The container building was automated (see #17 (comment) ) and pushes to the IIASA Harbor registry from where the images are pulled on deployment.

The HTCondor configuration files are Git pulled on pod start or re-start from per-subsystem (cm, execute, submit) branches of this configuration management repository: https://gitlab.iiasa.ac.at/ibiom/htcondor-configs