Add support for hook to customise compute and management nodes

[christopheredsall]

The current design optimises compute node startup time by installing a limited set of packages on top of the provided OS image. The recommendation to users is to install software to the shared filesystem /mnt/shared possibly using something like EasyBuild (#20) or Spack. This approach ensures that the software is build with modern compilers to take best advantage of the architectures available.

One disadvantage is the compiling everything from source is time consuming and error prone. For some packages which aren't performance critical yum installing them could be a user experience improvement.

I propose adding in hooks so that extra packages could be added with the OS package manger and user supplied scripts could be run.

Use cases

• Installing things provided by the OS package manger or third party repositories that are difficult to build by hand
• Making changes on a compute node (e.g. setting up permissions to write on local filesystems, or adding a different parallel filesystem as scratch)

Questions

• Should we do this?
• How could we implement this?
• How do we ensure the node doesn't check-in with the Slurm master until the user customisation script has finished?
• Names are important - are these the right names for the concepts?

[christopheredsall]

Implementation ideas

• model this on the exisiting single ansible playbook with [mgmt] and [compute] node groups
• define a well known location on the management node where, if this playbook was found it would be run