Cache package build base images more aggressively (#26756)

Allow the use of a cached base Docker image for package builds, _even
if_ the network connection is too poor to check if a newer version is
available.

Our packaging builds fail frequently, and one of the most common causes
is network timeouts while pulling base images. Though Docker is smart
about not pulling already-cached images, it still must fetch metadata to
determine if a newer image is available for the requested tag, and even
this will sometimes time out. So, work around it by:
1. Attempt to pull the image as a best-effort, but ignore failure to do
so.
2. Get the SHA256 repo-digest of the image, which is the mechanism
Docker uses to find changed images. Note repo-digest vs plain old digest
is important here as the former as repo-digest is per-tag, whereas
digest is per-architecture per-tag, and we sometimes do multi-arch
builds.
3. Reference the base image by tag _and_ digest in the Dockerfile to be
built, so Docker sees we have that exact image present locally without
needing to connect to a remote registry.

This workaround may become unnecessary or could be simplified in the
future, if Docker's (version of) BuildKit allows specifying not to check
for a newer image than the local copy; see
moby/buildkit#5340,
docker/buildx#1889, etc.

An alternate solution to this would be creating a local (non
pull-through) registry to cache all the images we want, using it for all
builds, and updating its copies of base images when possible. I didn't
go this route as we build on multiple machines and it would be unwieldy
to either 1) share one registry between them (especially if the network
changes in the future) or 2) create and update a separate registry
across multiple machines.

While here, also adjust some Dockerfiles to always `apt-get update`
before and on the same line as an `upgrade` or `install`, to avoid
[known caching
issues](https://docs.docker.com/build/building/best-practices/#apt-get),
which we've encountered in practice.

Resolves https://github.com/Cray/chapel-private/issues/6824.

[reviewed by @jabraham17 , thanks!]

- [x] To be merged with corresponding CI config changes PR.

Testing:
- [x] test version of the package build job succeeds

util/cron/create_release_aptrpm.bash

@@ -10,6 +10,7 @@
 #   PACKAGE_NAME: The name of the package to build. e.g. 'chapel'
 #   PACKAGE_VERSION: The version of the package to build, usually this is '1'
 #   DOCKER_DIR_NAME: The name of the directory in util/packaging/{PACKAGE_TYPE} that contains the Dockerfile to use.
+#   DOCKER_IMAGE_BASE: The name of the Docker image to use, including tag.
 #   PARALLEL: The number of cores to use for building the package. Default is 1.
 
 #
@@ -21,6 +22,7 @@ if [ -z "$OS" ]; then echo "OS must be set."; exit 1; fi
 if [ -z "$PACKAGE_NAME" ]; then echo "PACKAGE_NAME must be set."; exit 1; fi
 if [ -z "$PACKAGE_VERSION" ]; then echo "PACKAGE_VERSION must be set."; exit 1; fi
 if [ -z "$DOCKER_DIR_NAME" ]; then echo "DOCKER_DIR_NAME must be set."; exit 1; fi
+if [ -z "$DOCKER_IMAGE_BASE" ]; then echo "DOCKER_IMAGE_BASE must be set."; exit 1; fi
 PARALLEL=${PARALLEL:-1}
 
 
@@ -53,10 +55,10 @@ fi
 # if BUILD_CROSS_PLATFORM is set, build the cross-platform package
 if [ -n "$BUILD_CROSS_PLATFORM" ]; then
   log_info "Building cross-platform $PACKAGE_NAME $PACKAGE_TYPE package on $OS"
-  __build_all_packages $PACKAGE_TYPE $OS $PACKAGE_NAME $CHPL_VERSION $PACKAGE_VERSION $DOCKER_DIR_NAME $PARALLEL
+  __build_all_packages $PACKAGE_TYPE $OS $PACKAGE_NAME $CHPL_VERSION $PACKAGE_VERSION $DOCKER_DIR_NAME $DOCKER_IMAGE_BASE $PARALLEL
 else
   log_info "Building $PACKAGE_NAME $PACKAGE_TYPE package on $OS"
-  __build_native_package $PACKAGE_TYPE $OS $PACKAGE_NAME $CHPL_VERSION $PACKAGE_VERSION $DOCKER_DIR_NAME $PARALLEL
+  __build_native_package $PACKAGE_TYPE $OS $PACKAGE_NAME $CHPL_VERSION $PACKAGE_VERSION $DOCKER_DIR_NAME $DOCKER_IMAGE_BASE $PARALLEL
 fi
 
 log_info "Testing $PACKAGE_NAME"

util/packaging/apt/common/fill_docker_template.py

@@ -9,6 +9,14 @@ class MyTemplate(Template):
 
 
 substitutions = dict()
+
+substitutions[
+    "FROM"
+] = """
+ARG DOCKER_IMAGE_NAME_FULL=scratch
+FROM $DOCKER_IMAGE_NAME_FULL AS build
+"""
+
 substitutions[
     "ARGUMENTS"
 ] = """

util/packaging/apt/debian11-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM debian:bullseye AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/apt/debian11/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM debian:bullseye AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/apt/debian12-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM debian:bookworm AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/apt/debian12/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM debian:bookworm AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/apt/test/Dockerfile.template

@@ -13,7 +13,7 @@ RUN apt-get update && \
 COPY --chown=user @@{HOST_PACKAGE_PATH}/@@{PACKAGE_NAME} /home/user/@@{PACKAGE_NAME}
 
 USER root
-RUN apt-get install -y ./@@{PACKAGE_NAME}
+RUN apt-get update && apt-get install -y ./@@{PACKAGE_NAME}
 USER user
 WORKDIR /home/user
 

util/packaging/apt/ubuntu20/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:20.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu22-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:22.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu22-ofi-slurm/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:22.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu22/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:22.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu24-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:24.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu24-ofi-slurm/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:24.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/apt/ubuntu24/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM ubuntu:24.04 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 ARG DEBIAN_FRONTEND=noninteractive

util/packaging/common/build_helpers.sh

@@ -22,16 +22,16 @@ __wget_chpl_release() {
 }
 
 __build_all_packages() {
-  __build_packages $1 $2 $3 $4 $5 $6 '--platform=linux/amd64,linux/arm64' $7
+  __build_packages $1 $2 $3 $4 $5 $6 $7 '--platform=linux/amd64,linux/arm64' $8
 }
 __build_x8664_package() {
-  __build_packages $1 $2 $3 $4 $5 $6 '--platform=linux/amd64' $7
+  __build_packages $1 $2 $3 $4 $5 $6 $7 '--platform=linux/amd64' $8
 }
 __build_arm64_package() {
-  __build_packages $1 $2 $3 $4 $5 $6 '--platform=linux/arm64' $7
+  __build_packages $1 $2 $3 $4 $5 $6 $7 '--platform=linux/arm64' $8
 }
 __build_native_package() {
-  __build_packages $1 $2 $3 $4 $5 $6 '' $7
+  __build_packages $1 $2 $3 $4 $5 $6 $7 '' $8
 }
 
 __build_packages() {
@@ -41,10 +41,11 @@ __build_packages() {
   local chapel_version=$4
   local package_version=$5
   local docker_dir_name=$6
-  local architecture_string=$7
+  local docker_image_base=$7
+  local architecture_string=$8
 
   # default to 1 core
-  local para=${8:-1}
+  local para=${9:-1}
 
   __wget_chpl_release $chapel_version
 
@@ -54,6 +55,18 @@ __build_packages() {
 
   pushd ${package_dir}
 
+  # Try to pull the latest version of this image, as a best effort.
+  docker pull $docker_image_base || true
+  # Whether we pulled successfully or not, use the image SHA256 digest to
+  # identify the copy of it we have locally (if present), so build can proceed
+  # with that version.
+  if [ -z "$(docker image ls -q $docker_image_base)" ]; then
+    echo "Error: Failed to pull image $docker_image_base and a copy does not exist locally"
+    exit 1
+  fi
+  local docker_image_name_full="$(docker inspect --format='{{index .RepoDigests 0}}' $docker_image_base)"
+  echo "Using image digest ${docker_image_name_full} for $docker_image_base"
+
   # if there is a template file, use it to generate the Dockerfile
   if [ -f Dockerfile.template ]; then
     ${fill_script} Dockerfile.template
@@ -69,6 +82,7 @@ __build_packages() {
     --build-arg "PACKAGE_VERSION=$package_version" \
     --build-arg "OS_NAME=$os" \
     --build-arg "DOCKER_DIR_NAME=$docker_dir_name" \
+    --build-arg "DOCKER_IMAGE_NAME_FULL=$docker_image_name_full" \
     --build-arg "PARALLEL=$para" \
     -t $__docker_tag \
     -f Dockerfile ../..
@@ -83,6 +97,7 @@ __build_image() {
   local chapel_version=$4
   local package_version=$5
   local docker_dir_name=$6
+  local docker_image_base=$7
 
   # default to 1 core
   local para=${7:-1}
@@ -95,6 +110,18 @@ __build_image() {
 
   pushd ${package_dir}
 
+  # Try to pull the latest version of this image, as a best effort.
+  docker pull $docker_image_base || true
+  # Whether we pulled successfully or not, use the image SHA256 digest to
+  # identify the copy of it we have locally (if present), so build can proceed
+  # with that version.
+  if [ -z "$(docker image ls -q $docker_image_base)" ]; then
+    echo "Error: Failed to pull image $docker_image_base and a copy does not exist locally"
+    exit 1
+  fi
+  local docker_image_name_full="$(docker inspect --format='{{index .RepoDigests 0}}' $docker_image_base)"
+  echo "Using image digest ${docker_image_name_full} for $docker_image_base"
+
   # if there is a template file, use it to generate the Dockerfile
   if [ -f Dockerfile.template ]; then
     ${fill_script} Dockerfile.template
@@ -108,6 +135,7 @@ __build_image() {
     --build-arg "PACKAGE_VERSION=$package_version" \
     --build-arg "OS_NAME=$os" \
     --build-arg "DOCKER_DIR_NAME=$docker_dir_name" \
+    --build-arg "DOCKER_IMAGE_NAME_FULL=$docker_image_name_full" \
     --build-arg "PARALLEL=$para" \
     -t $__docker_tag \
     -f Dockerfile ../..

util/packaging/common/test_package.py

@@ -23,7 +23,7 @@ class MyTemplate(Template):
 def run_command(cmd, **kwargs):
     if verbose:
         print(f"Running command: \"{' '.join(cmd)}\"")
-    return sp.check_call(cmd, **kwargs)
+    return sp.check_output(cmd, **kwargs)
 
 def determine_arch(package):
     # if the arch is aarch64 or arm64, return arm64
@@ -45,8 +45,8 @@ def infer_docker_os(package):
         "amzn2023": "amazonlinux:2023",
         "ubuntu22": "ubuntu:22.04",
         "ubuntu24": "ubuntu:24.04",
-        "debian11": "debian:bullseye",
-        "debian12": "debian:bookworm",
+        "debian11": "debian:11",
+        "debian12": "debian:12",
     }
     for tag, docker in os_tag_to_docker.items():
         if ".{}.".format(tag) in package:
@@ -60,6 +60,16 @@ def infer_docker_os(package):
 
     return ValueError(f"Could not infer docker image from package {package}")
 
+
+def add_digest_to_image(docker_os):
+    cmd = [
+        "docker",
+        "inspect",
+        "--format='{{index .RepoDigests 0}}'",
+        docker_os
+    ]
+    return run_command(cmd).decode("utf-8").strip()
+
 def infer_env_vars(package):
     if "gasnet-udp" in package:
         return """
@@ -190,6 +200,8 @@ def main():
     if docker_os is None:
         docker_os = infer_docker_os(package)
 
+    docker_os = add_digest_to_image(docker_os)
+
     imagetag = docker_build_image(test_dir, package, docker_os)
     if args.run:
         docker_run_container(imagetag)

util/packaging/docker/test/Dockerfile

@@ -11,8 +11,7 @@ ENV HOMEBREW_DEVELOPER=1
 
 RUN mkdir -p /home/linuxbrew
 # only really needed for interactive debugging
-# RUN sudo apt-get update
-# RUN sudo apt-get install -y vim
+# RUN sudo apt-get update && sudo apt-get install -y vim
 # ENV EDITOR=vim
 
 # Do the updates before applying our chapel.rb changes

util/packaging/rpm/amzn2023-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM amazonlinux:2023 as build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/amzn2023-ofi-slurm/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM amazonlinux:2023 as build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/amzn2023/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM amazonlinux:2023 as build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/common/fill_docker_template.py

@@ -9,6 +9,14 @@ class MyTemplate(Template):
 
 
 substitutions = dict()
+
+substitutions[
+    "FROM"
+] = """
+ARG DOCKER_IMAGE_NAME_FULL=scratch
+FROM $DOCKER_IMAGE_NAME_FULL AS build
+"""
+
 substitutions[
     "ARGUMENTS"
 ] = """

util/packaging/rpm/el9-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM rockylinux/rockylinux:9 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/el9-ofi-slurm/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM rockylinux/rockylinux:9 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/el9/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM rockylinux/rockylinux:9 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/fc40-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM fedora:40 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/fc40/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM fedora:40 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/fc41-gasnet-udp/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM fedora:41 AS build
+@@{FROM}
 
 @@{ARGUMENTS}
 

util/packaging/rpm/fc41/Dockerfile.template

@@ -1,4 +1,4 @@
-FROM fedora:41 AS build
+@@{FROM}
 
 @@{ARGUMENTS}