add-test-linux_rootfs_propagation #3024

saku3 · 2024-12-15T12:36:41Z

This implements the validation linux_rootfs_propagation in #361

saku3 · 2024-12-16T08:32:22Z

I'm not sure if this is related, but this issue might be relevant:
opencontainers/runc#1755

YJDoc2 · 2025-01-02T05:11:30Z

Hey, thanks for the PR! I will try to take a look soon. In the meantime, can you rebase on main and fix the conflicts? Thanks!

Signed-off-by: Yusuke Sakurai <yusuke.sakurai@3-shake.com>

utam0k · 2025-01-06T02:37:27Z

It appears not to pass CI. May I ask you to check it?

saku3 · 2025-01-06T10:40:48Z

@utam0k @YJDoc2

When I run just validate-contest-runc locally, it fails in the same way as in the CI.(rootfs_propagation_shared_test)

# Start group rootfs_propagation
1 / 4 : rootfs_propagation_private_test : ok
2 / 4 : rootfs_propagation_shared_test : not ok
        container stderr was not empty, found : Error: shared root propagation failed to expose "/tmp/target9AGYIG/tmp/mountJk8Iig/example"

3 / 4 : rootfs_propagation_slave_test : ok
4 / 4 : rootfs_propagation_unbindable_test : ok
# End group rootfs_propagation

I also checked the following issue, but I'm not sure if it's related:
opencontainers/runc#1755

I will continue investigating this on my side, but I would appreciate any advice you could provide.

utam0k · 2025-01-07T07:04:29Z

@saku3 Let me make sure the premise that runtime-tools only tests if the container comes up with shared option, right?
How about referring to runc's e2e?
https://github.com/opencontainers/runc/blob/81b13172bea2e6e4cf50f6bdd29a5fdeb5a6acf5/libcontainer/integration/exec_test.go#L1240

saku3 · 2025-02-03T23:41:06Z

@utam0k
Sorry for the slow response!
I created my test based on the validateRootfsPropagation function from runtime-tools, as referenced below:
https://github.com/opencontainers/runtime-tools/blob/f7e3563b0271e5cd52d5c915684ea11ef2779572/cmd/runtimetest/main.go#L538

However, I am struggling because only runc does not work correctly.
Even when I checked it manually on my local environment, runc behaves differently from the others.

Below are the results of my investigation.
I apologize for the long text.
I would really appreciate any advice you can give me.

Investigation

Investigation Approach

I manually created a container using the config.json file that is generated when running the integration test,
and I manually executed the same commands as in the integration test to verify the behavior.

I added the following code to output the generated JSON configuration:

youki/tests/contest/contest/src/utils/test_utils.rs

Line 198 in c7a10ac

println!(
  "Spec contents as JSON: {}",
  serde_json::to_string_pretty(&spec).unwrap()
);

The output result is as follows.

Click to expand code

{
  "ociVersion": "1.0.2-dev",
  "root": {
    "path": "rootfs",
    "readonly": false
  },
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc"
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "tmpfs",
      "source": "shm",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "mode=1777",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "source": "cgroup",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "ro"
      ]
    }
  ],
  "process": {
    "terminal": false,
    "user": {
      "uid": 0,
      "gid": 0
    },
    "args": [
      "runtimetest",
      "rootfs_propagation"
    ],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "TERM=xterm"
    ],
    "cwd": "/",
    "capabilities": {
      "bounding": [
        "CAP_SYS_ADMIN"
      ],
      "effective": [
        "CAP_SYS_ADMIN"
      ],
      "inheritable": [
        "CAP_SYS_ADMIN"
      ],
      "permitted": [
        "CAP_SYS_ADMIN"
      ],
      "ambient": [
        "CAP_SYS_ADMIN"
      ]
    },
    "rlimits": [
      {
        "type": "RLIMIT_NOFILE",
        "hard": 1024,
        "soft": 1024
      }
    ],
    "noNewPrivileges": true
  },
  "hostname": "youki",
  "annotations": {},
  "linux": {
    "resources": {
      "devices": []
    },
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "network"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "cgroup"
      }
    ],
    "seccomp": {
      "defaultAction": "SCMP_ACT_ALLOW"
    },
    "rootfsPropagation": "shared",
    "maskedPaths": [
      "/proc/acpi",
      "/proc/asound",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/sys/firmware",
      "/proc/scsi"
    ],
    "readonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  }
}

Manual Execution of the Above Integration Test

mkdir -p tutorial/rootfs
cd tutorial
docker export $(docker create busybox) | tar -C rootfs -xvf -

# Use the config.json output from the above test
# However, modify args to `sleep 300`
touch tutorial/config.json

./youki create -b tutorial youki

# Check the PID

# 1. Create the target directory
TARGET_DIR=$(mktemp -d -t target-XXXXXX)
echo "Created target directory: $TARGET_DIR"

# Bind-mount the root
mount --bind / $TARGET_DIR

# 2. Create a test mount directory and file
MOUNT_DIR=$(mktemp -d -t mount-XXXXXX)
TEST_DIR=$(mktemp -d -t test-XXXXXX)
echo "Created mount directory: $MOUNT_DIR"
echo "Created test directory: $TEST_DIR"

TMPFILE="$TEST_DIR/example"
touch "$TMPFILE"
echo "Created test file: $TMPFILE"

# 3. Bind-mount the test directory to another directory
mount --bind "$TEST_DIR" "$MOUNT_DIR"
echo "Bound $TEST_DIR to $MOUNT_DIR"

# 4. Check the file path inside the target directory
TARGET_FILE="$TARGET_DIR/$(echo $MOUNT_DIR | sed 's#^/##')/example"

ls $TARGET_FILE

Similarly, I executed the same test using runc and crun.

Results

In the case of youki

TARGET_FILE exists.

./youki create -b tutorial/ youki
./youki list
DEBUG youki: started by user 0 with ArgsOs { inner: ["./youki", "list"] }
ID     PID    STATUS   BUNDLE                                          CREATED                    CREATOR
youki  20927  Created  /var/snap/amazon-ssm-agent/9881/youki/tutorial  2025-02-03T23:04:42+00:00  root
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# PID=20927
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# nsenter --target $PID --mount --uts --ipc --net --pid sh
/ # TARGET_DIR=$(mktemp -d -t target-XXXXXX)
/ # echo "Created target directory: $TARGET_DIR"
Created target directory: /tmp/target-zow2pU
/ # mount --bind / $TARGET_DIR
/ # MOUNT_DIR=$(mktemp -d -t mount-XXXXXX)
/ # TEST_DIR=$(mktemp -d -t test-XXXXXX)
/ # echo "Created mount directory: $MOUNT_DIR"
Created mount directory: /tmp/mount-VDjxd0
/ # echo "Created test directory: $TEST_DIR"
Created test directory: /tmp/test-JPQr2e
/ # TMPFILE="$TEST_DIR/example"
/ # touch "$TMPFILE"
/ # echo "Created test file: $TMPFILE"
Created test file: /tmp/test-JPQr2e/example
/ # mount --bind "$TEST_DIR" "$MOUNT_DIR"
/ # echo "Bound $TEST_DIR to $MOUNT_DIR"
Bound /tmp/test-JPQr2e to /tmp/mount-VDjxd0
/ # TARGET_FILE="$TARGET_DIR/$(echo $MOUNT_DIR | sed 's#^/##')/example"
/ #
/ # ls $TARGET_FILE
/tmp/target-zow2pU/tmp/mount-VDjxd0/example

In the case of crun

TARGET_FILE exists.

root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# crun create -b tutorial/ crun
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# crun list
NAME PID       STATUS   BUNDLE PATH                             CREATED                        OWNER
crun 20957     created  /var/snap/amazon-ssm-agent/9881/youki/tutorial 2025-02-03T23:08:14.510045Z    root
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# PID=20957
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# nsenter --target $PID --mount --uts --ipc --net --pid sh

/ # TARGET_DIR=$(mktemp -d -t target-XXXXXX)
/ # echo "Created target directory: $TARGET_DIR"
Created target directory: /tmp/target-SOD8P2
/ # mount --bind / $TARGET_DIR
/ # MOUNT_DIR=$(mktemp -d -t mount-XXXXXX)
/ # TEST_DIR=$(mktemp -d -t test-XXXXXX)
/ # echo "Created mount directory: $MOUNT_DIR"
Created mount directory: /tmp/mount-72pCxu
/ # echo "Created test directory: $TEST_DIR"
Created test directory: /tmp/test-g0fhU1
/ # TMPFILE="$TEST_DIR/example"
/ # touch "$TMPFILE"
/ # echo "Created test file: $TMPFILE"
Created test file: /tmp/test-g0fhU1/example
/ # mount --bind "$TEST_DIR" "$MOUNT_DIR"
/ # echo "Bound $TEST_DIR to $MOUNT_DIR"
Bound /tmp/test-g0fhU1 to /tmp/mount-72pCxu
/ # TARGET_FILE="$TARGET_DIR/$(echo $MOUNT_DIR | sed 's#^/##')/example"
/ #
/ # ls $TARGET_FILE
/tmp/target-SOD8P2/tmp/mount-72pCxu/example

In the case of runc

TARGET_FILE does NOT exist.

root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# runc create -b tutorial runc
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# runc list
ID          PID         STATUS      BUNDLE                                           CREATED                          OWNER
runc        20992       created     /var/snap/amazon-ssm-agent/9881/youki/tutorial   2025-02-03T23:10:27.269997132Z   root
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# PID=20992
root@ip-172-31-11-229:/var/snap/amazon-ssm-agent/9881/youki# nsenter --target $PID --mount --uts --ipc --net --pid sh
/ # TARGET_DIR=$(mktemp -d -t target-XXXXXX)
/ # echo "Created target directory: $TARGET_DIR"
Created target directory: /tmp/target-NO1xUG
/ # mount --bind / $TARGET_DIR
/ # MOUNT_DIR=$(mktemp -d -t mount-XXXXXX)
/ # TEST_DIR=$(mktemp -d -t test-XXXXXX)
/ # echo "Created mount directory: $MOUNT_DIR"
Created mount directory: /tmp/mount-1U0CTR
/ # echo "Created test directory: $TEST_DIR"
Created test directory: /tmp/test-bb90sL
/ # TMPFILE="$TEST_DIR/example"
/ # touch "$TMPFILE"
/ # echo "Created test file: $TMPFILE"
Created test file: /tmp/test-bb90sL/example
/ # mount --bind "$TEST_DIR" "$MOUNT_DIR"
/ # echo "Bound $TEST_DIR to $MOUNT_DIR"
Bound /tmp/test-bb90sL to /tmp/mount-1U0CTR
/ # TARGET_FILE="$TARGET_DIR/$(echo $MOUNT_DIR | sed 's#^/##')/example"
/ #
/ # ls $TARGET_FILE
ls: /tmp/target-NO1xUG/tmp/mount-1U0CTR/example: No such file or directory

Additionally, I checked the rootfs propagation information using cat /proc/<PID>/mountinfo.

In the case of youki

The 7th field in the corresponding line is shared, indicating shared mount propagation.

cat /proc/20927/mountinfo
361 262 259:1 /var/snap/amazon-ssm-agent/9881/youki/tutorial/rootfs / rw,relatime shared:343 - ext4 /dev/root rw,discard,errors=remount-ro,commit=30

In the case of crun

Similar to youki, the 7th field is also shared.

cat /proc/20957/mountinfo
352 112 259:1 /var/snap/amazon-ssm-agent/9881/youki/tutorial/rootfs / rw,relatime shared:342 - ext4 /dev/root rw,discard,errors=remount-ro,commit=30

In the case of runc

The 7th field is -, indicating that mount propagation is not shared.

cat /proc/20992/mountinfo
292 119 259:1 /var/snap/amazon-ssm-agent/9881/youki/tutorial/rootfs / rw,relatime - ext4 /dev/root rw,discard,errors=remount-ro,commit=30

This suggests that runc does not have shared mount propagation enabled, whereas youki and crun do.

crun --version
crun version 1.19.1.0.0.0.54-73d52
commit: 73d52bd6fc6b1886a0176701596d82a4f6b0fac3
rundir: /run/user/0/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL


runc --version
runc version 1.2.2
commit: v1.2.2-0-g7cb3632
spec: 1.2.0
go: go1.22.9
libseccomp: 2.5.5


./youki --version
youki version 0.4.1
commit: 0.4.1-0-37bceb174ae1ec2491c74557ef36b30f7298c43a

utam0k · 2025-02-09T02:04:56Z

@saku3 Your investigation is incredible. How about reporting the findings to the runc team?

utam0k · 2025-02-09T02:06:04Z

Maybe contest needs a mechanism to skip some tests for runc.

YJDoc2 added the kind/test label Dec 16, 2024

add-test-linux_rootfs_propagation

c87a97e

Signed-off-by: Yusuke Sakurai <yusuke.sakurai@3-shake.com>

saku3 force-pushed the add-test-linux_rootfs_propagation branch from 2979c20 to c87a97e Compare January 2, 2025 10:15

utam0k self-assigned this Jan 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add-test-linux_rootfs_propagation #3024

add-test-linux_rootfs_propagation #3024

saku3 commented Dec 15, 2024

saku3 commented Dec 16, 2024

YJDoc2 commented Jan 2, 2025

utam0k commented Jan 6, 2025

saku3 commented Jan 6, 2025

utam0k commented Jan 7, 2025 •

edited

Loading

saku3 commented Feb 3, 2025

utam0k commented Feb 9, 2025

utam0k commented Feb 9, 2025

add-test-linux_rootfs_propagation #3024

Are you sure you want to change the base?

add-test-linux_rootfs_propagation #3024

Conversation

saku3 commented Dec 15, 2024

saku3 commented Dec 16, 2024

YJDoc2 commented Jan 2, 2025

utam0k commented Jan 6, 2025

saku3 commented Jan 6, 2025

utam0k commented Jan 7, 2025 • edited Loading

saku3 commented Feb 3, 2025

Investigation

Investigation Approach

Manual Execution of the Above Integration Test

Results

In the case of youki

In the case of crun

In the case of runc

In the case of youki

In the case of crun

In the case of runc

utam0k commented Feb 9, 2025

utam0k commented Feb 9, 2025

utam0k commented Jan 7, 2025 •

edited

Loading