From 9be6dacf1545e51461e685fb6f014367da2ebcea Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Fri, 11 Oct 2024 11:41:05 -0700 Subject: [PATCH 1/9] WIP:docs: document process of setting up a local server Preliminary documentation for setup. Requires several subsequent commits on this branch. Signed-off-by: Eric Fahlgren --- local-server.md | 182 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 local-server.md diff --git a/local-server.md b/local-server.md new file mode 100644 index 00000000..aab45069 --- /dev/null +++ b/local-server.md @@ -0,0 +1,182 @@ +#### Setting up a local server + +Assumptions: + - You're using a recent Ubuntu for install, below examples developed on a qemu VM using 24.04. + - Examples below use `apt` + - Has Python 3.12, `git` already installed + - You are going to use the server on your LAN for local installs, and not expose it to the internet, hence no discussion of proxies or whatnot. + +First check IPv6 connectivity from your VM: +```bash +curl -6 https://sysupgrade.openwrt.org/json/v1/overview.json +``` + +If that fails to connect, then you will have all sorts of issues unless you resolve them. The easiest thing to do is just disable IPv6 on your VM: +```bash +sudo vi /etc/sysctl.d/10-ipv6-privacy.conf +``` +Add one line: +``` +net.ipv6.conf.all.disable_ipv6 = 1 +``` +and reload: +```bash +sudo sysctl -f /etc/sysctl.d/10-ipv6-privacy.conf +``` + +If you can figure out how to get qemu to punch through the IPv6 blocking, @efahl would really (really) like to know. + +Make sure you have `podman`, Ubuntu 24.04 did not: + +```bash +cd ~ +sudo apt -y install podman +systemctl --user enable podman.socket +systemctl --user start podman.socket +systemctl --user status podman.socket +``` + +Test that podman is working by running the Alpine Linux container. This will drop you on its command line, and if you see the `/ #` prompt, simply `exit` to return to your server's command line. + + podman run --rm -it docker.io/library/alpine:latest + / # cat /etc/os-release + NAME="Alpine Linux" + ID=alpine + VERSION_ID=3.20.3 + PRETTY_NAME="Alpine Linux v3.20" + HOME_URL="https://alpinelinux.org/" + BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" + / # exit + + +Create a new Python virtual environment using `venv`: + +```bash +sudo apt -y install python3-venv +python3 -m venv asu-venv +. asu-venv/bin/activate +``` + +Test your new virtual environment. Verify that the executables are in your venv, and that the Python version is 3.11 or newer. + +```bash +$ which python +/home/efahlgren/asu-venv/bin/python +$ which pip +/home/efahlgren/asu-venv/bin/pip +$ python --version +Python 3.12.3 +``` + +Next, install the basic Python tools (`poetry` will be used to easily install all the rest of the requirements): + +```bash +pip install poetry podman-compose +``` + +Get ASU and install all its requirements: + +```bash +git clone https://github.com/openwrt/asu.git +cd asu/ +poetry install +``` + +Set up your local podman environment. The `.env` file contains primary definitions or overrides for the contents of `settings`, which may be found in `asu/config.py`. + + echo "# where to store images and json files + PUBLIC_PATH=$(pwd)/public + HOST_PATH=$(pwd)/public + # absolute path to podman socket mounted into worker containers + CONTAINER_SOCK=/run/user/$(id -u)/podman/podman.sock + # allow host cli tools access to redis database + REDIS_URL=redis://localhost:6379 + # turn on the 'defaults' option on the server + ALLOW_DEFAULTS=True + " > .env + +Edit `podman-compose.yml` and make the server listen on the VM's WAN port at `0.0.0.0`: +```bash +server: + ... + ports: + - "0.0.0.0:8000:8000" +``` + +Start up the server: +```bash +$ podman-compose up -d +$ podman logs asu_server_1 +INFO: Started server process [2] +INFO: Waiting for application startup. +INFO:root:ASU server starting up +INFO: Application startup complete. +INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) +``` + + +Check that the server is up. `ssh` into your router and fetch the front page, this should spew a pile of html: +```bash +asu_server= +uclient-fetch -O - "http://$asu_server:8000/" +``` + +On a host with "real" curl (we need `--headers`), pick a version, target and subtarget and compose an update query as follows. This is the mechanism by which your ASU server will learn about new releases, so for each version/target/subtarget combination, you need to run a similar query. (To update almost everything, you can run `python misc/update_all_targets.py`, but that's fairly wasteful of time and bandwidth.) + +```bash +curl -v --header "x-update-token: foobar" "http://$asu_server:8000/api/v1/update/SNAPSHOT/x86/64" +``` +Note that the value of "x-update-token" is "foobar" by default, but can be changed in `asu/config.py` or by adding `UPDATE_TOKEN=whatever` in the `.env` file. + +Selectively add more versions to the server from your router (if you have curl installed), or from your workstation using the data from the router. Here's how you'd go about it on the router: + +```bash +$ eval $(ubus call system board | jsonfilter -e 'version=$.release.version' -e 'target=$.release.target') +$ echo "$version $target" +23.05.5 mediatek/mt7622 +$ curl -v --header "x-update-token: foobar" "http://$asu_server:8000/api/v1/update/$version/$target" +``` +(Note that you can run these `curl` queries on the ASU server itself, it has `curl` and you just use `localhost` as the value for `$asu_server`.) + +Back on your ASU server, look at the worker log and see what happened: + +```bash +$ podman logs asu_worker_1 +... +01:18:20 default: asu.update.update(target_subtarget='x86/64', version='SNAPSHOT') (2376baed-c4bf-4d37-ba9c-4021feec54b6) +01:18:20 SNAPSHOT: Found 86 targets +01:18:20 SNAPSHOT/x86/64: Found 1 profiles +01:18:20 SNAPSHOT/x86/64: Found revision r27707-084665698b +01:18:20 default: Job OK (2376baed-c4bf-4d37-ba9c-4021feec54b6) +01:18:20 Result is kept for 500 seconds +``` + +You can now try to do a download using LuCI ASU, `auc` or `owut`. First point the `attendedsysupgrade` config at your server. + +```bash +uci set attendedsysupgrade.server.url="http://$asu_server:8000" +uci commit +``` +(To revert, simply substitute `https://sysupgrade.openwrt.org` as the `url`.) + +On snapshot, run an `owut` check with `--verbose` to see where it's getting data: +``` +$ owut check -v +owut - OpenWrt Upgrade Tool +Downloaded http://$asu_server:8000/json/v1/overview.json to /tmp/owut-overview.json (16073B at 0.245 Mbps) +... +``` + +Or for 23.05 and earler, use `auc`: +```bash +$ auc -c +auc/0.3.2-1 +Server: https://10.1.1.207:8000 +Running: 23.05.5 r24106-10cc5fcd00 on mediatek/mt7622 (linksys,e8450-ubi) +Available: 23.05.5 r24106-10cc5fcd00 +Requesting package lists... + luci-app-adblock: git-24.224.28330-dc8b3a6 -> git-24.284.61672-4b84d8e + adblock: 4.2.2-5 -> 4.2.2-6 + luci-mod-network: git-24.264.56960-63ba3cb -> git-24.281.58052-a6c2279 +``` + From 99ca66a8c53b0ba1cd350f9874f8bfd3a9ee60e1 Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Fri, 11 Oct 2024 11:42:00 -0700 Subject: [PATCH 2/9] WIP:config: detangle volume mounts of host and containers Make explicit where containers and the hosts mount their shared volumes. First, so it works. Second, so it's easy to manage on the host side. Copy and configure '.env` to the containers, removing the mount points where appropriate. Signed-off-by: Eric Fahlgren --- Containerfile | 2 ++ asu/build.py | 36 ++++++++++++++++++++---------------- asu/config.py | 13 ++++++++++--- asu/util.py | 2 +- podman-compose.yml | 9 +++++++-- tests/conftest.py | 10 +++++++++- 6 files changed, 49 insertions(+), 23 deletions(-) diff --git a/Containerfile b/Containerfile index b77f689c..258ae42a 100644 --- a/Containerfile +++ b/Containerfile @@ -10,5 +10,7 @@ RUN poetry config virtualenvs.create false \ && poetry install --only main --no-interaction --no-ansi COPY ./asu/ ./asu/ +RUN --mount=type=bind,source=./.env,target=/tmp/.env \ + grep -vE 'REDIS_URL|PUBLIC_PATH' /tmp/.env > ./.env CMD uvicorn --host 0.0.0.0 'asu.main:app' diff --git a/asu/build.py b/asu/build.py index d021ecfb..6a128682 100644 --- a/asu/build.py +++ b/asu/build.py @@ -41,7 +41,10 @@ def build(build_request: BuildRequest, job=None): request_hash = get_request_hash(build_request) bin_dir: Path = settings.public_path / "store" / request_hash bin_dir.mkdir(parents=True, exist_ok=True) - log.debug(f"Bin dir: {bin_dir}") + host_dir: Path = settings.host_path / "store" / request_hash + bldr_dir: Path = settings.builder_path + + log.debug(f"Build dirs:\n {bin_dir = !s}\n {host_dir = !s}\n {bldr_dir = !s}") job = job or get_current_job() job.meta["detail"] = "init" @@ -98,8 +101,8 @@ def build(build_request: BuildRequest, job=None): mounts.append( { "type": "bind", - "source": str(bin_dir / "keys" / fingerprint), - "target": "/builder/keys/" + fingerprint, + "source": str(host_dir / "keys" / fingerprint), + "target": str(bldr_dir / "keys" / fingerprint), "read_only": True, }, ) @@ -120,8 +123,8 @@ def build(build_request: BuildRequest, job=None): mounts.append( { "type": "bind", - "source": str(bin_dir / "repositories.conf"), - "target": "/builder/repositories.conf", + "source": str(host_dir / "repositories.conf"), + "target": str(bldr_dir / "repositories.conf"), "read_only": True, }, ) @@ -130,13 +133,14 @@ def build(build_request: BuildRequest, job=None): log.debug("Found defaults") defaults_file = bin_dir / "files/etc/uci-defaults/99-asu-defaults" - defaults_file.parent.mkdir(parents=True) + log.info(f"Found defaults, storing at {defaults_file = !s}") + defaults_file.parent.mkdir(parents=True, exist_ok=True) defaults_file.write_text(build_request.defaults) mounts.append( { "type": "bind", - "source": str(bin_dir / "files"), - "target": str(bin_dir / "files"), + "source": str(host_dir / "files"), + "target": str(bldr_dir / "files"), "read_only": True, }, ) @@ -237,11 +241,11 @@ def build(build_request: BuildRequest, job=None): f"PROFILE={build_request.profile}", f"PACKAGES={' '.join(build_cmd_packages)}", f"EXTRA_IMAGE_NAME={packages_hash}", - f"BIN_DIR=/builder/{request_hash}", + f"BIN_DIR={bldr_dir!s}/{request_hash}", ] if build_request.defaults: - job.meta["build_cmd"].append(f"FILES={bin_dir}/files") + job.meta["build_cmd"].append(f"FILES={bldr_dir!s}/files") # Check if custom rootfs size is requested if build_request.rootfs_size_mb: @@ -256,7 +260,7 @@ def build(build_request: BuildRequest, job=None): returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd( container, job.meta["build_cmd"], - copy=["/builder/" + request_hash, bin_dir.parent], + copy=[str(bldr_dir / request_hash), str(bin_dir.parent)], ) container.kill() @@ -297,7 +301,7 @@ def build(build_request: BuildRequest, job=None): # job.meta["imagebuilder_status"] = "signing_images" job.save_meta() - build_key = getenv("BUILD_KEY") or str(Path.cwd() / "key-build") + build_key = getenv("BUILD_KEY") or str(host_dir / "key-build") if Path(build_key).is_file(): log.info(f"Signing images with key {build_key}") @@ -307,18 +311,18 @@ def build(build_request: BuildRequest, job=None): { "type": "bind", "source": build_key, - "target": "/builder/key-build", + "target": str(bldr_dir / "key-build"), "read_only": True, }, { "type": "bind", "source": build_key + ".ucert", - "target": "/builder/key-build.ucert", + "target": str(bldr_dir / "key-build.ucert"), "read_only": True, }, { "type": "bind", - "source": str(bin_dir), + "source": str(host_dir), "target": request_hash, "read_only": False, }, @@ -327,7 +331,7 @@ def build(build_request: BuildRequest, job=None): working_dir=request_hash, environment={ "IMAGES_TO_SIGN": " ".join(images), - "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builder/staging_dir/host/bin", + "PATH": f"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:{bldr_dir!s}/staging_dir/host/bin", }, auto_remove=True, ) diff --git a/asu/config.py b/asu/config.py index 34c014f8..e41b7db0 100644 --- a/asu/config.py +++ b/asu/config.py @@ -7,9 +7,16 @@ class Settings(BaseSettings): model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8") + # The following two vary between host and container. Default values + # are for the container, and should not be overridden in copied .env, see + # Containerfile for where we remove them. + redis_url: str = "redis://redis/" # host value = "redis://localhost:6379" public_path: Path = Path.cwd() / "public" - json_path: Path = public_path / "json" / "v1" - redis_url: str = "redis://localhost:6379" + + host_path: Path = "" # The fixed host "public" path, must be in .env. + builder_path: Path = Path("/builder") # Path to working directory on builder. + json_path: Path = Path(public_path) / "json" / "v1" + upstream_url: str = "https://downloads.openwrt.org" allow_defaults: bool = False async_queue: bool = True @@ -19,7 +26,7 @@ class Settings(BaseSettings): repository_allow_list: list = [] base_container: str = "ghcr.io/openwrt/imagebuilder" update_token: Union[str, None] = "foobar" - container_host: str = "localhost" + container_sock: str = "" container_identity: str = "" branches: dict = { "SNAPSHOT": { diff --git a/asu/util.py b/asu/util.py index 655dbb12..6a31a4b0 100644 --- a/asu/util.py +++ b/asu/util.py @@ -221,7 +221,7 @@ def get_container_version_tag(input_version: str) -> str: def get_podman(): return PodmanClient( - base_url=settings.container_host, + base_url=f"unix://{settings.container_sock}", identity=settings.container_identity, ) diff --git a/podman-compose.yml b/podman-compose.yml index dd48a72f..11ada81e 100644 --- a/podman-compose.yml +++ b/podman-compose.yml @@ -2,6 +2,8 @@ version: "3" volumes: redis: + public: + $PUBLIC_PATH:$PUBLIC_PATH:rw,rshared services: server: @@ -14,6 +16,8 @@ services: - REDIS_URL=redis://redis/ ports: - "127.0.0.1:8000:8000" + volumes: + - $PUBLIC_PATH:/app/public:rw,rshared depends_on: - redis @@ -25,10 +29,9 @@ services: restart: always command: rqworker --with-scheduler environment: - - CONTAINER_HOST=unix://$CONTAINER_SOCK - REDIS_URL=redis://redis/ volumes: - - $PUBLIC_PATH:$PUBLIC_PATH:rw + - $PUBLIC_PATH:/app/public:rw,rshared - $CONTAINER_SOCK:$CONTAINER_SOCK:rw depends_on: - redis @@ -38,6 +41,8 @@ services: restart: always volumes: - redis:/data/:rw + ports: + - "127.0.0.1:6379:6379" squid: image: "docker.io/ubuntu/squid:latest" diff --git a/tests/conftest.py b/tests/conftest.py index df67cc34..c2f6f0bf 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -54,7 +54,7 @@ def redis_load_mock_data(redis): @pytest.fixture -def redis_server(): +def redis_server(unicode=True): r = FakeStrictRedis() redis_load_mock_data(r) yield r @@ -94,14 +94,22 @@ def mocked_redis_client(*args, **kwargs): return redis_server settings.public_path = Path(test_path) / "public" + settings.host_path = settings.public_path + settings.public_path.mkdir(parents=True) + settings.async_queue = False for branch in "1.2", "19.07", "21.02": if branch not in settings.branches: settings.branches[branch] = {"path": "releases/{version}"} + settings.allow_defaults = False monkeypatch.setattr("asu.util.get_redis_client", mocked_redis_client) monkeypatch.setattr("asu.routers.api.get_redis_client", mocked_redis_client) + from os import getuid + + assert settings.container_sock == f"/run/user/{getuid()}/podman/podman.sock" + yield real_app From 56d5bbb1b1caf0bafa639271b0cc9472fda0d1cd Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Fri, 11 Oct 2024 19:12:09 -0700 Subject: [PATCH 3/9] WIP:docs: add missing information Added the crucial information about setting up a proper '.env'. Minor formatting and wording changes. Signed-off-by: Eric Fahlgren --- local-server.md | 72 +++++++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 32 deletions(-) diff --git a/local-server.md b/local-server.md index aab45069..a952e427 100644 --- a/local-server.md +++ b/local-server.md @@ -1,12 +1,16 @@ -#### Setting up a local server +# Setting up a local server Assumptions: - You're using a recent Ubuntu for install, below examples developed on a qemu VM using 24.04. - Examples below use `apt` - - Has Python 3.12, `git` already installed + - `git` is already installed + - Has Python 3.12 (3.11 is ok) - You are going to use the server on your LAN for local installs, and not expose it to the internet, hence no discussion of proxies or whatnot. -First check IPv6 connectivity from your VM: +## First check IPv6 connectivity from your VM + +Run curl against an external server forcing IPv6, if this works, then skip forward. + ```bash curl -6 https://sysupgrade.openwrt.org/json/v1/overview.json ``` @@ -26,6 +30,8 @@ sudo sysctl -f /etc/sysctl.d/10-ipv6-privacy.conf If you can figure out how to get qemu to punch through the IPv6 blocking, @efahl would really (really) like to know. +## Podman installation + Make sure you have `podman`, Ubuntu 24.04 did not: ```bash @@ -36,18 +42,7 @@ systemctl --user start podman.socket systemctl --user status podman.socket ``` -Test that podman is working by running the Alpine Linux container. This will drop you on its command line, and if you see the `/ #` prompt, simply `exit` to return to your server's command line. - - podman run --rm -it docker.io/library/alpine:latest - / # cat /etc/os-release - NAME="Alpine Linux" - ID=alpine - VERSION_ID=3.20.3 - PRETTY_NAME="Alpine Linux v3.20" - HOME_URL="https://alpinelinux.org/" - BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" - / # exit - +## Python configuration Create a new Python virtual environment using `venv`: @@ -68,13 +63,15 @@ $ python --version Python 3.12.3 ``` -Next, install the basic Python tools (`poetry` will be used to easily install all the rest of the requirements): +Install the basic tools (`poetry` will be used to easily install all the rest of the requirements): ```bash pip install poetry podman-compose ``` -Get ASU and install all its requirements: +## Attended Sysupgrade installation and configuration + +Get ASU and install all of its requirements: ```bash git clone https://github.com/openwrt/asu.git @@ -82,19 +79,6 @@ cd asu/ poetry install ``` -Set up your local podman environment. The `.env` file contains primary definitions or overrides for the contents of `settings`, which may be found in `asu/config.py`. - - echo "# where to store images and json files - PUBLIC_PATH=$(pwd)/public - HOST_PATH=$(pwd)/public - # absolute path to podman socket mounted into worker containers - CONTAINER_SOCK=/run/user/$(id -u)/podman/podman.sock - # allow host cli tools access to redis database - REDIS_URL=redis://localhost:6379 - # turn on the 'defaults' option on the server - ALLOW_DEFAULTS=True - " > .env - Edit `podman-compose.yml` and make the server listen on the VM's WAN port at `0.0.0.0`: ```bash server: @@ -103,9 +87,28 @@ server: - "0.0.0.0:8000:8000" ``` +Set up your initial podman environment: + +```bash +echo "# where to store images and json files +PUBLIC_PATH=$(pwd)/public +HOST_PATH=$(pwd)/public +# absolute path to podman socket mounted into worker containers +CONTAINER_SOCK=/run/user/$(id -u)/podman/podman.sock +# allow host cli tools access to redis database +REDIS_URL=redis://localhost:6379 +# turn on the 'defaults' option on the server +ALLOW_DEFAULTS=True +" > .env +``` + +## Running the server + Start up the server: ```bash $ podman-compose up -d +... + $ podman logs asu_server_1 INFO: Started server process [2] INFO: Waiting for application startup. @@ -114,8 +117,7 @@ INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ``` - -Check that the server is up. `ssh` into your router and fetch the front page, this should spew a pile of html: +Check that the server is accessible. `ssh` into your router and fetch the front page, this should spew a pile of html: ```bash asu_server= uclient-fetch -O - "http://$asu_server:8000/" @@ -180,3 +182,9 @@ Requesting package lists... luci-mod-network: git-24.264.56960-63ba3cb -> git-24.281.58052-a6c2279 ``` +## Deployment notes + +If you want your server to remain active after you log out of the server, you must enable "linger" in `loginctl`: +```bash +loginctl enable-linger +``` From ae7984a8939e45f10cc755bda739eef2faec01fb Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Wed, 16 Oct 2024 09:58:22 -0700 Subject: [PATCH 4/9] ci: use new container socket setting Make the CI test runs conform to the reworked container paths. Signed-off-by: Eric Fahlgren --- .github/workflows/test.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 53e75d00..9fd4717d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -53,8 +53,8 @@ jobs: - name: Test with pytest run: | - podman system service --time=0 unix:///tmp/podman.sock & - export CONTAINER_HOST="unix:///tmp/podman.sock" + export CONTAINER_SOCK="/tmp/podman.sock" + podman system service --time=0 "unix://$CONTAINER_SOCK" & poetry run coverage run -m pytest -vv --runslow poetry run coverage xml From b9dd768e90915c7f848be0c9c7e85b7f07506d32 Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Wed, 16 Oct 2024 10:03:01 -0700 Subject: [PATCH 5/9] tests: remove check on settings.container_sock CI and userland testing use different values for the 'container_sock', so remove it. Signed-off-by: Eric Fahlgren --- tests/conftest.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/conftest.py b/tests/conftest.py index c2f6f0bf..641a3851 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -106,10 +106,6 @@ def mocked_redis_client(*args, **kwargs): monkeypatch.setattr("asu.util.get_redis_client", mocked_redis_client) monkeypatch.setattr("asu.routers.api.get_redis_client", mocked_redis_client) - from os import getuid - - assert settings.container_sock == f"/run/user/{getuid()}/podman/podman.sock" - yield real_app From 28c167a2acdc235e4d121a688b1b3db549d64edd Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Thu, 17 Oct 2024 08:20:49 -0700 Subject: [PATCH 6/9] readme: fix up .env setup Make the README track the contents of local_server documentation and changes to config. Signed-off-by: Eric Fahlgren --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 563d5b1f..111d9ed3 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,7 @@ the dependencies: #### Running a worker # podman unix socket (not path), no need to mount anything - export CONTAINER_HOST=unix:///run/user/1001/podman/podman.sock + export CONTAINER_SOCK=/run/user/$(id -u)/podman/podman.sock poetry run rq worker #### Update targets From be4efb63ab906433794b52b03a4f3091a9180554 Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Fri, 25 Oct 2024 18:25:05 -0700 Subject: [PATCH 7/9] builder: code cleanup Fix abbreviations and remove string directives from f-strings. Signed-off-by: Eric Fahlgren --- asu/build.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/asu/build.py b/asu/build.py index 6a128682..e1b0e19c 100644 --- a/asu/build.py +++ b/asu/build.py @@ -42,9 +42,9 @@ def build(build_request: BuildRequest, job=None): bin_dir: Path = settings.public_path / "store" / request_hash bin_dir.mkdir(parents=True, exist_ok=True) host_dir: Path = settings.host_path / "store" / request_hash - bldr_dir: Path = settings.builder_path + builder_dir: Path = settings.builder_path - log.debug(f"Build dirs:\n {bin_dir = !s}\n {host_dir = !s}\n {bldr_dir = !s}") + log.debug(f"Build dirs:\n {bin_dir = }\n {host_dir = }\n {builder_dir = }") job = job or get_current_job() job.meta["detail"] = "init" @@ -102,7 +102,7 @@ def build(build_request: BuildRequest, job=None): { "type": "bind", "source": str(host_dir / "keys" / fingerprint), - "target": str(bldr_dir / "keys" / fingerprint), + "target": str(builder_dir / "keys" / fingerprint), "read_only": True, }, ) @@ -124,7 +124,7 @@ def build(build_request: BuildRequest, job=None): { "type": "bind", "source": str(host_dir / "repositories.conf"), - "target": str(bldr_dir / "repositories.conf"), + "target": str(builder_dir / "repositories.conf"), "read_only": True, }, ) @@ -133,14 +133,14 @@ def build(build_request: BuildRequest, job=None): log.debug("Found defaults") defaults_file = bin_dir / "files/etc/uci-defaults/99-asu-defaults" - log.info(f"Found defaults, storing at {defaults_file = !s}") + log.info(f"Found defaults, storing at {defaults_file = }") defaults_file.parent.mkdir(parents=True, exist_ok=True) defaults_file.write_text(build_request.defaults) mounts.append( { "type": "bind", "source": str(host_dir / "files"), - "target": str(bldr_dir / "files"), + "target": str(builder_dir / "files"), "read_only": True, }, ) @@ -241,11 +241,11 @@ def build(build_request: BuildRequest, job=None): f"PROFILE={build_request.profile}", f"PACKAGES={' '.join(build_cmd_packages)}", f"EXTRA_IMAGE_NAME={packages_hash}", - f"BIN_DIR={bldr_dir!s}/{request_hash}", + f"BIN_DIR={builder_dir}/{request_hash}", ] if build_request.defaults: - job.meta["build_cmd"].append(f"FILES={bldr_dir!s}/files") + job.meta["build_cmd"].append(f"FILES={builder_dir}/files") # Check if custom rootfs size is requested if build_request.rootfs_size_mb: @@ -260,7 +260,7 @@ def build(build_request: BuildRequest, job=None): returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd( container, job.meta["build_cmd"], - copy=[str(bldr_dir / request_hash), str(bin_dir.parent)], + copy=[str(builder_dir / request_hash), str(bin_dir.parent)], ) container.kill() @@ -311,13 +311,13 @@ def build(build_request: BuildRequest, job=None): { "type": "bind", "source": build_key, - "target": str(bldr_dir / "key-build"), + "target": str(builder_dir / "key-build"), "read_only": True, }, { "type": "bind", "source": build_key + ".ucert", - "target": str(bldr_dir / "key-build.ucert"), + "target": str(builder_dir / "key-build.ucert"), "read_only": True, }, { @@ -331,7 +331,7 @@ def build(build_request: BuildRequest, job=None): working_dir=request_hash, environment={ "IMAGES_TO_SIGN": " ".join(images), - "PATH": f"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:{bldr_dir!s}/staging_dir/host/bin", + "PATH": f"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:{builder_dir}/staging_dir/host/bin", }, auto_remove=True, ) From 903a77fd0d3d49582db5f28cd58d9ea89c5737a0 Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Fri, 25 Oct 2024 18:42:27 -0700 Subject: [PATCH 8/9] ci-test: remove envvar Get rid of CONTAINER_SOCK as it's unneeded. Signed-off-by: Eric Fahlgren --- .github/workflows/test.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9fd4717d..cd1613d8 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -53,8 +53,7 @@ jobs: - name: Test with pytest run: | - export CONTAINER_SOCK="/tmp/podman.sock" - podman system service --time=0 "unix://$CONTAINER_SOCK" & + podman system service --time=0 "unix:///tmp/podman.sock" & poetry run coverage run -m pytest -vv --runslow poetry run coverage xml From 4b6c9a2849e2ba03a5396b1e86cfddd1d24a4d12 Mon Sep 17 00:00:00 2001 From: Eric Fahlgren Date: Sat, 26 Oct 2024 01:27:47 -0700 Subject: [PATCH 9/9] ci-test: restore envvar Restore CONTAINER_SOCK as it's needed by the worker container when creating builders. Signed-off-by: Eric Fahlgren --- .github/workflows/test.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index cd1613d8..9fd4717d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -53,7 +53,8 @@ jobs: - name: Test with pytest run: | - podman system service --time=0 "unix:///tmp/podman.sock" & + export CONTAINER_SOCK="/tmp/podman.sock" + podman system service --time=0 "unix://$CONTAINER_SOCK" & poetry run coverage run -m pytest -vv --runslow poetry run coverage xml