[CI] Fixed container images packaging #6972

pazone · 2025-02-21T14:59:25Z

What does this PR do?

Fixed container image packaging by setting proper packaging variables
We now have 9 packaging steps:

Packaging: linux/amd64 rpm

Artifacts :

elastic-agent-9.0.0-SNAPSHOT-x86_64.rpm
elastic-agent-9.0.0-SNAPSHOT-x86_64.rpm.sha512

Packaging: linux/arm64 rpm

Artifacts :

elastic-agent-9.0.0-SNAPSHOT-aarch64.rpm
elastic-agent-9.0.0-SNAPSHOT-aarch64.rpm.sha512

Packaging: linux/amd64 deb

Artifacts :

elastic-agent-9.0.0-SNAPSHOT-amd64.deb
elastic-agent-9.0.0-SNAPSHOT-amd64.deb.sha512

Packaging: linux/arm64 deb

Artifacts:

elastic-agent-9.0.0-SNAPSHOT-arm64.deb
elastic-agent-9.0.0-SNAPSHOT-arm64.deb.sha512

Packaging: linux/amd64 tar.gz

Artifacts:

elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz
elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512

Packaging: linux/arm64 tar.gz

Artifacts:

elastic-agent-9.0.0-SNAPSHOT-linux-arm64.tar.gz
elastic-agent-9.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512

Packaging: Containers linux/amd64

14 amd64 images. See the build

Packaging: Containers linux/arm64

14 arm64 images. See the build

Packaging: windows/amd64 zip

elastic-agent-9.0.0-SNAPSHOT-windows-x86_64.zip
elastic-agent-9.0.0-SNAPSHOT-windows-x86_64.zip.sha512

Why is it important?

The container images will be used by k8s integration tests instead of rebuilding them.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

…tic-agent into ci_fix_packaging_containers

pazone · 2025-02-25T23:57:57Z

.buildkite/bk.integration.pipeline.yml

        command: |
-          buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-x86-64' --build ${BUILDKITE_TRIGGERED_FROM_BUILD_ID}


Build matrix is not used because we need unique step keys

mergify · 2025-02-26T07:35:03Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ci_fix_packaging_containers upstream/ci_fix_packaging_containers
git merge upstream/main
git push upstream ci_fix_packaging_containers

pazone · 2025-02-26T10:26:33Z

Build takes too long because of the emulation

mergify · 2025-03-03T10:44:02Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ci_fix_packaging_containers upstream/ci_fix_packaging_containers
git merge upstream/main
git push upstream ci_fix_packaging_containers

elasticmachine · 2025-03-04T10:52:07Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

elasticmachine · 2025-03-04T16:08:22Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: cbc3489

Failed CI Steps

History

💔 Build #18033 failed c84271d
💔 Build #18014 failed a404b1d
💔 Build #18009 failed 4c6d45e
💔 Build #17996 failed 8e20566
💔 Build #17569 failed ea79860

cc @pazone

elastic-sonarqube · 2025-03-04T17:46:54Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

dliappis · 2025-03-04T19:35:28Z

.buildkite/bk.integration.pipeline.yml

@@ -108,7 +122,7 @@ steps:
      - integration-ess
    steps:
      - label: "x86_64:non-sudo: {{matrix}}"
-        # only packaging-ubuntu-x86-64 artifact dependency is required
+        depends_on: packaging-ubuntu-x86-64


nit: let's stick with one style (can be done in a follow up) either like this, or like below using a list

This is about to be removed #7184

dliappis · 2025-03-04T19:38:59Z

.buildkite/bk.integration.pipeline.yml

@@ -27,6 +27,8 @@ steps:
      - integration-ess
    steps:
      - label: "Win2022:sudo:{{matrix}}"
+        depends_on:
+          - packaging-windows


Just curious: how come we didn't need such explicit dependencies until now?

Good question. The integration tests trigger step depends on the entire int-packaging group that wraps all the packaging steps. So these dependensies give no effect. On the other hand, they strictly outline the exact step dependencies

dliappis · 2025-03-04T19:39:19Z

.buildkite/bk.integration.pipeline.yml

@@ -78,6 +84,9 @@ steps:
          provider: "gcp"
          machineType: "n1-standard-8"


nit: in a future PR let's get rid of those ancient n1 instance types in favor of n2.

dliappis · 2025-03-04T19:40:54Z

.buildkite/integration.pipeline.yml

-        key: "packaging-ubuntu-x86-64"
+      # Build matrix is not used for packaging in favor to unique step keys
+      # Packaging linux/amd64
+      - label: "Packaging: linux/amd64 rpm"


should we update this to show rpm,deb,tgz since it's doing all three?

It is still possible to use the build matrix with 2 axes for arm and x86. The comment is added for questions like this one

[CI] Fixed container images packaging

4aa8682

pazone added enhancement New feature or request skip-changelog Team:Ingest-EngProd backport-active-all Automated backport with mergify to all the active branches labels Feb 21, 2025

pazone requested a review from pkoutsovasilis February 21, 2025 14:59

mergify bot assigned pazone Feb 21, 2025

Packaging Ubuntu matrix

2e78431

pkoutsovasilis mentioned this pull request Feb 24, 2025

feat: bump go to 1.24.0 #6932

Open

8 tasks

pazone added 7 commits February 24, 2025 13:46

Improve matrix representation

fa0aa63

Removed matrix packaging if favour to unique step keys

cb26440

Cleanup

97ca59d

Added binfmt

4f97115

Merge branch 'main' into ci_fix_packaging_containers

08efd9f

Pinned binfmt to v8.1.5

dcfd931

Merge branch 'ci_fix_packaging_containers' of github.com:elastic/elas…

57591b2

…tic-agent into ci_fix_packaging_containers

pazone commented Feb 25, 2025

View reviewed changes

Correct step names

841ab70

pazone marked this pull request as ready for review February 26, 2025 00:14

pazone requested a review from a team as a code owner February 26, 2025 00:14

pazone requested a review from dliappis February 26, 2025 00:14

pazone added 3 commits February 26, 2025 10:06

Changed packaging ARM machine type

e8e4546

pinned qemu-v9.2.0-51

0a73c85

Resolved conflicts

cd3e102

Remove qemu

ea79860

oakrizan approved these changes Mar 3, 2025

View reviewed changes

pazone mentioned this pull request Mar 3, 2025

fix(go/base): set GOTOOLCHAIN=local and skip building arm using the base Dockerfile elastic/golang-crossbuild#524

Merged

pazone added 2 commits March 3, 2025 21:16

Resolved conflicts

9cdbb8b

Removed deb and rpm integration packaging

1f76c24

pazone requested a review from a team as a code owner March 3, 2025 20:23

pazone requested a review from kaanyalti March 3, 2025 20:23

pazone added 2 commits March 3, 2025 21:27

Reordered packaging steps

ccce73c

Merge branch 'main' into ci_fix_packaging_containers

8e20566

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Mar 4, 2025

pazone added 5 commits March 4, 2025 12:49

[CI] Build FIPS ARM Linux on ARM machine. Added missing retries

b4c15c4

Resolved conflicts

4c6d45e

Fixed FIPS

a404b1d

[CI] Corrected integration tests packaging dependencies

c84271d

[CI] single buildkite download invocation

cbc3489

Added missing retries

7677e85

dliappis reviewed Mar 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Fixed container images packaging #6972

[CI] Fixed container images packaging #6972

pazone commented Feb 21, 2025 •

edited

Loading

pazone Feb 25, 2025

mergify bot commented Feb 26, 2025

pazone commented Feb 26, 2025

mergify bot commented Mar 3, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

elastic-sonarqube bot commented Mar 4, 2025

dliappis Mar 4, 2025

pazone Mar 4, 2025

dliappis Mar 4, 2025

pazone Mar 4, 2025

dliappis Mar 4, 2025

dliappis Mar 4, 2025

pazone Mar 4, 2025

		command: \|
		buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-x86-64' --build ${BUILDKITE_TRIGGERED_FROM_BUILD_ID}

		@@ -78,6 +84,9 @@ steps:
		provider: "gcp"
		machineType: "n1-standard-8"

[CI] Fixed container images packaging #6972

Are you sure you want to change the base?

[CI] Fixed container images packaging #6972

Conversation

pazone commented Feb 21, 2025 • edited Loading

What does this PR do?

Packaging: linux/amd64 rpm

Packaging: linux/arm64 rpm

Packaging: linux/amd64 deb

Packaging: linux/arm64 deb

Packaging: linux/amd64 tar.gz

Packaging: linux/arm64 tar.gz

Packaging: Containers linux/amd64

Packaging: Containers linux/arm64

Packaging: windows/amd64 zip

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

Choose a reason for hiding this comment

mergify bot commented Feb 26, 2025

pazone commented Feb 26, 2025

mergify bot commented Mar 3, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

⏳ Build in-progress, with failures

Failed CI Steps

History

elastic-sonarqube bot commented Mar 4, 2025

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pazone commented Feb 21, 2025 •

edited

Loading