[Flaky Test]: TestFleetAirGappedUpgradePrivileged – Agent status is not online #6924
Assignees
Labels
Comments
pkoutsovasilis
added
flaky-test
|
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
pkoutsovasilis
changed the title
|
Looking at the logs to the most recent failure, it doesn't even seem like the Elastic Agent has upgraded at all. |
8 tasks
|
Should we temporarily skip this test? It blocks a few PRs |
|
I got the same error but because the unprivileged test: It seems to be the same issue, so I believe we can also track it here. What do you think? |
|
Yes I believe all test failures that are |
8 tasks
4 tasks
This was referenced Feb 28, 2025
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Failing test case
TestFleetAirGappedUpgradePrivileged
Error message
Agent status is not online
Build
https://buildkite.com/elastic/elastic-agent-extended-testing-bk/builds/1869#019518fb-ffb9-4b1f-8a3b-98ac6375a598/347-1141
OS
Windows
Stacktrace and notes
=== FAIL: testing/integration TestFleetAirGappedUpgradePrivileged (829.92s) fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:292: Downloading artifact progress 3.87% fetcher_artifact.go:292: Downloading artifact progress 7.24% fetcher_artifact.go:292: Downloading artifact progress 11.10% fetcher_artifact.go:292: Downloading artifact progress 14.51% fetcher_artifact.go:292: Downloading artifact progress 18.00% fetcher_artifact.go:292: Downloading artifact progress 20.76% fetcher_artifact.go:292: Downloading artifact progress 24.11% fetcher_artifact.go:292: Downloading artifact progress 27.65% fetcher_artifact.go:292: Downloading artifact progress 31.49% fetcher_artifact.go:292: Downloading artifact progress 35.18% fetcher_artifact.go:292: Downloading artifact progress 38.40% fetcher_artifact.go:292: Downloading artifact progress 42.38% fetcher_artifact.go:292: Downloading artifact progress 45.67% fetcher_artifact.go:292: Downloading artifact progress 49.25% fetcher_artifact.go:292: Downloading artifact progress 52.79% fetcher_artifact.go:292: Downloading artifact progress 56.52% fetcher_artifact.go:292: Downloading artifact progress 60.04% fetcher_artifact.go:292: Downloading artifact progress 63.45% fetcher_artifact.go:292: Downloading artifact progress 67.44% fetcher_artifact.go:292: Downloading artifact progress 70.24% fetcher_artifact.go:292: Downloading artifact progress 73.60% fetcher_artifact.go:292: Downloading artifact progress 77.47% fetcher_artifact.go:292: Downloading artifact progress 80.93% fetcher_artifact.go:292: Downloading artifact progress 84.30% fetcher_artifact.go:292: Downloading artifact progress 87.71% fetcher_artifact.go:292: Downloading artifact progress 90.97% fetcher_artifact.go:292: Downloading artifact progress 94.16% fetcher_artifact.go:292: Downloading artifact progress 98.12% fetcher_artifact.go:292: Downloading artifact progress 100.00% fetcher_artifact.go:292: Downloading artifact progress 100.00% fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradePrivileged2678865022 fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradePrivileged2678865022 fixture.go:908: Components were not modified from the fetched artifact fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:292: Downloading artifact progress 31.79% fetcher_artifact.go:292: Downloading artifact progress 33.24% fetcher_artifact.go:292: Downloading artifact progress 33.24% fetcher_artifact.go:292: Downloading artifact progress 33.24% fetcher_artifact.go:292: Downloading artifact progress 40.46% fetcher_artifact.go:292: Downloading artifact progress 40.46% fetcher_artifact.go:292: Downloading artifact progress 40.46% fetcher_artifact.go:292: Downloading artifact progress 63.59% fetcher_artifact.go:292: Downloading artifact progress 100.00% fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc ArtifactsServer root dir "/tmp/TestFleetAirGappedUpgradePrivileged249152131/001", served files ["elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz" "elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc" "elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512"] upgrade_fleet_test.go:495: found [34.120.127.130 2600:1901:0:1d7::] IPs for "artifacts.elastic.co", blocking them... upgrade_fleet_test.go:514: added iptables rule [OUTPUT -j DROP -d 34.120.127.130] upgrade_fleet_test.go:518: use "iptables -D OUTPUT -j DROP -d 34.120.127.130" to remove it upgrade_fleet_test.go:514: added iptables rule [OUTPUT -j DROP -d 2600:1901:0:1d7::] upgrade_fleet_test.go:518: use "ip6tables -D OUTPUT -j DROP -d 2600:1901:0:1d7::" to remove it fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradePrivileged1950969600 fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradePrivileged1950969600 fixture.go:908: Components were not modified from the fetched artifact upgrade_fleet_test.go:325: Testing Elastic Agent upgrade from 9.0.0-SNAPSHOT to 9.0.0-SNAPSHOT with Fleet... upgrade_fleet_test.go:333: creating download source "local-air-gaped-d7986e04-1f0d-4435-9b95-b31e31b12547", using "http://127.0.0.1:34389/downloads/beats/elastic-agent/". fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradePrivileged1950969600/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml] fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradePrivileged2678865022/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml] upgrade_fleet_test.go:375: Creating Agent policy... upgrade_fleet_test.go:380: Creating Agent enrollment API key... upgrade_fleet_test.go:387: Getting default Fleet Server URL... upgrade_fleet_test.go:391: Installing Elastic Agent (unprivileged: false)... fixture_install.go:199: [test TestFleetAirGappedUpgradePrivileged] Inside fixture install function fixture_install.go:227: [test TestFleetAirGappedUpgradePrivileged] Inside fixture installNoPkgManager function fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradePrivileged1950969600/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --url https://d7f759ef35dedca1f31ae9a719e282a2.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token czd3SUdaVUJwWWpOX05UeTFUa2M6TzhqQzZjWFl4ZzBubzhKY1ZVbEgtdw==] upgrade_fleet_test.go:408: Waiting for Agent to be correct version and healthy... fixture.go:665: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json] upgrade_fleet_test.go:412: Waiting for enrolled Agent status to be online... upgrade_fleet_test.go:420: Upgrading from version "9.0.0-SNAPSHOT-831032e2355bcdc0906962c926890d671b990a60" to version "9.0.0-SNAPSHOT-8200a86e0d3c7431b020aefbfe02d7f673254c19"... upgrade_fleet_test.go:426: Waiting from upgrade details to show up in Fleet upgrade_fleet_test.go:439: Waiting for upgrade watcher to start... upgrade_fleet_test.go:442: Upgrade watcher started fixture.go:665: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json] upgrade_fleet_test.go:448: Waiting for enrolled Agent status to be online... check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] check.go:56: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1739881246704742035.c.elastic-ci-prod.internal] for policy [418e7d57-7efd-4153-94f5-8d6326e1f80f]. Found: [] upgrade_fleet_test.go:449: Error Trace: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1739881246704742035/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:449 /opt/buildkite-agent/builds/bk-agent-prod-gcp-1739881246704742035/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:341 /opt/buildkite-agent/builds/bk-agent-prod-gcp-1739881246704742035/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:132 Error: Condition never satisfied Test: TestFleetAirGappedUpgradePrivileged Messages: Agent status is not online fixture_install.go:307: [test TestFleetAirGappedUpgradePrivileged] Inside fixture cleanup function fixture_install.go:323: collecting diagnostics; test failedThe text was updated successfully, but these errors were encountered: