Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky Test]: TestStandaloneUpgradeUninstallKillWatcher – TarInstaller: creating file open package.version: permission denied #4143

Closed
cmacknz opened this issue Jan 25, 2024 · 8 comments
Closed

[Flaky Test]: TestStandaloneUpgradeUninstallKillWatcher – TarInstaller: creating file open package.version: permission denied #4143

cmacknz opened this issue Jan 25, 2024 · 8 comments
Assignees
@rdner
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team

Comments

@cmacknz
Copy link
Member

cmacknz commented Jan 25, 2024 •
edited
Loading

Failing test case

TestStandaloneUpgradeUninstallKillWatcher

Error message

Failed trigger upgrade of daemon: TarInstaller: creating file /opt/Elastic/Agent/data/elastic-agent-cb1c24/package.version: open /opt/Elastic/Agent/data/elastic-agent-cb1c24/package.version: permission denied

Build

Main: https://buildkite.com/elastic/elastic-agent/builds/6552#018d412d-b1b7-4958-8294-b8f761011c38
Branch: https://buildkite.com/elastic/elastic-agent/builds/6555#018d4136-3471-4cbb-b66e-3cab14032f1a

OS

Linux

Stacktrace and notes

=== RUN   TestStandaloneUpgradeUninstallKillWatcher
    fetcher.go:91: Using existing artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz
    fixture.go:261: Extracting artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/001
    fixture.go:274: Completed extraction of artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/001
    fixture.go:829: Components were not modified from the fetched artifact
    fixture.go:624: >> running binary with: [/tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/001/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent version --binary-only --yaml]
    fetcher.go:91: Using existing artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz
    fixture.go:261: Extracting artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/002
    fixture.go:274: Completed extraction of artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/002
    fixture.go:829: Components were not modified from the fetched artifact
    fixture.go:624: >> running binary with: [/tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent version --binary-only --yaml]
    fixture.go:624: >> running binary with: [/tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/001/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent version --binary-only --yaml]
    upgrader.go:205: installation of Elastic Agent will use --unprivileged as both start and end version support --unprivileged mode
    upgrader.go:243: Installing version "8.13.0-SNAPSHOT"
    fixture_install.go:125: [test TestStandaloneUpgradeUninstallKillWatcher] Inside fixture install function
    fixture.go:624: >> running binary with: [/tmp/TestStandaloneUpgradeUninstallKillWatcher3431983411/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent install --force --non-interactive --unprivileged]
    fixture.go:624: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrader.go:279: Upgrading from version "8.13.0-SNAPSHOT-249430b3493fe2ca9ebfde9e72b91b9dd49b2d2e" to version "8.13.0-SNAPSHOT-cb1c24fa0a4144d152603e1c01f0ea3a3c7ee1a0"
    fixture.go:624: >> running binary with: [/opt/Elastic/Agent/elastic-agent upgrade 8.13.0-SNAPSHOT --source-uri file:///tmp/agent-upgrade-934987204 --skip-verify]
    upgrade_uninstall_test.go:67: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_uninstall_test.go:67
        	Error:      	Received unexpected error:
        	            	failed to start agent upgrade to version "8.13.0": exit status 1
        	            	Error: Failed trigger upgrade of daemon: TarInstaller: creating file /opt/Elastic/Agent/data/elastic-agent-cb1c24/package.version: open /opt/Elastic/Agent/data/elastic-agent-cb1c24/package.version: permission denied
        	            	For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
        	Test:       	TestStandaloneUpgradeUninstallKillWatcher
    fixture_install.go:165: [test TestStandaloneUpgradeUninstallKillWatcher] Inside fixture cleanup function
    fixture_install.go:180: collecting diagnostics; test failed
    fixture.go:624: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /home/ubuntu/agent/build/diagnostics/TestStandaloneUpgradeUninstallKillWatcher-diagnostics-2024-01-25T15:46:04Z.zip]
    fixture.go:624: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
--- FAIL: TestStandaloneUpgradeUninstallKillWatcher (54.54s)
@cmacknz cmacknz added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels Jan 25, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@cmacknz
Copy link
Member Author

cmacknz commented Jan 25, 2024

The change most closely related to this area is #4100. It's not clear exactly what the mechanism for the failure is yet.

@cmacknz cmacknz mentioned this issue Jan 25, 2024
Merged
7 tasks
@pchila
Copy link
Member

pchila commented Jan 25, 2024 •
edited
Loading

This is the same symptom fixed by #4100
However the latest version of elastic agent pulled from artifact API is before that change (we pull a version corresponding to commit 249430b ) hence the issue.
We need to check if it happens again now that we may have new unified release builds after the whole docker debacle

@cmacknz cmacknz mentioned this issue Jan 25, 2024
Merged
7 tasks
@rdner
Copy link
Member

rdner commented Jan 26, 2024

@leehinman @cmacknz #4146 does not seem to fix the issue, the build on main is still failing with the same error https://buildkite.com/elastic/elastic-agent/builds/6579#018d425a-b7e0-4794-a715-6599232f289f

@pchila
Copy link
Member

pchila commented Jan 26, 2024

@rdner we need a newer version on artifact API and that happens only after a unified release build.
The build you linked still uses an old commit 249430b (taken from the log below)

 upgrader.go:279: Upgrading from version "8.13.0-SNAPSHOT-249430b3493fe2ca9ebfde9e72b91b9dd49b2d2e" to version "8.13.0-SNAPSHOT-5f578c5d4cc65371d9a6cee31fb22edf6150960d"

What matters is the starting version, as that is the one that controls the whole upgrade process and the one we retrieved from artifact API does not have the fix.

@rdner
Copy link
Member

rdner commented Jan 26, 2024

Looks like the last build succeeded https://buildkite.com/elastic/elastic-agent/builds/6589
So, the fix @pchila mentioned in #4143 (comment) seems to be working.

@rdner rdner closed this as completed Jan 26, 2024
@rdner rdner self-assigned this Jan 26, 2024
@rdner rdner mentioned this issue Jan 31, 2024
Closed
@rdner
Copy link
Member

rdner commented Jan 31, 2024 •
edited
Loading

@pchila is this issue back or it's a different one here #4164 ?

This time it's failing on serverless integration tests.

@pchila
Copy link
Member

pchila commented Jan 31, 2024

The issue is different: in #4164 we are failing to uninstall the agent (agent.exe is busy)
This issue was about upgrading when in unprivileged mode due to wrong permissions assigned on containing directories for package.version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rdner rdner

Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rdner @cmacknz @elasticmachine @pchila