[Flaky Test]: TestSwitchPrivilegedWithBasePath – Received unexpected error: exit status 1

[pkoutsovasilis]

Failing test case

TestSwitchPrivilegedWithBasePath

Error message

Received unexpected error: exit status 1

Build

https://buildkite.com/elastic/elastic-agent-extended-testing-bk/builds/1988#01952460-12d0-4c7c-a666-1d79053f086c/288-934

OS

Linux

Stacktrace and notes

=== Failed
=== FAIL: testing/integration TestSwitchPrivilegedWithBasePath (57.95s)
    fetcher.go:93: Using existing artifact elastic-agent-9.0.0-SNAPSHOT-windows-x86_64.zip
    fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestSwitchPrivilegedWithBasePath3915442953
    fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestSwitchPrivilegedWithBasePath3915442953
    fixture.go:908: Components were not modified from the fetched artifact
    fixture_install.go:199: [test TestSwitchPrivilegedWithBasePath] Inside fixture install function
    fixture_install.go:227: [test TestSwitchPrivilegedWithBasePath] Inside fixture installNoPkgManager function
    fixture.go:665: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestSwitchPrivilegedWithBasePath3915442953\elastic-agent-9.0.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --base-path C:\ --force --unprivileged]
    fixture.go:665: >> running binary with: [C:\Elastic\Agent\elastic-agent.exe privileged -f]
[  ==] Failed to install service  [1s] Error: failed to install service (Elastic Agent): service Elastic Agent already exists
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
    switch_privileged_test.go:134:
        	Error Trace:	C:/buildkite-agent/builds/bk-agent-prod-gcp-1740072248710733418/elastic/elastic-agent-extended-testing-bk/testing/integration/switch_privileged_test.go:134
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	TestSwitchPrivilegedWithBasePath
    fixture_install.go:310: [test TestSwitchPrivilegedWithBasePath] Inside fixture cleanup function
    fixture_install.go:326: collecting diagnostics; test failed
    fixture.go:665: >> running binary with: [C:\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1740072248710733418\elastic\elastic-agent-extended-testing-bk\build\diagnostics\TestSwitchPrivilegedWithBasePath-2025-02-20T17-41-52Z-diagnostics.zip]
    fixture_install.go:744: failed to collect diagnostics to C:\buildkite-agent\builds\bk-agent-prod-gcp-1740072248710733418\elastic\elastic-agent-extended-testing-bk\build\diagnostics\TestSwitchPrivilegedWithBasePath-2025-02-20T17-41-52Z-diagnostics.zip (exit status 1): [WARNING]: failed to fetch agent diagnostics: error in DiagnosticAgent RPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."[WARNING]: failed to fetch unit diagnostics: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."[WARNING]: failed to fetch component diagnostics: error in DiagnosticComponents RPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."Error: failed collecting diagnostics: no diags could be fetched
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
    fixture_install.go:750: retrying in 15 seconds due to connection error; possible Elastic Agent was not fully started
    fixture.go:665: >> running binary with: [C:\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1740072248710733418\elastic\elastic-agent-extended-testing-bk\build\diagnostics\TestSwitchPrivilegedWithBasePath-2025-02-20T17-41-52Z-diagnostics.zip]
    fixture_install.go:754: failed to collect diagnostics a second time at C:\buildkite-agent\builds\bk-agent-prod-gcp-1740072248710733418\elastic\elastic-agent-extended-testing-bk\build\diagnostics\TestSwitchPrivilegedWithBasePath-2025-02-20T17-41-52Z-diagnostics.zip (exit status 1): [WARNING]: failed to fetch agent diagnostics: error in DiagnosticAgent RPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."[WARNING]: failed to fetch unit diagnostics: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."[WARNING]: failed to fetch component diagnostics: error in DiagnosticComponents RPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: open \\\\.\\pipe\\elastic-agent-system: The system cannot find the file specified."Error: failed collecting diagnostics: no diags could be fetched
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pkoutsovasilis]

taking a closer look at the test output in build\default_sudo_Microsoft_Windows_Server_2022_Datacenter_64-bit.integration.out.json reveals the following:

agent/testing/integration","Test":"TestSwitchPrivilegedWithBasePath","Output":"    switch_privileged_test.go:133: privileged output: \r[    ] Stopping service  [0s] \r                              \r\r[    ] Successfully stopped service  [0s] \r                                          \r\r[    ] Adjusting permissions  [0s] \r                                          \r\r[  ==] Removing service  [1s] \r                                          \r\r[  ==] Installing service  [1s] \r                                          \r\r[  ==] Failed to install service  [1s] Error: failed to install service (Elastic Agent): service Elastic Agent already exists\n"}

sentence of interest is failed to install service (Elastic Agent): service Elastic Agent already exists. Unfortunately we explicitly ignore any error that comes from UninstallService for this reason:

// error is ignored because it's possible that its already uninstalled
//
// this can happen if this action failed in the middle of this critical section, so to allow the
// command to be called again we don't error on the uninstall
//
// the install error below will include an error about the service still existing if this failed
// to uninstall (really this should never fail, but the unexpected can happen) 

which IMO is not that helpful understanding why this failed. Printing a warning message here would be helpful instead of ignoring the error completely. All that said, I believe that this issue might be affected by the windows permission bug we have here