[Flaky Test]: TestEventLogOutputConfiguredViaFleet – could not find event log file

[rdner]

Failing test case

TestEventLogOutputConfiguredViaFleet

Error message

could not find event log file

Build

• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1274
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1309
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1333
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1341
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1346
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1373
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2086#01917b0e-553b-4fd2-a2d1-4f224c11c6ed
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2088#01917b0e-e4e4-4d20-af8a-2bd4b206b9e1

OS

Linux

Stacktrace and notes

=== RUN   TestEventLogOutputConfiguredViaFleet
    container_cmd_test.go:75: Creating enrollment API key...
    fetcher.go:95: Using existing artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:282: Extracting artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestEventLogOutputConfiguredViaFleet188574346/001
    fixture.go:300: Completed extraction of artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestEventLogOutputConfiguredViaFleet188574346/001
    fixture.go:906: Components were not modified from the fetched artifact
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    fixture.go:657: >> running binary with: [/tmp/TestEventLogOutputConfiguredViaFleet188574346/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent status --output json]
    event_logging_test.go:316: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/event_logging_test.go:316
        	            				/home/ubuntu/agent/testing/integration/event_logging_test.go:231
        	Error:      	Condition never satisfied
        	Test:       	TestEventLogOutputConfiguredViaFleet
        	Messages:   	could not find event log file
    container_cmd_test.go:98: >> cleaning up: killing the Elastic-Agent process
--- FAIL: TestEventLogOutputConfiguredViaFleet (89.09s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pierrehilbert]

This has been introduced by #4932
@belimawr mentioned before going into PTO that it seems to be working when run alone but to be flaky when run with all the other tests so we probably have some conflicts between tests somewhere

[belimawr]

The error reported here is different from the one fixed by #5341

[ycombinator]

More recent builds failing on this test + assertion:

• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2086#01917b0e-553b-4fd2-a2d1-4f224c11c6ed
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2088#01917b0e-e4e4-4d20-af8a-2bd4b206b9e1

Added to list in issue description.

[ycombinator]

This test is failing quite regularly on PRs so I'm going to disable it now to unblock PRs that have been blocked for a while, e.g. #5267.. Never mind, it's skipped already (since yesterday). Some older PRs just need to be rebased on main.

[belimawr]

How to reproduce

(Spoiler alert: the test itself is not flaky, it's just reading the "wrong configuration" 🤯)

1. Create /usr/share/elastic-agent/state/container-paths.yml containing:

state_path: /usr/share/elastic-agent/state
config_path: /usr/share/elastic-agent/state
socket_path: unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock

2. Download the Elastic-Agent and extract the tar.gz
3. Enter the folder and run the elastic-agent binary as root (root is not required, but it will prevent permission issues): ./elastic-agent
4. This will cause the logger path to be /usr/share/elastic-agent/state/data/logs
5. This will make the Elastic-Agent to use the configuration/fleet.enc from /usr/share/elastic-agent/state/ instead of starting with an empty configuration as expected.

[belimawr]

Another way to reproduce the issue is to create /usr/share/elastic-agent/state/container-paths.yml as shown in my last post, then:

1. Unpack an Elastic-Agent
2. Create any Policy in Fleet
3. Enrol the Elastic-Agent in Fleet
4. Collect the diagnostics
5. The path map in computed-config.yaml will look like this:

path:
    config: /usr/share/elastic-agent/state
    data: /usr/share/elastic-agent/state/data
    home: /usr/share/elastic-agent/state/data
    logs: /home/ubuntu/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64

Even though it shows path.logs: /home/ubuntu/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64 the logs are actually stored at /usr/share/elastic-agent/state/data/logs/.

[belimawr]

The same happens if the Elastic-Agent is installed directly the paths is:

path:
    config: /usr/share/elastic-agent/state
    data: /usr/share/elastic-agent/state/data
    home: /usr/share/elastic-agent/state/data
    logs: /opt/Elastic/Agent

But the logs are at /usr/share/elastic-agent/state/data/logs/

[VihasMakwana]

@belimawr I think we should backport this fix as well. I faced the failure in my backport PR

You've already done that. Thanks!

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2195#0191929b-a7df-495b-b7c4-3fd3d0221093