[Flaky Test]: TestForceInstallOverProtectedPolicy – Policy Response was: {{ 0 false [] { } }}

[pkoutsovasilis]

Failing test case

TestForceInstallOverProtectedPolicy

Error message

Policy Response was: {{ 0 false [] { } }}

Build

https://buildkite.com/elastic/elastic-agent-extended-testing-bk/builds/2011#019525dd-d7fc-4048-adc9-1ddaaa41bb90/336-1001

OS

Linux

Stacktrace and notes

=== Failed
=== FAIL: testing/integration TestForceInstallOverProtectedPolicy (107.61s)
    endpoint_security_test.go:854: Enrolling the agent in Fleet
    endpoint_security_test.go:854: Protected with uninstall token: e30df3f877deab8d335e40198c2e3eef
    tools.go:104: Creating enrollment API key...
    tools.go:95: Creating enrollment API key...
    fixture_install.go:199: [test TestForceInstallOverProtectedPolicy] Inside fixture install function
    fixture_install.go:227: [test TestForceInstallOverProtectedPolicy] Inside fixture installNoPkgManager function
    fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestForceInstallOverProtectedPolicy3080064164
    fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestForceInstallOverProtectedPolicy3080064164
    fixture.go:908: Components were not modified from the fetched artifact
    fixture.go:665: >> running binary with: [/tmp/TestForceInstallOverProtectedPolicy3080064164/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --url https://b8e86b692001e78b7fe16bbdc3e61a2f.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token UnpEb0paVUJSTV9ad2ZxOHQwcGo6Y1N3Q2ZhYVFwNGtacDFTVVE1b2dTdw==]
    tools.go:95: >>> Enroll succeeded. Output: Installing in non-interactive mode.
[    ] Service Started  [4s] Elastic Agent successfully installed, starting enrollment.
[    ] Waiting For Enroll...  [4s] {"log.level":"info","@timestamp":"2025-02-21T00:29:42.919Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"Starting enrollment to URL: https://b8e86b692001e78b7fe16bbdc3e61a2f.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2025-02-21T00:29:44.459Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":487},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2025-02-21T00:29:44.461Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":305},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.
[==  ] Done  [6s]
        Elastic Agent has been successfully installed.
    fixture.go:665: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    check.go:71: Agent fleet status: enrolling Local status: {{f7c0ad8e-333f-44f2-8f60-100e7c99c7d2 9.0.0 9b572efd27e4829dfbf329f38eaed78b6e74c55a 2025-02-21 00:09:22 +0000 UTC true 68304 false} 2 Running [{beat/metrics-monitoring beat/metrics 2 Healthy: communicating with pid '68463' [{beat/metrics-monitoring 1 2 Healthy {}} {beat/metrics-monitoring-metrics-monitoring-beats 0 2 Healthy {}}] {beat-v2-client  {2025-02-19 13:12:51 +0000 UTC c61c0fe44c387bb4559febd02be8d178f4d12ffa}}} {filestream-monitoring filestream 2 Healthy: communicating with pid '68451' [{filestream-monitoring 1 2 Healthy {}} {filestream-monitoring-filestream-monitoring-agent 0 2 Healthy {}}] {beat-v2-client  {2025-02-19 13:12:51 +0000 UTC c61c0fe44c387bb4559febd02be8d178f4d12ffa}}} {http/metrics-monitoring http/metrics 2 Healthy: communicating with pid '68491' [{http/metrics-monitoring 1 2 Healthy {}} {http/metrics-monitoring-metrics-monitoring-agent 0 2 Healthy {}}] {beat-v2-client  {2025-02-19 13:12:51 +0000 UTC c61c0fe44c387bb4559febd02be8d178f4d12ffa}}}] <nil> 2 Connected <nil>}
    endpoint_security_test.go:864: Installing Elastic Defend
    endpoint_security_test.go:865: Templating endpoint package policy request
    endpoint_security_test.go:865: POST /api/fleet/package_policies
    endpoint_security_test.go:865: Error installing fleet package: Error installing endpoint 8.11.0: version_conflict_engine_exception
        	Root causes:
        		version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.11.0]: version conflict, document already exists (current version [1])
    endpoint_security_test.go:866:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1740097271705038246/elastic/elastic-agent-extended-testing-bk/testing/integration/endpoint_security_test.go:866
        	Error:      	Received unexpected error:
        	            	error installing fleet package: Error installing endpoint 8.11.0: version_conflict_engine_exception
        	            		Root causes:
        	            			version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.11.0]: version conflict, document already exists (current version [1])
        	Test:       	TestForceInstallOverProtectedPolicy
        	Messages:   	Policy Response was: {{ 0 false [] { }     }}

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pkoutsovasilis]

@pazone I believe this is caused by the "new" grouping of our CI. Specifically there are two test that run in parallel (each one in different concurrent group) that cause each other to fail as they both try to install the Elastic Defend Package:

The one that fails build/fleet-endpoint-security_sudo_Linux_6.8.0-1021-gcp_x86_64.integration.out.json:

{"Time":"2025-02-21T00:29:17.683833508Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration","Test":"TestForceInstallOverProtectedPolicy","Output":"=== RUN   TestForceInstallOverProtectedPolicy\n"}

The concurrent one that causes the failure of the above build/fleet_sudo_Linux_6.8.0-1021-gcp_x86_64.integration.out.json:

{"Time":"2025-02-21T00:29:07.961023992Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration","Test":"TestEndpointAgentServiceMonitoring","Output":"=== RUN   TestEndpointAgentServiceMonitoring\n"}

Look at the timestamps these are indeed running in parallel. I believe that TestEndpointAgentServiceMonitoring should be in the same group with TestForceInstallOverProtectedPolicy

cc @dliappis