[Flaky Test]: TestInstallWithEndpointSecurityAndRemoveEndpointIntegration – Error installing fleet package: Error installing endpoint 8.11.0: runtime_exception

[pchila]

Failing test case

TestInstallWithEndpointSecurityAndRemoveEndpointIntegration

Error message

version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_united-default-8.11.0]: version conflict, document already exists (current version [1])

Build

https://buildkite.com/elastic/elastic-agent/builds/6612

OS

Linux

Stacktrace and notes

There are multiple Endpoint Security integration failures in the same run but they seem related:

- TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/unprotected
  
   endpoint_security_test.go:370: POST /api/fleet/package_policies
    endpoint_security_test.go:370: Error installing fleet package: Error installing endpoint 8.11.0: runtime_exception
        	Caused by:
        		version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_united-default-8.11.0]: version conflict, document already exists (current version [1])
        	Root causes:
        		runtime_exception: Failed to persist transform statistics for transform [endpoint.metadata_united-default-8.11.0]
    endpoint_security_test.go:371: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:371
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:143
        	Error:      	Received unexpected error:
        	            	error installing fleet package: Error installing endpoint 8.11.0: runtime_exception
        	            		Caused by:
        	            			version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_united-default-8.11.0]: version conflict, document already exists (current version [1])
        	            		Root causes:
        	            			runtime_exception: Failed to persist transform statistics for transform [endpoint.metadata_united-default-8.11.0]
        	Test:       	TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/unprotected
        	Messages:   	Policy Response was: kibana.PackagePolicyResponse{Item:kibana.PackagePolicy{ID:"", Revision:0, Enabled:false, Inputs:[]map[string]interface {}(nil), Package:kibana.PackagePolicyRequestPackage{Name:"", Version:""}, Namespace:"", OutputID:"", PolicyID:"", Name:"", Description:""}}

• TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected

  endpoint_security_test.go:370: POST /api/fleet/package_policies
    endpoint_security_test.go:370: Error installing fleet package: Error installing endpoint 8.11.0: resource_not_found_exception
        	Root causes:
        		resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found
    endpoint_security_test.go:371: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:371
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:143
        	Error:      	Received unexpected error:
        	            	error installing fleet package: Error installing endpoint 8.11.0: resource_not_found_exception
        	            		Root causes:
        	            			resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found
        	Test:       	TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected
        	Messages:   	Policy Response was: kibana.PackagePolicyResponse{Item:kibana.PackagePolicy{ID:"", Revision:0, Enabled:false, Inputs:[]map[string]interface {}(nil), Package:kibana.PackagePolicyRequestPackage{Name:"", Version:""}, Namespace:"", OutputID:"", PolicyID:"", Name:"", Description:""}}
  

• TestInstallAndUnenrollWithEndpointSecurity/unprotected

      endpoint_security_test.go:257: POST /api/fleet/package_policies
    endpoint_security_test.go:257: Error installing fleet package: Error installing endpoint 8.11.0: runtime_exception
        	Caused by:
        		version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.11.0]: version conflict, document already exists (current version [1])
        	Root causes:
        		runtime_exception: Failed to persist transform statistics for transform [endpoint.metadata_current-default-8.11.0]
    endpoint_security_test.go:258: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:258
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:115
        	Error:      	Received unexpected error:
        	            	error installing fleet package: Error installing endpoint 8.11.0: runtime_exception
        	            		Caused by:
        	            			version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.11.0]: version conflict, document already exists (current version [1])
        	            		Root causes:
        	            			runtime_exception: Failed to persist transform statistics for transform [endpoint.metadata_current-default-8.11.0]
  

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

I also see:

=== RUN   TestStandaloneUpgradeRollbackOnRestarts
    upgrade_rollback_test.go:170: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_rollback_test.go:170
        	Error:      	Received unexpected error:
        	            	error retrieving versions from Artifact API: 503: bad http status code
        	Test:       	TestStandaloneUpgradeRollbackOnRestarts
--- FAIL: TestStandaloneUpgradeRollbackOnRestarts (0.05s)

I suspect that the resource not found error might be related, perhaps this is what Fleet gives us if it can't get the package artifacts?

    endpoint_security_test.go:370: POST /api/fleet/package_policies
    endpoint_security_test.go:370: Error installing fleet package: Error installing endpoint 8.11.0: resource_not_found_exception
        	Root causes:
        		resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found
    endpoint_security_test.go:371: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/endpoint_security_test.go:371
        	            				/home/ubuntu/agent/testing/integration/endpoint_security_test.go:143
        	Error:      	Received unexpected error:
        	            	error installing fleet package: Error installing endpoint 8.11.0: resource_not_found_exception
        	            		Root causes:
        	            			resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found
        	Test:       	TestInstallWithEndpointSecurityAndRemoveEndpointIntegration/protected
        	Messages:   	Policy Response was: kibana.PackagePolicyResponse{Item:kibana.PackagePolicy{ID:"", Revision:0, Enabled:false, Inputs:[]map[string]interface {}(nil), Package:kibana.PackagePolicyRequestPackage{Name:"", Version:""}, Namespace:"", OutputID:"", PolicyID:"", Name:"", Description:""}}

I suspect the version conflict errors probably follow from this incomplete package installation.

@kpollich any thoughts on these Fleet errors?

[kpollich]

resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found

This indicates to me that while installing endpoint we tried to fetch or start a transform that doesn't exist. The resource_not_found error is Elasticsearch's error text for a 404.

This could even trigger from the cleanup logic where we deleted previous version transforms during an install/upgrade of a package that includes transform assets:

https://github.com/elastic/kibana/blob/ace2b78cd8edcdda10f803019716cc59d74bd58b/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts#L460-L476

My suspicion here is that asset reference to this transform is still present even though the transform was never created in this case. So, when we try to "clean up" that existing transform we get this 404.

version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_united-default-8.11.0]: version conflict, document already exists (current version [1])

I am not familiar enough with Elasticsearch transforms to know what this might be referring to, so I'm going to ping @qn895 who implemented almost all of Fleet's transform management logic. Maybe her expertise here can point us in the right direction 🤞

[qn895]

Currently investigating this... @pchila, is this a problem only with your PR or across multiple other PRs as well?

[cmacknz]

This happens inconsistently across different branches. Nothing in this repository changes Fleet or Kibana at all, all we are doing is installing and uninstalling packages using the package policies API.

We may have created an unintentional stress test by installing and uninstall the Elastic Defend integration rapidly, where rapidly is something like once per minute over 5 minutes. So faster than most users would ever do, but not unreasonably quickly.

[pchila]

@qn895 I only saw that error in that specific run of the PR CI build. It came up after another run where the artifact API misbehaved a lot with 503 http status codes.
As new procedure for failures in our CI I filed an issue for that but as @cmacknz was saying it may be caused by external factors.

I didn't see it happen since and the next run for the same code was green 🤷‍♂️

[qn895]

@cmacknz @pchila Thanks all for the context. I'll try to reproduce it, and will reach out for questions.

[przemekwitek]

I am not familiar enough with Elasticsearch transforms to know what this might be referring to

This error is coming from the transform code:
https://github.com/elastic/elasticsearch/blob/42f3a8f6ca57e52d2856c2d3b7c59cbce9ac3689/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/persistence/IndexBasedTransformConfigManager.java#L716
It happens when there is a race condition and 2 threads try to index the new version of transform stats document at the same time.
Maybe we could handle these conflicts better in the backend. But first I need to understand what exact scenario is being executed.
Could you point me at the test scenario code? Is it in endpoint_security_test.go file?

[cmacknz]

Yes this is the function we use to install the Elastic Defend package

elastic-agent/testing/integration/endpoint_security_test.go

Line 452 in d8b446f

func installElasticDefendPackage(t *testing.T, info *define.Info, policyID string) (r kibana.PackagePolicyResponse, err error) {

This is the package template we render and POST to /api/fleet/package_policies https://github.com/elastic/elastic-agent/blob/main/testing/integration/endpoint_security_package.json.tmpl

We have several tests that enroll an agent into a new agent policy within a short time. We likely end up with Elastic Defend installed 10-20 times on the same Fleet instance with each variant of the integration in its own agent policy.

[cmacknz]

@qn895 not sure if there's been any progress here, but we uncovered a similar bug in Fleet related to concurrently installing the same package.

#4102 (comment)

I have a suspicion the root cause is probably similar here, since concurrently installing the same package many times in a short duration seems to be the pattern in our tests leading to problems on the Kibana side.

[qn895]

Thanks @cmacknz for the context. I'm planning to investigate this more thoroughly tomorrow and the upcoming week. Will also sync up with @przemekwitek what's the best approach to fix this issue moving forward.

[rdner]

@qn895 any update on the investigation?

[qn895]

Hi @rdner. Sorry I was out yesterday. This was rather hard for me to reproduce even using the steps here #4102 (comment). My methodology was to use the API to create integration policies and install the packages in rapid succession, and it looks to me there's similar errors outside of Transforms. For example, I got:

[ERROR][plugins.securitySolution] ResponseError: [endpoint:user-artifact-manifest:endpoint-manifest-v1]: version conflict, document already exists (current version [1]): version_conflict_engine_exception
        Root causes:
                version_conflict_engine_exception: [endpoint:user-artifact-manifest:endpoint-manifest-v1]: version conflict, document already exists (current version [1])
    at KibanaTransport.request (/Users/quynh/Documents/projects/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Which to me raise the question if the unintentional stress testing is potentially affecting other types of assets as well outside of Transforms. For Transforms specifically, I see two categories of errors:

1. version_conflict_engine_exception: [data_frame_transform_state_and_stats-endpoint.metadata_current-default-8.11.0]: version conflict, document already exists (current version [1])

This will require @przemekwitek to investigate a bit further on the Elasticsearch side to see if there's anything we can do to handle this better. We can also choose to ignore this error on the Fleet Kibana's installation side, and I can create a PR for that.

2. resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found

This is happening during the install package phase, is something that should be non-blocking. I can create a PR to better handle this scenario.

So to recap: For the short term, I will create a Kibana PR that would no longer bubble up the errors we are seeing to no longer block the uninstall/install process. That should hopefully fix some of the errors we are seeing. However, I think the process might highlight issues that affect other asset types other than Transforms. In the longer term, would also like to know what's the best practice for handling assets in Fleet in scenarios where uninstall/install is rapidly triggered.

[rdner]

For the short term, I will create a Kibana PR that would no longer bubble up the errors we are seeing to no longer block the uninstall/install process.

@qn895 thank you! please link your PR to this issue.

In the longer term, would also like to know what's the best practice for handling assets in Fleet in scenarios where uninstall/install is rapidly triggered.

For me, this sounds similar to another issue we had we package installation, here is a short summary #4102 (comment)

Perhaps looking into the fix for that issue can help.

[qn895]

@rdner @cmacknz Created PR for it here: elastic/kibana#177380. Hopefully that should resolves this flaky test issue. Appreciate any extra testing that the team can do with this PR as I had a hard time replicating the same error even with API calls in quick sequence or concurrently:

So replicate this you need API calls to:
Create a new empty agent policy with no integrations.
Create a new installation of Elastic Defend by making a call to the package policy API for this newly created policy.

[rdner]

Failed again in https://buildkite.com/elastic/elastic-agent/builds/7632#018e1a5c-a65d-4d5a-9d4d-1feeca31c961

@qn895 I see that the PR is still in draft after 2 weeks, any updates?

[qn895]

Sorry I was out the last two weeks. I'll open the PR up for review tomorrow, but would appreciate some additional testing before updating all the Kibana unit tests. The main goal with this PR is to isolate whether this is something that can be fixed on Kibana's end, or Elasticsearch's end.

[rdner]

Another failure in https://buildkite.com/elastic/elastic-agent/builds/7782#018e3f6a-f5a4-4899-b60d-9d719bd14233