[Flaky Test]: TestLogIngestionFleetManaged, TestDebLogIngestFleetManaged – list of error messages is expected to be empty

[belimawr]

Failing test case

TestLogIngestionFleetManaged, TestDebLogIngestFleetManaged

Error message

Should be empty, but was

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2207

OS

Linux

Stacktrace and notes

(linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): === Failed
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): === FAIL: testing/integration TestLogIngestionFleetManaged/Monitoring_logs_are_shipped (30.58s)
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:138: Making sure metricbeat logs are populated
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:142: metricbeat: Got 46 documents
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:147: Making sure all components are healthy
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:158: Making sure there are no error logs
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:175: error logs: Got 1 documents
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:178: map[string]interface {}{"@timestamp":"2024-08-27T15:39:48.588Z", "agent":map[string]interface {}{"ephemeral_id":"bba78f69-f68b-456f-8664-052c3e5a5ae8", "id":"5ee3ab82-5d6a-4713-9a36-40047846650e", "name":"ogc-linux-amd64-ubuntu-2404-fleet-cde6", "type":"filebeat", "version":"8.16.0"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"8363546705712402721", "name":"ogc-linux-amd64-ubuntu-2404-fleet-cde6"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"kgvxjqex3ah68m1bah1k7m3p8v6ctey5orf4xaro", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"5ee3ab82-5d6a-4713-9a36-40047846650e", "snapshot":true, "version":"8.16.0"}, "event":map[string]interface {}{"dataset":"elastic_agent.metricbeat"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2404-fleet-cde6", "id":"f330ede3e09c4f8e89d44db21af07310", "ip":[]interface {}{"10.128.0.48", "fe80::4001:aff:fe80:30"}, "mac":[]interface {}{"42-01-0A-80-00-30"}, "name":"ogc-linux-amd64-ubuntu-2404-fleet-cde6", "os":map[string]interface {}{"codename":"noble", "family":"debian", "kernel":"6.8.0-1013-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"24.04 LTS (Noble Numbat)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"1070250", "path":"/opt/Elastic/Agent/data/elastic-agent-8.16.0-SNAPSHOT-25ce66/logs/elastic-agent-20240827.ndjson"}, "offset":63236, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"publisher_pipeline_output", "log.origin":map[string]interface {}{"file.line":148, "file.name":"pipeline/client_worker.go", "function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"}, "message":"Failed to connect to backoff(elasticsearch(http://127.0.0.1:9200)): Get "http://127.0.0.1:9200\": dial tcp 127.0.0.1:9200: connect: connection refused", "service.name":"metricbeat"}
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): logs_ingestion_test.go:189:
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): Error Trace: /home/ubuntu/agent/testing/integration/logs_ingestion_test.go:189
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): /home/ubuntu/agent/testing/integration/logs_ingestion_test.go:104
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): Error: Should be empty, but was [{.ds-logs-elastic_agent.metricbeat-kgvxjqex3ah68m1bah1k7m3p8v6ctey5orf4xaro-2024.08.27-000001 2.9783452 map[@timestamp:2024-08-27T15:39:48.588Z agent:map[ephemeral_id:bba78f69-f68b-456f-8664-052c3e5a5ae8 id:5ee3ab82-5d6a-4713-9a36-40047846650e name:ogc-linux-amd64-ubuntu-2404-fleet-cde6 type:filebeat version:8.16.0] cloud:map[account:map[id:elastic-platform-ingest] availability_zone:us-central1-a instance:map[id:8363546705712402721 name:ogc-linux-amd64-ubuntu-2404-fleet-cde6] machine:map[type:e2-standard-2] project:map[id:elastic-platform-ingest] provider:gcp region:us-central1 service:map[name:GCE]] component:map[binary:metricbeat dataset:elastic_agent.metricbeat id:system/metrics-default type:system/metrics] data_stream:map[dataset:elastic_agent.metricbeat namespace:kgvxjqex3ah68m1bah1k7m3p8v6ctey5orf4xaro type:logs] ecs:map[version:8.0.0] elastic_agent:map[id:5ee3ab82-5d6a-4713-9a36-40047846650e snapshot:true version:8.16.0] event:map[dataset:elastic_agent.metricbeat] host:map[architecture:x86_64 containerized:false hostname:ogc-linux-amd64-ubuntu-2404-fleet-cde6 id:f330ede3e09c4f8e89d44db21af07310 ip:[10.128.0.48 fe80::4001:aff:fe80:30] mac:[42-01-0A-80-00-30] name:ogc-linux-amd64-ubuntu-2404-fleet-cde6 os:map[codename:noble family:debian kernel:6.8.0-1013-gcp name:Ubuntu platform:ubuntu type:linux version:24.04 LTS (Noble Numbat)]] input:map[type:filestream] log:map[file:map[device_id:2049 inode:1070250 path:/opt/Elastic/Agent/data/elastic-agent-8.16.0-SNAPSHOT-25ce66/logs/elastic-agent-20240827.ndjson] offset:63236 source:system/metrics-default] log.level:error log.logger:publisher_pipeline_output log.origin:map[file.line:148 file.name:pipeline/client_worker.go function:github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run] message:Failed to connect to backoff(elasticsearch(http://127.0.0.1:9200)): Get "http://127.0.0.1:9200": dial tcp 127.0.0.1:9200: connect: connection refused service.name:metricbeat]}]
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): Test: TestLogIngestionFleetManaged/Monitoring_logs_are_shipped
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): Messages: list of error messages is expected to be empty, found:
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): Failed to connect to backoff(elasticsearch(http://127.0.0.1:9200)): Get "http://127.0.0.1:9200": dial tcp 127.0.0.1:9200: connect: connection refused
2024-08-27 15:41:15 UTC >>> (linux-amd64-ubuntu-2404-fleet) Test output (sudo) (stdout): === FAIL: testing/integration TestLogIngestionFleetManaged (118.35s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)