[Flaky Test]: TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped – checkin retry loop was stopped

[rdner]

Failing test case

TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped

Error message

checkin retry loop was stopped

Build

• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1962#0191587e-8807-4c94-a4d4-1c8439f3a164
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1963
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1970
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1977
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1985

OS

Linux

Stacktrace and notes

=== RUN   TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped
    logs_ingestion_test.go:138: Making sure metricbeat logs are populated
    logs_ingestion_test.go:142: metricbeat: Got 300 documents
    logs_ingestion_test.go:147: Making sure all components are healthy
    logs_ingestion_test.go:158: Making sure there are no error logs
    logs_ingestion_test.go:175: error logs: Got 1 documents
    logs_ingestion_test.go:178: map[string]interface {}{"@timestamp":"2024-08-16T01:28:16.509Z", "agent":map[string]interface {}{"ephemeral_id":"434f3652-3c21-4662-a1d5-b424f491ae70", "id":"81fe984b-1f65-4d0c-b4e6-4f53aa583fdb", "name":"ogc-linux-amd64-rhel-8-rpm-16dc", "type":"filebeat", "version":"8.16.0"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-f", "instance":map[string]interface {}{"id":"6264460097818400828", "name":"ogc-linux-amd64-rhel-8-rpm-16dc"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "data_stream":map[string]interface {}{"dataset":"elastic_agent", "namespace":"nd1rpyfwjgb7tjcrv4h9ujuvtwsqt2tinojxhcr1om", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"81fe984b-1f65-4d0c-b4e6-4f53aa583fdb", "snapshot":true, "version":"8.16.0"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent", "ingested":"2024-08-16T01:28:57Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-rhel-8-rpm-16dc", "id":"b8f266aa9dba5bd537592f46cf47cedb", "ip":[]interface {}{"10.128.0.73", "fe80::12f8:890:120a:c28"}, "mac":[]interface {}{"42-01-0A-80-00-49"}, "name":"ogc-linux-amd64-rhel-8-rpm-16dc", "os":map[string]interface {}{"codename":"Ootpa", "family":"redhat", "kernel":"4.18.0-553.16.1.el8_10.x86_64", "name":"Red Hat Enterprise Linux", "platform":"rhel", "type":"linux", "version":"8.10 (Ootpa)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2050", "inode":"235054683", "path":"/var/lib/elastic-agent/data/elastic-agent-8.16.0-SNAPSHOT-b832c1/logs/elastic-agent-20240816.ndjson"}, "offset":64115, "source":"elastic-agent"}, "log.level":"error", "log.origin":map[string]interface {}{"file.line":200, "file.name":"fleet/fleet_gateway.go", "function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(*FleetGateway).doExecute"}, "message":"checkin retry loop was stopped"}
    logs_ingestion_test.go:189: 
        	Error Trace:	/home/rhel/agent/testing/integration/logs_ingestion_test.go:189
        	            				/home/rhel/agent/testing/integration/linux_rpm_test.go:93
        	Error:      	Should be empty, but was [{.ds-logs-elastic_agent-nd1rpyfwjgb7tjcrv4h9ujuvtwsqt2tinojxhcr1om-2024.08.16-000001 4.017711 map[@timestamp:2024-08-16T01:28:16.509Z agent:map[ephemeral_id:434f3652-3c21-4662-a1d5-b424f491ae70 id:81fe984b-1f65-4d0c-b4e6-4f53aa583fdb name:ogc-linux-amd64-rhel-8-rpm-16dc type:filebeat version:8.16.0] cloud:map[account:map[id:elastic-platform-ingest] availability_zone:us-central1-f instance:map[id:6264460097818400828 name:ogc-linux-amd64-rhel-8-rpm-16dc] machine:map[type:e2-standard-2] project:map[id:elastic-platform-ingest] provider:gcp region:us-central1 service:map[name:GCE]] data_stream:map[dataset:elastic_agent namespace:nd1rpyfwjgb7tjcrv4h9ujuvtwsqt2tinojxhcr1om type:logs] ecs:map[version:8.0.0] elastic_agent:map[id:81fe984b-1f65-4d0c-b4e6-4f53aa583fdb snapshot:true version:8.16.0] event:map[agent_id_status:verified dataset:elastic_agent ingested:2024-08-16T01:28:57Z] host:map[architecture:x86_64 containerized:false hostname:ogc-linux-amd64-rhel-8-rpm-16dc id:b8f266aa9dba5bd537592f46cf47cedb ip:[10.128.0.73 fe80::12f8:890:120a:c28] mac:[42-01-0A-80-00-49] name:ogc-linux-amd64-rhel-8-rpm-16dc os:map[codename:Ootpa family:redhat kernel:4.18.0-553.16.1.el8_10.x86_64 name:Red Hat Enterprise Linux platform:rhel type:linux version:8.10 (Ootpa)]] input:map[type:filestream] log:map[file:map[device_id:2050 inode:235054683 path:/var/lib/elastic-agent/data/elastic-agent-8.16.0-SNAPSHOT-b832c1/logs/elastic-agent-20240816.ndjson] offset:64115 source:elastic-agent] log.level:error log.origin:map[file.line:200 file.name:fleet/fleet_gateway.go function:github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(*FleetGateway).doExecute] message:checkin retry loop was stopped]}]
        	Test:       	TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped
        	Messages:   	list of error messages is expected to be empty, found:
        	            	checkin retry loop was stopped
--- FAIL: TestRpmLogIngestFleetManaged/Monitoring_logs_are_shipped (30.50s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[rdner]

Also fails on 8.15 https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1918#01915121-b869-4cc3-9ab1-fa053f0eb39f

[rdner]

The failing test was introduced in #5260 and it seems like it was unstable from the start.

[rdner]

Skipping the test for now #5312

[blakerouse]

This doesn't look to be an RPM specific error... strange