Internal monitoring to collector metrics-elastic_agent.elastic_agent-* indices pointlessly collects metrics from all components

[cmacknz]

We want to fetch only the metrics from elastic agent’s /stats endpoint and write them to metrics-elastic_agent.elastic_agent-* but the input created to do this is in a for loop that iterates over every running agent component with no guarding if condition:

elastic-agent/internal/pkg/agent/application/monitoring/v1_monitor.go

Lines 775 to 784 in 5d91620

	streams = append(streams, map[string]interface{}{
	idKey: fmt.Sprintf("%s-%s-1", monitoringMetricsUnitID, name),
	"data_stream": map[string]interface{}{
	"type": "metrics",
	"dataset": fmt.Sprintf("elastic_agent.%s", fixedAgentName),
	"namespace": monitoringNamespace,
	},
	"metricsets": []interface{}{"json"},
	"hosts": endpoints,
	"path": "/stats",

Compare this the one just above that is specific to Beats, it has an if on the binary name

elastic-agent/internal/pkg/agent/application/monitoring/v1_monitor.go

Line 712 in 5d91620

if isSupportedBeatsBinary(binaryName) {

If you look at the components/http-metrics-monitoring/beat-rendered-config.yaml file in diagnostics you can see all the duplicated http/metrics streams pointing at each Beat for the metrics-elastic_agent.elastic_agent-default data stream that should contain only metrics from the Elastic Agent process itself.

Definition of Done:

• A test exists proving we only create the minimal set of inputs to collect the monitoring data we need.
• There is only a single http/metrics input for collecting data from Elastic Agent's /stats input.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)