Replace the beat/metrics input with http/metrics for collecting Beat process stats

[cmacknz]

This is a follow up from:

• Stop collecting the beat state metricset as part of agent monitoring #4153

Today we use the Beat metrics input to collect metrics from Beat processes:

elastic-agent/internal/pkg/agent/application/monitoring/v1_monitor.go

Lines 617 to 625 in b39b9af

	if isSupportedBeatsBinary(binaryName) {
	beatsStreams = append(beatsStreams, map[string]interface{}{
	idKey: "metrics-monitoring-" + name,
	"data_stream": map[string]interface{}{
	"type": "metrics",
	"dataset": fmt.Sprintf("elastic_agent.%s", name),
	"namespace": monitoringNamespace,
	},
	"metricsets": []interface{}{"stats", "state"},

Once we remove use of the Beat module's state metricset as described in #4153 we are only collecting stats, which could instead be done using the http/metrics input scraping the Beat processes /stats endpoint. This will allow us to remove use of the beat/metrics input remove one of the three monitoring processes we start.

We will need to make sure that the fields captured using http/metrics match what was collected by the beat stats metricset for the fields we actually map and use to avoid breaking changes.

This will remove an entire beat process from the agent process tree, including the memory overhead of the queue and its outgoing network connection to the output.

This will also ensure we collect metrics from Beats even if the process has not yet connected to the output, which happens lazily when the first event is published. This can result in some unintuitive behaviour where the existence of the metrics-elastic-agent.*-* indices only exist if the beat is doing useful work.

See elastic/cloud-on-k8s#7389 and elastic/cloud-on-k8s#7389 (comment) for context on this problem. The beat module does not publish events unless the Elasticsearch cluster UUID is available, which has also meant we do not collect metrics when using other output types as expected.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)