[8.17](backport #7029) Add the full set of ECS metadata sent to Fleet to Diagnostics

[mergify]

What does this PR do?

Adds the full set of ECS metadata that is sent to Fleet when the agent checks in to diagnostics. If you don't know what that is, it's this:

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 327 to 330 in d2047ac

	ecsMeta, err := info.Metadata(ctx, f.log)
	if err != nil {
	f.log.Error(errors.New("failed to load metadata", err))
	}

elastic-agent/internal/pkg/agent/application/info/agent_metadata.go

Lines 25 to 31 in d2047ac

	// ECSMeta is a collection of agent related metadata in ECS compliant object form.
	type ECSMeta struct {
	Elastic *ElasticECSMeta `json:"elastic"`
	Host *HostECSMeta `json:"host"`
	OS *SystemECSMeta `json:"os"`
	}

Why is it important?

When all providers were enabled by default, you were able to get the information about the host from the host provider (hostname, OS version, etc) in the variables.yaml file. Now that providers are only enabled when referenced explicitly in the policy, the content of the variables.yaml file is by default:

variables:
    - {}

This makes sure that detailed information about the host is available in diagnostics all the time separately from providers. This is useful if you are say, debugging why you can't find agent by the hostname you think it should have in Fleet. The format of the agent-info.yaml file now looks like:

headers: {}
log_level: info
log_level_raw: info
metadata:
    elastic:
        agent:
            buildoriginal: '9.1.0-SNAPSHOT (build: d2047ac48df2f4536ca69a86ad4922b3e264501a at 2025-02-25 21:52:49 +0000 UTC)'
            complete: false
            id: 9ba45d59-e33f-42c7-a47d-50a9a37705ef
            loglevel: info
            snapshot: true
            unprivileged: false
            upgradeable: true
            version: 9.1.0
    host:
        arch: arm64
        hostname: Craigs-Macbook-Pro.local
        id: 48DA13D6-B83B-5C71-A4F3-494E674F9F37
        ip:
            - 127.0.0.1/8
            - ...
        mac:
            - ...
        name: craigs-macbook-pro.local
    os:
        family: darwin
        fullname: macOS(14.7.3)
        kernel: 23.6.0
        name: macOS
        platform: darwin
        version: 14.7.3

Disruptive User Impact

I removed some of the duplicated keys from the top level of agent-info.yaml. I am assuming nobody was relying on this in a critical way.

How to test this PR locally

elastic-development-agent diagnostics and then read the agent-info.yaml file.

---

This is an automatic backport of pull request #7029 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of d668787 has failed:

On branch mergify/bp/8.17/pr-7029
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit d6687876c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1740521494-diagnostics-include-all-metadata-that-is-sent-to-Fleet-by-default.yaml
	modified:   internal/pkg/agent/application/coordinator/coordinator.go
	modified:   internal/pkg/agent/application/coordinator/diagnostics_test.go
	modified:   internal/pkg/agent/application/info/agent_info.go
	modified:   pkg/component/runtime/runtime_comm_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   testing/mocks/internal_/pkg/agent/application/info/agent_mock.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 68a8d23

Failed CI Steps

• Win2022:sudo:fleet-endpoint-security
• Win2025:sudo:fleet-endpoint-security
• x86_64:sudo: fleet-endpoint-security

History

• 💔 Build #18048 failed fa3f8e7

cc @cmacknz

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
85.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube