[Fleet]: Agent logging level is not updated automatically for Upgraded agent from info to debug on making changes to Global agent logging level.

[amolnater-qasource]

Kibana Build details:

VERSION: 9.0.0-SNAPSHOT
BUILD: 83694
COMMIT: 522b0d2de9bbcbcde9331799811760c80b8f54b8

Artifact Link: https://staging.elastic.co/8.18.0-01901d62/downloads/beats/elastic-agent/elastic-agent-8.18.0-linux-x86_64.tar.gz

Preconditions:

1. 9.0.0-SNAPSHOT Kibana cloud environment should be available.

Steps to reproduce:

1. Install 8.18.0 BC4 agent.
2. Now install another 9.0.0 agent with the same policy.
3. Trigger 8.18.0 agent upgrade to 9.0.0-beta1 and observe agent gets upgraded successfully.
4. Now go the agent policy settings and change global agent logging level to debug.
5. Observe logging level is not updated to debug for upgraded agent.

Expected Result:
Agent logging level should be updated automatically for Upgraded agent from info to debug on making changes to Global agent logging level.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2025-02-20.15-14-09.mp4

Logs:

elastic-agent-diagnostics-2025-02-20T09-45-01Z-00.zip

Agent policy:

elastic-agent (1).zip

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[cmacknz]

Looking at the agent policy I see the log level is set to debug in pre-config.yaml:

agent:
    download:
        sourceURI: https://artifacts.elastic.co/downloads/
    features: null
    logging:
        level: debug

In the agent state, the log level is still info in state.yaml:

fleet_message: Connected
fleet_state: 2
log_level: info
message: Running
state: 2

The local agent-info.yaml still shows info:

agent_id: 61990689-398b-4073-a2ad-26fd50f00b8e
headers: {}
log_level: info
snapshot: false
unprivileged: false
version: 9.0.0-beta1

I see the following in the agent logs:

{"log.level":"info","@timestamp":"2025-02-20T09:17:43.184Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level info from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2025-02-20T09:33:26.474Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level debug from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

I think the problem is on the agent side of this.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)