[Fleet]: Unable to Upgrade Debian agent on extracting latest version artifact.

[harshitgupta-qasource]

Kibana Build details:

VERSION: 8.16.0 SNAPSHOT
BUILD: 77679
COMMIT: 6b091fe3b410eaae9d4805c0a3c0ea6168bf66b0

Host OS : Debian 12, Ubuntu 24

Preconditions:

1. 8.16.0 SNAPSHOT Kibana Cloud environment should be available.
2. 8.14.3 / 8.15.0 deb agent should be installed.

Steps to reproduce:

1. Download the Deb agent artifact on the Debian VM.
2. Install it using: sudo dpkg -i elastic-agent-8.14.3-amd64.deb.
3. Enroll into Fleet using: sudo elastic-agent enroll --url={url} --enrollment-token={token}.
4. Start the service with:
5. sudo systemctl enable elastic-agent
6. sudo systemctl start elastic-agent
7. Observe 8.14.3 debian agent is installed.
8. Upgrading to the Latest Debian Package:
9. Download the latest 8.16.0-SNAPSHOT Deb agent artifact on the Debian VM.
10. Install it using: sudo dpkg -i elastic-agent-8.16.0-SNAPSHOT-amd64.deb.
11. Observe Agent is not upgraded and installed agent gets offline.

Expected:
Debian agent should upgrade to latest version on extracting latest version artifact.

Screenshot:

Feature Ticket:
#5260

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[blakerouse]

This statement in the description is not accurate Host OS and Browser version: All, All. This is specific to Linux.

Please provide the specific version of Linux being used to test, that way I can try to reproduce.

[pierrehilbert]

As this is backported to 8.15.1, could you also try to upgrade to 8.15.1 and confirm you are facing the same issue?

[harshitgupta-qasource]

Hi Team,

Thank you for looking into this issue.

This statement in the description is not accurate: Host OS and Browser version: All, All.
This is specific to Linux. Please provide the specific version of Linux being used to test, that way I can try to reproduce.

Sorry for the confusion. We have updated the description and validated this issue on Debian 12 and Ubuntu 24.

As this is backported to 8.15.1, could you also try to upgrade to 8.15.1 and confirm you are facing the same issue?

We have further validated this issue while upgrading debian agent from 8.15.0 to 8.15.1, and found this issue reproducible there too.

We have observed that the agent does not upgrade, causing the installed agent(8.15.0) to go offline.

Screenshot

Please let us know if anything else is required from our end.

Thank you

[pierrehilbert]

As we are releasing 8.15.1 next week, @blakerouse could you please have a look here?

[blakerouse]

I am not seeing any issues. I was able to start at 8.15.0 and upgraded to 8.15.1-SNAPSHOT just fine.

blake@blakerouse-debian-12:~$ sudo dpkg -i elastic-agent-8.15.0-amd64.deb 
Selecting previously unselected package elastic-agent.
(Reading database ... 69231 files and directories currently installed.)
Preparing to unpack elastic-agent-8.15.0-amd64.deb ...
Unpacking elastic-agent (8.15.0) ...
Setting up elastic-agent (8.15.0) ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent
blake@blakerouse-debian-12:~$ sudo elastic-agent enroll --url=https://9462521391ba40b2885a1df95bb11434.fleet.us-central1.gcp.foundit.no:443 --enrollment-token=VXZGaW5wRUJ3MUt0Q2dmNmNrSHg6Z29iMm44R3ZRYmlMZ09zZ0V4RXJHQQ== 
sudo systemctl enable elastic-agent 
sudo systemctl start elastic-agent
{"log.level":"info","@timestamp":"2024-08-29T13:45:57.192Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://9462521391ba40b2885a1df95bb11434.fleet.us-central1.gcp.foundit.no:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:45:58.235Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:45:58.235Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:46:00.236Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:46:00.236Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:46:04.237Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:46:04.238Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:46:12.240Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:46:12.241Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:46:28.241Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:46:28.242Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:47:00.244Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":295},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"","ecs.version":"1.6.0"}
Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.15/fleet-troubleshooting.html
Synchronizing state of elastic-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable elastic-agent
Created symlink /etc/systemd/system/multi-user.target.wants/elastic-agent.service → /lib/systemd/system/elastic-agent.service.
blake@blakerouse-debian-12:~$ wget https://snapshots.elastic.co/8.15.1-6ed69fc1/downloads/beats/elastic-agent/elastic-agent-8.15.1-SNAPSHOT-amd64.deb
--2024-08-29 13:49:10--  https://snapshots.elastic.co/8.15.1-6ed69fc1/downloads/beats/elastic-agent/elastic-agent-8.15.1-SNAPSHOT-amd64.deb
Resolving snapshots.elastic.co (snapshots.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to snapshots.elastic.co (snapshots.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 344177774 (328M) [application/vnd.debian.binary-package]
Saving to: ‘elastic-agent-8.15.1-SNAPSHOT-amd64.deb’

elastic-agent-8.15.1-SNAPSHOT-amd64.deb                        100%[===================================================================================================================================================>] 328.23M  10.0MB/s    in 38s     

2024-08-29 13:49:49 (8.62 MB/s) - ‘elastic-agent-8.15.1-SNAPSHOT-amd64.deb’ saved [344177774/344177774]

blake@blakerouse-debian-12:~$ sudo dpkg -i elastic-agent-8.15.1-SNAPSHOT-amd64.deb 
(Reading database ... 69673 files and directories currently installed.)
Preparing to unpack elastic-agent-8.15.1-SNAPSHOT-amd64.deb ...
previous installation directory /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f
migrate state from /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7
didn't find /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/state.yml
found /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/state.enc, copy to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7.
Unpacking elastic-agent (8.15.1) over (8.15.0) ...
stopping elastic-agent
Warning: The unit file, source configuration file or drop-ins of elastic-agent.service changed on disk. Run 'systemctl daemon-reload' to reload units.
found symlink /usr/share/elastic-agent/bin/elastic-agent, unlink
systemd daemon-reload
dpkg: warning: unable to delete old directory '/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f': Directory not empty
Setting up elastic-agent (8.15.1) ...
Installing new version of config file /etc/elastic-agent/.elastic-agent.active.commit ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7/elastic-agent
systemd enable/restart elastic-agent
blake@blakerouse-debian-12:~$ 

I noticed that you are upgrading to 8.16.0-SNAPSHOT, are you sure the stack you are enrolled into is also at 8.16-SNAPSHOT? If not an upgrade of an Elastic Agent to 8.16 connected to a 8.15 stack will result in the Elastic Agent not being able to communicate with Fleet.

[blakerouse]

I also see it working on Ubuntu 24.04 as well:

blake@blakerouse-ubuntu-2404:~$ sudo dpkg -i elastic-agent-8.15.0-amd64.deb 
Selecting previously unselected package elastic-agent.
(Reading database ... 67860 files and directories currently installed.)
Preparing to unpack elastic-agent-8.15.0-amd64.deb ...
Unpacking elastic-agent (8.15.0) ...
Setting up elastic-agent (8.15.0) ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent
blake@blakerouse-ubuntu-2404:~$ sudo elastic-agent enroll --url=https://9462521391ba40b2885a1df95bb11434.fleet.us-central1.gcp.foundit.no:443 --enrollment-token=VXZGaW5wRUJ3MUt0Q2dmNmNrSHg6Z29iMm44R3ZRYmlMZ09zZ0V4RXJHQQ== 
sudo systemctl enable elastic-agent 
sudo systemctl start elastic-agent
{"log.level":"info","@timestamp":"2024-08-29T13:52:48.362Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://9462521391ba40b2885a1df95bb11434.fleet.us-central1.gcp.foundit.no:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:52:49.400Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:52:49.401Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:52:51.401Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:52:51.402Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:52:55.403Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:52:55.404Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:53:03.406Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:53:03.407Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-29T13:53:19.407Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:53:19.408Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-29T13:53:51.411Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":295},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"","ecs.version":"1.6.0"}
Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.15/fleet-troubleshooting.html
Synchronizing state of elastic-agent.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable elastic-agent
Created symlink /etc/systemd/system/multi-user.target.wants/elastic-agent.service → /usr/lib/systemd/system/elastic-agent.service.
blake@blakerouse-ubuntu-2404:~$ sudo dpkg -i elastic-agent-8.15.1-SNAPSHOT-amd64.deb 
(Reading database ... 68302 files and directories currently installed.)
Preparing to unpack elastic-agent-8.15.1-SNAPSHOT-amd64.deb ...
previous installation directory /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f
migrate state from /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7
didn't find /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/state.yml
found /var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/state.enc, copy to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7.
Unpacking elastic-agent (8.15.1) over (8.15.0) ...
stopping elastic-agent
Warning: The unit file, source configuration file or drop-ins of elastic-agent.service changed on disk. Run 'systemctl daemon-reload' to reload units.
found symlink /usr/share/elastic-agent/bin/elastic-agent, unlink
systemd daemon-reload
dpkg: warning: unable to delete old directory '/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f': Directory not empty
Setting up elastic-agent (8.15.1) ...
Installing new version of config file /etc/elastic-agent/.elastic-agent.active.commit ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-8.15.1-SNAPSHOT-15b8b7/elastic-agent
systemd enable/restart elastic-agent
blake@blakerouse-ubuntu-2404:~$ 

[amolnater-qasource]

Hi @blakerouse

Thank you for sharing the details.
We have revalidated on latest 8.16.0-SNAPSHOT and 8.15.1 BC1 and found it working fine.

Observations:

• We are successfully able to upgrade 8.15.0>8.15.1 on Ubuntu 24.04 and 8.15.0>8.16.0-SNAPSHOT on Debian 12.

Screenshots:

Build details:
VERSION: 8.16.0-SNAPSHOT
BUILD: 77905
COMMIT: 50e46291f8e18dbd90c77b8938a41206b6bf122f
Artifact Link: https://snapshots.elastic.co/8.16.0-a3caaa65/downloads/beats/elastic-agent/elastic-agent-8.16.0-SNAPSHOT-amd64.deb

VERSION: 8.15.1 BC1
BUILD: 76516
COMMIT: 1796ec02f5523bff4e449c368a3fea574d44455a
Artifact Link: https://staging.elastic.co/8.15.1-af049989/downloads/beats/elastic-agent/elastic-agent-8.15.1-amd64.deb

Hence, we are closing this ticket and marking as QA:Validated.

Please let us know if anything else is required from our end.

Thanks!