Elastic agent version 8.11.2 upgrade with agent binary download through proxy failed

[DysonHo]

Kibana Build details:

VERSION: 8.11.3

note:

• IP: 192.168.1.115:9330 is my lab fleet server and service port
• the host OS which elastic agent install: Windows 10 pro

Describe the bug:
following the official doc：https://www.elastic.co/guide/en/fleet/current/fleet-agent-proxy-managed.html
I simulate a environment that the Elastic agent can only communicate to fleet、elasticsearch and upgrade through proxy,
control and data traffic is fine, I can manage elastic agent with fleet, and receive data, also.
but when I try to upgrade elastic agent(v8.11.2), it failed.

I configure the elastic agent log level to "debug" and re-upgrade to find out what's going on:

• upgrade test 1, I got 2 error：

11:16:04.214 elastic_agent [elastic_agent][warn] Skipped remote PGP located at "https://192.168.1.115:9330/api/agents/upgrades/8.11.3/pgp-public-key" because it's unavailable: 2 errors occurred:
* Get "https://192.168.1.115:9330/api/agents/upgrades/8.11.3/pgp-public-key": tls: failed to verify certificate: x509: certificate signed by unknown authority
* Remote PGP download failed
11:16:04.687 elastic_agent [elastic_agent][error] upgrade to version 8.11.3 failed: failed verification of agent binary: 2 errors occurred:
* fetching asc file from 'C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc': open C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc: The system cannot find the file specified.
* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc: failed loading public key: Get "https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc": dial tcp: lookup artifacts.elastic.co: no such host

accroding to official doc：https://www.elastic.co/guide/en/fleet/8.11/release-notes-8.11.0.html#bug-fixes-8.11.0
I know there is a mechanism elastic agent will download pgp key from fleet if the GPG URL is unreachable, and also it seems a name lookup problem, therefore I install the root CA on the host which I install elastic agent, and also resolve name lookup problem, and go on the second upgrade test.

• upgrade test 2, I got 1 error：

12:10:50.431 elastic_agent [elastic_agent][error] upgrade to version 8.11.3 failed: failed verification of agent binary: 2 errors occurred:
  * fetching asc file from 'C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc': open C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc: The system cannot find the file specified.
  * fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc: failed loading public key: Get "https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc": dial tcp: lookup artifacts.elastic.co: getaddrinfow: The requested name is valid, but no data of the requested type was found.

this is where I got stuck now, seems the lookup problem is solved, but got no data, and still download failed,
to make sure there is no misconfigured with network,
I run powershell cmdlet: invoke-webrequest on the same host which I install elastic agent to download pgp key, it success.

---

I understand this functionality is in beta, but still try to figure out is this my problem or a bug,
if anyone want more detail, please let me know,
sincerely,

[cmacknz]

12:10:50.431 elastic_agent [elastic_agent][error] upgrade to version 8.11.3 failed: failed verification of agent binary: 2 errors occurred:
  * fetching asc file from 'C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc': open C:\Program Files\Elastic\Agent\data\elastic-agent-1c21b0\downloads\elastic-agent-8.11.3-windows-x86_64.zip.asc: The system cannot find the file specified.
  * fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc: failed loading public key: Get "https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc": dial tcp: lookup artifacts.elastic.co: getaddrinfow: The requested name is valid, but no data of the requested type was found.

getaddrinfow: The requested name is valid, but no data of the requested type was found is a DNS resolution error. Since https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-windows-x86_64.zip.asc exists and can be downloaded this is likely be introduced by your proxy. I can't tell why though.

This doesn't immediately look like a bug in the agent itself. I'd suggest you continue troubleshooting this in https://discuss.elastic.co/c/elastic-stack/elastic-agent/91?page=1 which has a much wider audience.

For now I'll close this, please re-open if you confirm this isn't working properly.

[FlorianHeigl]

for the record, i had the opportunity to run into what appears the same issue.

{"log.level":"warn","@timestamp":"2024-06-30T18:32:20.635+0200","log.origin":
{"file.name":"composed/verifier.go","file.line":53},"message":"Verifier failed!","log":{"source":"elastic-agent"},"verifier":"http.verifier","error":{"message":"fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.2-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.2-linux-

x86_64.tar.gz.asc\": context deadline exceeded"},"ecs.version":"1.6.0"}

The commonality seems the 8.11.2. version. I'm sure i had previously upgraded those VM's agents from

a lower version, but then they just didn't want to make the jump anymore.
I wonder if there's a misleading element in this, the downloads work just the gpg involved stuff seems to not seeing the internetz... I dimly remember that GPG needs to be properly told how to get to the a keyserver through a proxy.

I don't know if it needed to access a GPG keyserver or import a new key. It says something like using default key. but it seems very likely.

The error handling must fall through to the GPG failure in too many cases (download failure of the actual agent as well as the GPG signature check. I would also argue that it is overkill to wipe the tar.gz if you can't validate due to a validation process failure vs. a FAILED validation. It's 500 meg after all. could burn through a network on a large scale.

FTR:
The same system can download via proxy using wget and the file + hash are fetched without problem, which matches with the messages in the logs. HTH

Here I ran

wget https://artifacts.elastic.co/GPG-KEY-elasticsearch
apt-key add ./GPG-KEY-elasticsearch

Getting me two installed keys and a further failed verification

t":{"cgroup":{"memory":{"mem":{"usage":{"bytes":1782841344}}}},"cpu":{"system":{"ticks":741410},"total":{"ticks":2061060,"time":{"ms":40},"value":2061060},"user":{"ticks":1319650,"time":{"ms":40}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":14},"info":{"ephemeral_id":"c0548b24-1e6d-45a4-a95e-0fa98978f4c0","uptime":{"ms":1748552215},"version":"8.11.2"},"memstats":{"gc_next":56998920,"memory_alloc":28386896,"memory_total":103513526776,"rss":124989440},"runtime":{"goroutines":67}},"filebeat":{"events":{"active":0,"added":24,"done":56},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":24,"active":197,"batches":3,"total":24},"read":{"bytes":1018},"write":{"bytes":5338}},"pipeline":{"clients":3,"events":{"active":0,"published":24,"total":24},"queue":{"acked":24}}},"processor":{"syslog":{"1":{"success":1}}},"registrar":{"states":{"current":4,"update":24},"writes":{"success":4,"total":4}},"system":{"load":{"1":0.15,"15":0.1,"5":0.13,"norm":{"1":0.0375,"15":0.025,"5":0.0325}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:00.520+0200","log.origin":{"file.name":"http/downloader.go","file.line":319},"message":"download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.2-linux-x86_64.tar.gz completed in 2 minutes @ 4.228MBps","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:00.538+0200","log.origin":{"file.name":"http/downloader.go","file.line":319},"message":"download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.2-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:03.024+0200","log.origin":{"file.name":"fs/verifier.go","file.line":119},"message":"Default PGP being appended","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:03.141+0200","log.origin":{"file.name":"fs/verifier.go","file.line":144},"message":"Using 2 PGP keys","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-06-30T18:55:03.141+0200","log.origin":{"file.name":"composed/verifier.go","file.line":53},"message":"Verifier failed!","log":{"source":"elastic-agent"},"verifier":"fs.verifier","error":{"message":"fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-1c21b0/downloads/elastic-agent-8.12.2-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-1c21b0/downloads/elastic-agent-8.12.2-linux-x86_64.tar.gz.asc: no such file or directory"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:04.669+0200","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":1877934080}}}},"cpu":{"system":{"ticks":3220,"time":{"ms":10}},"total":{"ticks":22570,"time":{"ms":20},"value":22570},"user":{"ticks":19350,"time":{"ms":10}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":14},"info":{"ephemeral_id":"32050ee9-6f11-4fc7-af95-339b0eb1a054","uptime":{"ms":5010111},"version":"8.11.2"},"memstats":{"gc_next":89443376,"memory_alloc":47502696,"memory_total":1883514984,"rss":182353920},"runtime":{"goroutines":53}},"filebeat":{"events":{"active":0,"added":6,"done":6},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"acked":5,"active":0,"batches":2,"total":5},"read":{"bytes":393},"write":{"bytes":4508}},"pipeline":{"clients":2,"events":{"active":0,"filtered":1,"published":5,"total":6},"queue":{"acked":5}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.11,"15":0.09,"5":0.12,"norm":{"1":0.0275,"15":0.0225,"5":0.03}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:05.633+0200","log.origin":{"file.name":"http/verifier.go","file.line":120},"message":"Default PGP being appended","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-30T18:55:05.684+0200","log.origin":{"file.name":"http/verifier.go","file.line":145},"message":"Using 2 PGP keys","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

it seems quite illogical.

edit:
found https://www.elastic.co/guide/en/fleet/current/fleet-troubleshooting.html#pgp-key-download-fail

the workaround seems horrendous though.
plural, actually.
my brain is still hurting even after i decided to just uninstall and reinstall and hope for the best.

In case someone from Elastic sees this:
If you offer an option to set a proxy for downloading the artifacts you need to make this work proper.
Do not use proxy for calling to fleet. Do use proxy to get GPG key. Do not get GPG key if you have it. Unless there's a reason. Then get it. Do handle the individual issues and fail accordingly. You can't go delete your signature file that you just downloaded, and then try verify it with a key you had but try to download using the wrong method.