[Fleet]: Unable to Uninstall Agent After Upgrade from 8.13.4 to 8.15.0 BC2

[harshitgupta-qasource]

Kibana Build details:

VERSION: 8.15.0 BC2
BUILD: 76111
COMMIT: 8d6510f30e7a52246ccebbf7d913a59b498c3bc2

Host OS and Browser version: All, All

Preconditions:

1. 8.15.0 BC2 Kibana Cloud environment should be available.
2. 8.13.4 agent should be deployed
3. Elastic Defend should be added to policy.

Steps to reproduce:

1. Navigate to Agents Tab

2. Upgrade the agent from 8.13.4 to 8.15.0.

3. Wait for the agent to upgrade and then uninstall Agent After Upgrade from 8.13.4 to 8.15.0 BC2.

4. Observe agent upgrade gets failed.

Expected:

• User should be able to Uninstall Agent After Upgrade from 8.13.4 to 8.15.0.

Screenshot:

• Windows
  

• Ubuntu
  

Agents Logs:

• Windows
  elastic-agent-diagnostics-2024-07-16T11-26-29Z-00(1).zip

• Linux
  elastic-agent-diagnostics-2024-07-16T11-26-29Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary Review for this ticket is Done.

[jlind23]

Hey @harshitgupta-qasource what was the system integration version you were using?

[jlind23]

This looks related to a change that @pierrehilbert recently introduced in elastic/integrations#10471 so if you are able to downgrade the system integration to the previous version 1.59.0 this would allow us to rule this out.

[kpollich]

@jsoriano - Does anything about the implementation of this condition for the processors in the above look off to you?

https://github.com/elastic/integrations/blob/b9a416c667a13e732c1a1f15eb722476c38b4ea5/packages/system/data_stream/auth/agent/stream/log.yml.hbs#L27-L39

Looking at other examples, I think this usage should be okay, but the errors above seem to indicate some kind of parsing error.

[pierrehilbert]

We already got this issue in the past, I'm investigating: elastic/go-ucfg#196

[pierrehilbert]

I think the error is coming from here:

elastic-agent/internal/pkg/config/config.go

Lines 179 to 186 in 27dacc5

	} else if subCfg.IsArray() {
	var subArr []interface{}
	err = subCfg.Unpack(&subArr, ucfg.ResolveNOOP)
	if err != nil {
	return nil, fmt.Errorf("error unpacking subarray in config for skip key %s: %w ", skip, err)
	}
	subUnpacked = subArr
	} else {

We are not using ucfgOpts and therefore we are not using IgnoreCommas

cc @fearful-symmetry as you spent time on the initial fix.

[fearful-symmetry]

yup, looks like we're missing the IgnoreCommas option from a few of the Unpack statements. Changing the statements to err = subCfg.Unpack(&subArr, ucfg.ResolveNOOP, ucfg.IgnoreCommas) should fix it.

[pierrehilbert]

@harshitgupta-qasource could you please test with the new System Integration package and confirm this is fixed?

[harshitgupta-qasource]

Hi @pierrehilbert

Thanks for the looking into this issue.

We have re-validated this issue on the latest 8.15.0 BC4 Kibana cloud environment and found it fixed now.

Observations:

• User is able to Uninstall Agent After Upgrade from 8.13.4 to 8.15.0.
• Available system version integration is 1.60.0

Build details:
VERSION: 8.15.0
BUILD: 76261
COMMIT: 9d62937675e62265342e86d8f0db601dc75498b8

Screen-shot

Hence, we are closing this issue and marking as QA: Validated.

Thanks