Recreate kind cluster after each K8s test

[swiatekm]

What does this PR do?

Creates a new kind cluster for each K8s test.

Why is it important?

This is a temporary measure to improve K8s test reliability before we can run them in parallel. We run these tests sequentially right now, and each kind cluster is only used once, but is only cleaned up after all tests finish running. Creating and deleting kind clusters is fast relative to test run time, so this shouldn't impact total run time significantly.

Kubernetes tests took 2h 31m on this branch: https://buildkite.com/elastic/elastic-agent/builds/17875#01955b43-3ce0-4b43-814b-e433929b6ff2.
On main they recently took 2h 30m: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/7791#01954e7c-ac7c-43fc-9987-1431e2c3b2ad

So there's little difference in runtime, and potential to fix some flakiness in the short-term.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas

How to test this PR locally

SNAPSHOT=true EXTERNAL=true PACKAGES=docker PLATFORMS="linux/amd64" mage -v  package
Running target: Package
INSTANCE_PROVISIONER=kind STACK_PROVISIONER=stateful SNAPSHOT=true mage integration:single TestOtelKubeStackHelm

Related issues

• Relates [Flaky Test]: Kubernetes platform-related integration test failures #7060

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ec65a9f

History

• 💛 Build #17875 was flaky 4cdf1ec
• 💔 Build #17807 failed 11bf290

cc @swiatekm