[CI] Serverless beats tests migration to beats

[pazone]

What does this PR do?

Moves Serverless Beats tests buildkite step to a separate pipeline.

Note: This PR is tightly coupled with elastic/beats#43007. This PR should be merged first.

The Beats workflow will be capable of triggering Serverless beats tests when we push changes to the main/release branch. Build example - it is tested with a custom branch that triggers the tests in elastic-agent repo.

The Serverless Beats tests have to be completely migrated to the beats repo, but it uses the elastic-agent testing framework. A follow-up issue will be created

Why is it important?

The Serverless beats tests are related to the beats repo and have to be triggered by its events. Furthermore, it affects elastic-agent productivity by the long duration and the concurrency limitations.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @pazone? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 7d257e5

Failed CI Steps

• Serverless integration test

cc @pazone

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)