[osquery_manager]: Fix osquery_manager data_stream values for 8.6.0 with ingest pipeline

[aleksmaus]

What does this PR do?

This adds the ingest pipeline to the osquery_manager in order to fix the known defect in 8.6.0 release where the data_stream fields on the document are set to incorrect values.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes OSQuery results are not viewable in Kibana when upgrading to the 8.6.0 Agent running the OSQuery Manager integration beats#34250

Screenshots

This is the new ingest pipeline that shows up in kibana when the integration is installed

Confirmed that the document is indexed with correct data_stream values

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-13T16:01:07.581+0000

• Duration: 12 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16
Skipped	0
Total	16

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ruflin]

Change LGTM. I have not tested this and someone should really test this e2e.

[ruflin]

I was just checking what the default of overwrite is which is set to true by default: https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html This is good but maybe to be extra careful, put it in :-)

[aleksmaus]

Added overrride: true, and retested e2e again, uncluding upgrade steps.

Here is the ingest pipeline updated, installed from the latest changes in this PR

[aleksmaus]

I tested this e2e, the clean install and the upgrade scenario.

Upgrade testing steps:

1. Run 8.6.0 Elasticsearch-Kibana-FleetServer
2. Create the agent policy with oquery manager integration 1.5.1 (that's the integration before this change without the pipeline)
3. Enroll the 8.5.3 agent.
4. Execute osquery live query. Observe the result is comes back and is shown on UI.
5. Enroll the 8.6.0 agent. This is where the defect is.
6. Execute osquery live query for 8.6.0 agent. Observe the result never comes back. The agent/osquerybeat log has error something along these lines:
   "[constant_keyword] field [data_stream.type] only accepts values that are equal to the value defined in the mappings [logs], but got [osquery]"
   where it fails to index the document due to wrong values in the data_stream
7. Update the integration to 1.6.0 (that has the new pipeline).
8. Execute osquery live query for 8.6.0 agent again. Now the pipeline should set the correct values for the data_stream fields and the query result is shown on UI or can be found in the results datastream

GET logs-osquery_manager.result-*/_search

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 3.774
Classes	100.0% (0/0)	💚 3.774
Methods	25.0% (1/4)	👎 -73.137
Lines	100.0% (0/0)	💚 8.991
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Package osquery_manager - 1.6.0 containing this change is available at https://epr.elastic.co/search?package=osquery_manager