OSQuery results are not viewable in Kibana when upgrading to the 8.6.0 Agent running the OSQuery Manager integration

[kevinlog]

When upgrading to the 8.6.0 Agent, OSQuery results will not be visible in Kibana due to changes in a couple values in documents that are shipped to ES by the OSQuery beat. These differing docs between OSQuery beat in the 8.5.x Agent and the 8.6.0 Agent cause the newer documents to be rejected by the logs-osquery_manager.result-* datastream.

Steps to Reproduce:

1. Install the 8.5.x stack and enroll a 8.5.x Agent with the OSQuery Manager integration installed on the Agent policy
2. Run a live query and see that the results are visible in Kibana
3. Install the 8.6.0 stack and upgrade the Agent to 8.6.0
4. Run a live query and see that the result are not visible in Kibana due to rejected documents as described above.

• Version: 8.6.0
• Operating System: Linux, Mac, Windows

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[botelastic]

This issue doesn't have a Team:<team> label.

[kevinlog]

This is an internally confirmed bug in Agent and OSQuery beat for 8.6.0. I opened this so that it can be tracked externally

[aleksmaus]

The proposal to support 8.6.0 and fix the issue in 8.6.1 is the following:

1. Update osquerybeat configuration transformation to enforce the correct values for data_stream fields.
2. Update osquery_manager integration to include the pipeline that sets the correct values for data_stream in order to fix the issues with 8.6.0 deployments already that are already installed.
3. Document the additional logstash filter that needs to be used for 8.6.0 agents that will overwrite the data_stream fields in order to route the data into the correct datastream.
4. Document in the release notes for 8.6.0

[aleksmaus]

Please disregard the linked #33587 PR above, this was linked by mistake. Not sure how to unlink it.

The correct relevant PR link is the next one #34246

[aleksmaus]

For the logstash the additional filter needs to be configured in the logstash pipeline only if the used with 8.6.0 version of the agent.
This is only needed for 8.6.0 elastic agent, since all the other versions are not affected.

This is want I tested so far works:

filter {
  if [data_stream][type] == "osquery" {
    mutate {
        replace => { "[data_stream][type]" => "logs" }
        replace => { "[data_stream][dataset]" => "osquery_manager.result" }
    }
  }
}

[kevinlog]

The latest osquery_manager package 1.6.0 is now available. Users who upgraded to 8.6.0 from 8.5.x can workaround this problem today by doing the following.

In their deployment, click on "Add Integrations"

Click on the "Installed Integrations" tab and "Updates available"

Click on "OSQuery Manager" and then "Upgrade to latest version"

After the new assets install, users should be able to run queries with both 8.5.x and 8.6.0 Agents.