[SLO] [Management] deduping task for temporary summary documents

[kdelemme]

🔩 Summary

Related to #191095

When an SLO is created, we insert a temporary summary document with dummy or nil values. This is to counter the lag effect introduced by the two layer of transforms, and so customers are not wondering why their SLO is not shown just after creating one.

When we list the SLO, we do a search on the summary document, and we dedupe the temp / non-temp summaries before returning the result. At this time, we delete the temp summaries that we know have real non-temp summaries.

The problem with this approach is that if you have many SLO instances, a search might never trigger the removal of the temporary documents since they are not returned from the search at the same time as the non temp summary document.

We should create a task that periodically removes the temporary summary documents that have at least one non-temporary summary document. This task should run probably hourly.
It would be useful to build the service behind the task in a way that can be reused from an API for example.

🙈 Spoiler query - Open only if you want one solution

We can use a composite agg (we would needto paginate through the results)

POST .slo-observability.summary-v3.4*/_search
{
  "size": 0,
  "aggs": {
    "duplicate_ids": {
      "composite": {
        "size": 10000,
        "sources": [
          {
            "id": {
              "terms": {
                "field": "slo.id"
              }
            }
          }
        ]
      },
      "aggs": {
        "unique_isTempDoc": {
          "cardinality": {
            "field": "isTempDoc"
          }
        },
        "find_duplicates": {
          "bucket_selector": {
            "buckets_path": {
              "unique_count": "unique_isTempDoc"
            },
            "script": "params.unique_count == 2"
          }
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)