[Obs AI Assistant] Re-index KB if the index doesn't support semantic text

[sorenlouv]

Timeline

• 8.10: Initial launch of Obs AI Assistant. The KB index is created automatically
• 8.11: Introduction of the sparse_vector field for semantic processing.
• 8.12: GA release of the Observability AI Assistant.
• 8.17: KB begins leveraging the semantic_text field.
• 8.18: GA release of the semantic_text field

Problem

Any user on 8.10 will have a KB index that does not support semantic_text. This means that if a user who has used 8.10 upgrades to 8.17 or later, they will no longer be able to ingest documents into the knowledge base.

Reproduce

• Create 8.10 cluster
• Verify that the knowledge base index exists (GET .kibana-observability-ai-assistant-kb)
• Upgrade to 8.17 or above
• Add item to KB

Solution

Upon adding documents to the KB we should catch the error. We should re-index the KB index if the error thrown is:

document_parsing_exception
Caused by:
unsupported_operation_exception: The [sparse_vector] field type is not supported on indices created on versions 8.0 to 8.10

Note: We should make sure to block writes to the index as soon as we start re-indexing in order to avoid multiple users starting the migration.

[elasticmachine]

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)