We have devised our own test framework to test the spec against an OpenSearch cluster. We're still adding more features to the framework as the needs arise, and this document will be updated accordingly. This test framework has also been integrated into the repo's CI/CD pipeline. Checkout the test-spec workflow for more details.
Set up an OpenSearch cluster with Docker:
(Replace <<your_password>> with your desired password. If not provided, the default password inside the docker-compose.yml file will be used.)
export OPENSEARCH_PASSWORD=<<your_password>>
cd .github/opensearch-cluster
docker compose up -dRun the tests (use --opensearch-insecure for a local cluster running in Docker that does not have a valid SSL certificate):
npm run test:spec -- --opensearch-insecureRun a specific test story:
npm run test:spec -- --opensearch-insecure --tests tests/_core/info.yamlVerbose output:
npm run test:spec -- --opensearch-insecure --verboseNote: Remember to set the OPENSEARCH_PASSWORD environment variable everytime you start a new shell to run the tests. Failing to do so will result in 401 Unauthorized errors.
The spec tests reside in the tests/ directory. Tests are organized in folders that match namespaces. For example, tests for APIs defined in spec/namespaces/indices.yaml can be found in tests/indices/index.yaml (for /{index}), and tests/indices/doc.yaml (for /{index}/_doc).
Each yaml file in the tests directory represents a test story that tests a collection of related operations.
A test story has 3 main components:
- prologues: These are the operations that are executed before the test story is run. They are used to set up the environment for the test story.
- chapters: These are the operations that are being tested.
- epilogues: These are the operations that are executed after the test story is run. They are used to clean up the environment after the test story.
Check the test_story JSON Schema for the complete structure of a test story.
Below is the simplified version of the test story that tests the index operations:
$schema: ../json_schemas/test_story.schema.yaml # The schema of the test story. Include this line so that your editor can validate the test story on the fly.
description: This story tests all endpoints relevant the lifecycle of an index, from creation to deletion.
prologues: [] # No prologues are needed for this story.
epilogues: # Clean up the environment by assuring that the `books` index is deleted afterward.
- path: /books
method: DELETE
status: [200, 404] # The index may not exist, so we accept 404 as a valid response. Default to [200, 201] if not specified.
chapters:
- synopsis: Create an index named `books` with mappings and settings.
path: /{index} # The test will fail if "PUT /{index}" operation is not found in the spec.
method: PUT
parameters: # All parameters are validated against their schemas in the spec
index: books
request_body: # The request body is validated against the schema of the requestBody in the spec
payload:
mappings:
properties:
name:
type: keyword
age:
type: integer
settings:
number_of_shards: 5
number_of_replicas: 2
response: # The response body is validated against the schema of the corresponding response in the spec
status: 200 # This is the expected status code of the response. Any other status code will fail the test.
- synopsis: Retrieve the mappings and settings of the `books` index.
path: /{index}
method: GET
parameters:
index: books
flat_settings: true
- synopsis: Delete the `books` index.
path: /{index}
method: DELETE
parameters:
index: booksConsider the following chapters in ml/model_groups test story:
- synopsis: Create model group.
id: create_model_group # Only needed if you want to refer to this chapter in another chapter.
path: /_plugins/_ml/model_groups/_register
method: POST
request_body:
payload:
name: "NLP_Group"
description: "Model group for NLP models"
response:
status: 200
output: # Save the model group id for later use.
test_model_group_id: "payload.model_group_id"
- synopsis: Query model group.
path: /_plugins/_ml/model_groups/{model_group_id}
method: GET
parameters:
# Use the output from the `create_model_group` chapter.
model_group_id: ${create_model_group.test_model_group_id}
response:
status: 200
- synopsis: Delete model group.
path: /_plugins/_ml/model_groups/{model_group_id}
method: DELETE
parameters:
# Use the output from the `create_model_group` chapter.
model_group_id: ${create_model_group.test_model_group_id}
response:
status: 200As you can see, the output field in the first chapter saves the model_group_id from the response body. This value is then used in the subsequent chapters to query and delete the model group.
It's common to add a feature to the next version of OpenSearch. When adding a new API in the spec, make sure to specify x-version-added, x-version-deprecated or x-version-removed. Finally, specify a semver range in your test stories or chapters as follows.
- synopsis: Search with `phase_took` added in OpenSearch 2.12.
version: '>= 2.12'
path: /{index}/_search
parameters:
index: movies
cancel_after_time_interval: 10s
method: POST
response:
status: 200The integration test workflow runs a matrix of OpenSearch versions, including the next version. Please check whether the workflow needs an update when adding version-specific tests.
Some APIs behave asynchronously and may require a test to wait for a task to complete. This can be achived with a combination of payload and retry.
For example, an ML task returns CREATED when created, and COMPLETED when it's done. The example below will retry 3 times with an interval of 30 seconds until the task is complete. The default wait time is 1s.
- synopsis: Wait for task.
path: /_plugins/_ml/tasks/{task_id}
method: GET
parameters:
task_id: ${create_model.task_id}
response:
status: 200
payload:
state: COMPLETED
retry:
count: 3
wait: 30000The test runner expects all tests in the same file to be variation of the same path in order to keep tests well-organized. Otherwise, a warning will be emitted.
WARNING Multiple paths detected, please group similar tests together and move paths not being tested to prologues or epilogues.
/_component_template/{name}
/_index_template/{name}
/{index}
The test runner may generate warnings that can be suppressed with warnings:. For example, to suppress the multiple paths detected warning.
- synopsis: Create an index.
method: PUT
path: /{index}
parameters:
index: movies
- synopsis: Search the index to make sure it has been created.
method: POST
warnings:
multiple-paths-detected: false
path: /{index}/_search
parameters:
index: movies