Add index_sorting, synthetic_source_mode and force_merge_max_num_segments track parameters to elastic/logs track

[martijnvg]

This allows to enable indexing sorting, that either sorts by hostname and timestamp or
timestamp and hostname.

Additionally this change adds force_merge_max_num_segments and synthetic_source parameters.
The former controls the number of segments after indexing and the latter whether synthetic source is enabled.

All newly added track params are unset by default, meaning index sorting, synthetic source and force merging to N segments are not enabled.

[jpountz]

Nit: it's a bit awkward that index sorting enables synthetic source, is there any downside with configuring it with another parameter?

[martijnvg]

There is no downside. But I did this with the index.mode=logs in mind. I think this would both enable index sorting and synthetic source. But I can add a synthetic source parameter for now, which would give us more benchmarking flexibility.

[jpountz]

Hopefully one day this will work automagically when synthetic source is enabled.

[martijnvg]

Yes, we should just fix this. For the match_only_text this is working as expected. If synthetic source is enabled then store defaults to true.

[jpountz]

store: true seems missing on more text fields, e.g. kafka.log.trace.message?

[martijnvg]

store: true seems missing on more text fields, e.g. kafka.log.trace.message?

That field is also covered by this: https://github.com/elastic/rally-tracks/pull/522/files#diff-6da3dab0371882c35be61a7d2fffa07d9b9c39351ebda2987e953459eb104104R161
So I think it is good?

[jpountz]

OK, cool. I was surprised that we only had to set store: true in a couple places given the total number of text fields, but it looks like they are all covered by being sub fields of another field. 👍

[martijnvg]

but it looks like they are all covered by being sub fields of another field.

Yes, most text fields are defined as sub fields. So no need to store these text fields.

[martijnvg]

@pmpailis why was this pr closed? I intend to merge this in after I have completed my investigation.

[pmpailis]

Hey @martijnvg ! No idea how this happened to be honest. Just merged my PR and at the same time a number of other open PRs were closed :/ Please reopen this one and apologies for the trouble. (I think that it'll close instantly again if I do it)

[martijnvg]

@pmpailis No worries. Something strange is going on this PR auto closed again and so did #500.

[gareth-ellis]

@elasticmachine update branch

[gareth-ellis]

LGTM. as I understand it the default run mode isnt being changed. Do you have any plans to update existing nightly benchmark with these new parameters, or are they going to be kept as is? (thus keeping currently nightlies comparable)

[martijnvg]

Do you have any plans to update existing nightly benchmark with these new parameters, or are they going to be kept as is?

No, for now we plan to keep the existing nightly benchmark as is. The changes added are initially be used for one of benchmark runs for specific research around storage optimisations for logs. In the future we may make changes that will also affect the nightly benchmark, however for now this isn't the case.