Add auto follow pattern challenge #734
Conversation
| "iterations": 1 | ||
| }, | ||
| { | ||
| "name": "shard-recovery-small_2", |
There was a problem hiding this comment.
nit: here you use "_2" (underscore) while below you use "-2" (dash)
elastic/logs/track.json
Outdated
| @@ -62,10 +62,13 @@ | |||
|
|
|||
| {% set p_recovery_target = (recovery_target | default('logs-k8-application.log-default') ) %} | |||
| {% set p_recovery_from_seq_no = (recovery_seq_no | default(0) ) %} | |||
| {% set p_recovery_from_seq_no_2 = (recovery_seq_no_2 | default(1000000) ) %} | |||
| {% set p_recovery_from_seq_no_3 = (recovery_seq_no_3 | default(2000000) ) %} | |||
| {% set p_recovery_from_seq_no_4 = (recovery_seq_no_4 | default(3000000) ) %} | |||
There was a problem hiding this comment.
Probably this is way too high. AFAIK there are only 1_230_000 docs in k8s logs.
|
LGTM apart a nit. |
Instead of invoking shard change api 30 times with same parameters. Do 4 seperate invocations with different from_seqno and no warmup.
martijnvg
force-pushed
the
elastic_logs_synthetic_source_recovery_more_improvement
branch
from
89a2a80 to
1775eef
Compare
|
LGTM 2. |
martijnvg
changed the title
| {% import "rally.helpers" as rally %} | ||
| {% set p_follow_index_pattern = "copy-{{leader_index}}" %} | ||
| { | ||
| "name": "auto-follow-pattern", |
There was a problem hiding this comment.
Maybe we can use a better name for the challenge...something that reminds what this does? Something like "ccr-shard-recovery". I guess this track is not only useful for synthetic source... but probably it would provide useful info in Serverless too where we probably retrieve data from S3?
There was a problem hiding this comment.
Maybe we can use a better name for the challenge...something that reminds what this does?
+1 - I will add ccr replication to the name and add a description.
. I guess this track is not only useful for synthetic source... but probably it would provide useful info in Serverless too where we probably retrieve data from S3?
IIRC CCR currently only works in stateful. To keep the follower indices up to date it relies on operation based replication, which doesn't exist in serverless.
|
LGTM |
Add a new challenge that uses auto follow pattern to auto follow data streams with
logs-*pattern into the local cluster. This allows to benchmark CCR in a semi realistic way while the data set gets indexed, without running multiple clusters.