Streams - keep track of first offset and timestamp per segment

[viktorerlingsson]

WHAT is this pull request doing?

Saves a hash with the first offset for each segment, greatly reducing the time (and memory used) to find a message by offset.
Does the same for timestamps.

Tests done by getting a message by a random offset 10 times in a stream with 10M messages.

main:

Getting 10 messages by random offsets...
Average time: 660.8 milliseconds
Maximum resident set size (kbytes): 824424

this branch:

Getting 10 messages by random offsets...
Average time: 52.9 milliseconds
Maximum resident set size (kbytes): 23692

HOW can this pull request be tested?

Existing specs in stream_queue_spec.cr should cover this pretty well.

[spuun]

Cool!
Maybe a sorted array could perform even better since that would enable binary search. Store tuples of {UInt32, Int64}.
Which makes me wonder... Will the keys in the Hash always be sorted? I.e. are they always inserted sorted?

[viktorerlingsson]

Maybe a sorted array could perform even better since that would enable binary search. Store tuples of {UInt32, Int64}. Which makes me wonder... Will the keys in the Hash always be sorted? I.e. are they always inserted sorted?

Yeah, probably! I don't think it will make a huge difference, since the number of segments won't be super big. But can give it a try 👍

Yes, it should always be sorted. We only add larger offsets and delete the smallest.

[carlhoerberg]

Good feature!

Could theoretically build an index over timestamps too, right?

[viktorerlingsson]

Good feature!

Could theoretically build an index over timestamps too, right?

Yes, should be possible to do pretty much the same way for timestamps I think. I'll give it a try!