Skip to content

v0.1.1: DLQ, Partitioning Strategies

Pre-release
Pre-release
Compare
Choose a tag to compare
@OpenSrcerer OpenSrcerer released this 01 Oct 12:42
· 23 commits to master since this release

Changelog

  1. Fixed ambiguous log annotations.
    Previously there were logs that began with ambiguous names such as [stream->error]. What is a stream in this case? Replaced all names to more intuitive terms, such as handler, client, etc.
  2. Add DLQ handling for failed Kinesis messages.
    The AWS Kinesis PutRecords API (which M2K uses to batch records) returns an item which lists all successful/failed puts. Failed puts are now put in an in-memory DLQ for another retry (to handle hot shard situations), and then discarded if they fail (due to the message being too large, etc.).
  3. Add the possibility for a partitioning strategy to be supplied by the user.
    The partition key is a very important part when working with Kinesis Data Streams that have multiple shards. This helps prevent the "hot sharding" issue by spreading the load when putting records, as each shard can receive a limited number of messages. In this release M2K allows the user to choose between 3 strategies to tackle this problem:
  • Payload Hash (an MD5 hash is computed from the whole payload and used as the partition key).
  • MQTT Topic (the MQTT topic is used as a partition key).
  • JSON Key (if the messages contain a JSON payload, a JSON Pointer Expression can be supplied by the user to extract that value and use it as a partition key).