v0.1.7: Infinite Retries

Changelog

1. After some testing, it was found that the single retry policy wasn't appropriate.
   There were cases where the shard limit was exceeded and messages were getting dropped. Hence, all messages that fail (even in the retries) are added to the DLQ.

v0.1.6: Empty Error Codes

Changelog

There was a minor issue with the previous release. Although messages were being sent successfully to the Kinesis data stream, the M2K was throwing an exception because it expected the serialization of the "errorCode" field to be not null. This issue was fixed.

v0.1.5: Minimize Dependency Overhead

Changelog

The previous release had a big issue: The uncompressed docker image size was 951MB which is huge for an app of this calibre. In order to save space two strategies were implemented:

1. A dependency only for AWS Kinesis was used instead of the whole client.
   This had quite large savings (nearly 500MB!), as seen on the image below.
2. Only selected JRE modules were used in the final docker image, using jdeps && jlink.
   Thanks to this gist for the example.

v0.1.4: Addition of Tests!

Changelog

1. Tests have been added that verify the functions of the application, yielding better reliability.
   MockK & sometimes Mockito are being used to provide this functionality.
2. Kover was added to measure coverage in the app.

v0.1.3: Docker Image Optimizations 2

Changelog

Ditto as previous release but I removed some leftovers I missed on the Dockerfile.

v0.1.2: Docker Image Optimizations

Changelog

1. Overly large docker image size.
   The previous release was using a single-stage docker build, and shipped with an image optimized for building, and not running. Now M2K ships with openjdk:21-ea-17-slim and as such should shave a lot of image weight.

v0.1.1: DLQ, Partitioning Strategies

Changelog

1. Fixed ambiguous log annotations.
   Previously there were logs that began with ambiguous names such as [stream->error]. What is a stream in this case? Replaced all names to more intuitive terms, such as handler, client, etc.
2. Add DLQ handling for failed Kinesis messages.
   The AWS Kinesis PutRecords API (which M2K uses to batch records) returns an item which lists all successful/failed puts. Failed puts are now put in an in-memory DLQ for another retry (to handle hot shard situations), and then discarded if they fail (due to the message being too large, etc.).
3. Add the possibility for a partitioning strategy to be supplied by the user.
   The partition key is a very important part when working with Kinesis Data Streams that have multiple shards. This helps prevent the "hot sharding" issue by spreading the load when putting records, as each shard can receive a limited number of messages. In this release M2K allows the user to choose between 3 strategies to tackle this problem:

• Payload Hash (an MD5 hash is computed from the whole payload and used as the partition key).
• MQTT Topic (the MQTT topic is used as a partition key).
• JSON Key (if the messages contain a JSON payload, a JSON Pointer Expression can be supplied by the user to extract that value and use it as a partition key).

v0.1.0: First Version

This includes the first version of the application. Barely tested - run at your own risk!