[HUDI-8940] Fix Bloom Index Partitioner to distribute keys uniformly across partitions

[vamsikarnika]

Change Logs

Bloom Index causes data skew when bucketized partitioning is used, during repartition and sorting stage when there are hollow buckets created. This happens when there are lot of writes to one partition and few writes to other partitions.

In this pr, we're partitioning based on the fileId + recordKey sort partitioner which distributes the keys equally while keeping same fileIds together.

Impact

Bloom Index can now use fileId and recordkey based partitioner to distribute the comparisons equally across partitions where bucketized partitioning is causing data skew.

Risk level (write none, low medium or high below)

Medium

Added Functional Tests

Documentation Update

public static final ConfigProperty<String> BLOOM_INDEX_FILE_GROUP_ID_KEY_SORT_PARTITIONER = ConfigProperty
      .key("hoodie.bloom.index.fileId.key.sort.partitioner")
      .defaultValue("false")
      .markAdvanced()
      .withDocumentation("Only applies if index type is BLOOM. "
          + "When true, fileId and key sort based partitioning is enabled "
          + "This reduces skew seen in bucket based bloom index lookup");

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 666815b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build