[HUDI-8844] Support secondary and expression index creation through async indexer and write configs #12653

codope · 2025-01-16T18:20:02Z

Change Logs

Added new write configs for secondary index (SI) and expression index (EI) conformant to sql syntax for create index.
Use those configs to check if we need to update new SI/EI definition in index definition file.
Initialize SI/EI if a new one is being created through write configs.
Add asyn indexer and spark datasource to validate index gets built successfully.

Impact

Support secondary and expression index creation through async indexer and write configs.

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

lokeshj1703

Thanks @codope for working on this! Please find my comments below.
Can we also add tests for creation of expression index with write configs?
Couple of cases we can cover:-

Creation of two or more expression index using write configs.
Further upserts without removing the expression index configs should not recreate the index
Failure scenarios with incomplete configs

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

...park-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala

hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieIndexer.java

hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieIndexer.java

hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java

lokeshj1703 · 2025-01-20T11:38:27Z

@codope We would also need to look into indempotency of the index creation using dataframe.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

nsivabalan · 2025-01-21T16:46:51Z

...ent-common/src/main/java/org/apache/hudi/table/action/index/ScheduleIndexActionExecutor.java

@@ -133,6 +133,7 @@ public Option<HoodieIndexPlan> execute() {

  private HoodieIndexPartitionInfo buildIndexPartitionInfo(MetadataPartitionType partitionType, HoodieInstant indexUptoInstant) {
    // for expression index, we need to pass the index name as the partition name
+    // TODO: see the index partition info is built correctly. Should we register the index here?


can we remove the comment then?

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java

...park-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala

codope · 2025-01-23T17:58:13Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

+   * For the second time, the index definition file will be updated if exists.
+   * Table Config is updated if necessary.
+   */
+  public static void register(HoodieTableMetaClient metaClient, HoodieIndexDefinition indexDefinition) {


note to reviewer: this is not new logic. Just moved the util method from HoodieSparkIndexClient to this util class.

codope · 2025-01-23T17:58:34Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

+    }
+  }
+
+  public static HoodieIndexDefinition getSecondaryOrExpressionIndexDefinition(HoodieTableMetaClient metaClient, String userIndexName, String indexType, Map<String, Map<String, String>> columns,


note to reviewer: this is not new logic. Just moved the util method from HoodieSparkIndexClient to this util class.

codope · 2025-01-23T17:58:43Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

+    return metaClient.getTableConfig().getMetadataPartitions().stream().anyMatch(partition -> partition.equals(indexName));
+  }
+
+  private static boolean isEligibleForSecondaryOrExpressionIndex(HoodieTableMetaClient metaClient, String indexType, Map<String, String> options, Map<String, Map<String, String>> columns) throws Exception {


note to reviewer: this is not new logic. Just moved the util method from HoodieSparkIndexClient to this util class.

nsivabalan

few minor comments

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkIndexClient.java

hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java

Add SI detection

codope · 2025-01-25T08:38:26Z

@hudi-bot run azure

hudi-bot · 2025-01-25T12:17:04Z

CI report:

de3579b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Jan 16, 2025

lokeshj1703 reviewed Jan 17, 2025

View reviewed changes

codope force-pushed the hudi-8844-si-indexer branch from 8021eba to 7baa38c Compare January 20, 2025 11:30

nsivabalan reviewed Jan 21, 2025

View reviewed changes

codope force-pushed the hudi-8844-si-indexer branch from 7baa38c to 4864f44 Compare January 23, 2025 17:52

codope commented Jan 23, 2025

View reviewed changes

codope force-pushed the hudi-8844-si-indexer branch from 4864f44 to af9db02 Compare January 23, 2025 18:20

nsivabalan reviewed Jan 24, 2025

View reviewed changes

codope force-pushed the hudi-8844-si-indexer branch from 4118634 to d1caff0 Compare January 24, 2025 17:46

nsivabalan approved these changes Jan 24, 2025

View reviewed changes

codope force-pushed the hudi-8844-si-indexer branch from d1caff0 to 137a2d1 Compare January 25, 2025 01:28

codope added 7 commits January 25, 2025 09:52

Refactor to reuse index registration

508767c

Add SI detection

fix SI configs integration and add some tests

2a37f7f

integrate EI configs and add a test

288dc1e

address review comments and fix tests

93c6efc

Address review comments, add test for util method

da6a987

address review comments

42bc3f7

fix indexer tests

3c71355

codope force-pushed the hudi-8844-si-indexer branch from 137a2d1 to 3c71355 Compare January 25, 2025 07:40

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Jan 25, 2025

fix checkstyle

281fd00

apache deleted a comment from hudi-bot Jan 25, 2025

reuse existing configs

de3579b

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels Jan 25, 2025

codope merged commit 739b79a into apache:master Jan 25, 2025
43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-8844] Support secondary and expression index creation through async indexer and write configs #12653

[HUDI-8844] Support secondary and expression index creation through async indexer and write configs #12653

codope commented Jan 16, 2025

lokeshj1703 left a comment •

edited

Loading

lokeshj1703 commented Jan 20, 2025

nsivabalan Jan 21, 2025

codope Jan 23, 2025

codope Jan 23, 2025

codope Jan 23, 2025

nsivabalan left a comment

codope commented Jan 25, 2025

hudi-bot commented Jan 25, 2025

[HUDI-8844] Support secondary and expression index creation through async indexer and write configs #12653

[HUDI-8844] Support secondary and expression index creation through async indexer and write configs #12653

Conversation

codope commented Jan 16, 2025

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

lokeshj1703 left a comment • edited Loading

Choose a reason for hiding this comment

lokeshj1703 commented Jan 20, 2025

nsivabalan Jan 21, 2025

Choose a reason for hiding this comment

codope Jan 23, 2025

Choose a reason for hiding this comment

codope Jan 23, 2025

Choose a reason for hiding this comment

codope Jan 23, 2025

Choose a reason for hiding this comment

nsivabalan left a comment

Choose a reason for hiding this comment

codope commented Jan 25, 2025

hudi-bot commented Jan 25, 2025

CI report:

lokeshj1703 left a comment •

edited

Loading