feat(schema): Phase 18 - HoodieAvroUtils removal (Part 3) #17659

voonhous · 2025-12-21T07:12:21Z

Describe the issue this Pull Request addresses

Reference issue: #14283

Remove methods that were migrated to HoodieSchemaUtils, consolidate remaining Avro-specific utilities, update documentation. The scope here only covers removal of Avro.Schema from TableSchemaResolver.

NOTE: Merge this after #17656

Key Changes:

Changing method signatures of TableSchemaResolver to HoodieSchema and removing deprecated unused methods.

Summary and Changelog

Swap out Avro.Schema to HoodieSchema for all methods in TableSchemaResolver
Cleanup tests

Impact

None

Risk Level

Low

Documentation Update

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

voonhous · 2025-12-26T09:57:19Z

@the-other-tim-brown I have changed the base to #17656 so that it's easier to review like you suggested :)

It's ready for review!

the-other-tim-brown · 2026-01-08T02:39:25Z

hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java

          .getFields()
          .stream()
-          .map(f -> new FieldSchema(f.name(), f.schema().getType().getName(), f.doc()))
+          .map(f -> new FieldSchema(f.name(), f.schema().getType().toAvroType().getName(), f.doc()))


Can we get the name without converting to avro type?

Yeap, will remove.

the-other-tim-brown · 2026-01-08T02:55:13Z

hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java

            ? getTableSchemaFromCommitMetadata(instantOpt.get(), includeMetadataFields)
            : getTableSchemaFromLatestCommitMetadata(includeMetadataFields))
            .or(() ->
                metaClient.getTableConfig().getTableCreateSchema()


how large of a change is it to modify the getTableCreateSchema to return HoodieSchema?

Shouldn't be too big, doing it now.

the-other-tim-brown · 2026-01-08T02:56:10Z

hudi-common/src/main/java/org/apache/hudi/index/secondary/SecondaryIndexManager.java


    for (String col : colNames) {
-      if (avroSchema.getField(col) == null) {
+      if (schema.getField(col) == null) {


Update this to isPresent instead of null check

the-other-tim-brown · 2026-01-08T02:57:53Z

hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestTableSchemaResolver.java

  }

-  @Test
-  void testGetTableSchema() throws Exception {


Is this now covered by other tests?

This is a compatibility test comparing between the types that return Avro.Schema and HoodieSchema, ensuring that they return the equivalent results.

Since we are changing all methods to return HoodieSchema, there is no Avro.Schema to compare against, hence test is removed as there is nothing to compare against.

the-other-tim-brown · 2026-01-08T02:59:33Z

...ource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriterWithTestFormat.scala

  private def fetchActualSchema(): HoodieSchema = {
    val tableMetaClient = createMetaClient(spark, tempBasePath)
-    new TableSchemaResolver(tableMetaClient).getTableSchema()
+    new TableSchemaResolver(tableMetaClient).getTableSchema(false)


This doesn't need to change right?

Nope, i was debugging, let me revert.

the-other-tim-brown · 2026-01-08T03:00:07Z

hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java

          .getFields()
          .stream()
-          .map(f -> new FieldSchema(f.name(), f.schema().getType().getName(), f.doc()))
+          .map(f -> new FieldSchema(f.name(), f.schema().getType().toAvroType().getName(), f.doc()))


Can we get the name without converting to avro type?

Yeap, removing.

…Part 3)

hudi-bot · 2026-01-08T13:45:27Z

CI report:

6326109 UNKNOWN
d360fde UNKNOWN
56f15a5 UNKNOWN
abf94da UNKNOWN
57f86f3 UNKNOWN
314210e Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

the-other-tim-brown · 2026-01-08T14:20:25Z

hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java

          .getFields()
          .stream()
-          .map(f -> new FieldSchema(f.name(), f.schema().getType().getName(), f.doc()))
+          .map(f -> new FieldSchema(f.name(), f.schema().getType().name(), f.doc()))


For fields that are logical types in avro, we will likely be returning their underlying data type. What impact will that have on the sync now that we would return the logical type name?

the-other-tim-brown

@voonhous, apologies but I think we may need to keep the toAvro step for field type names to keep compatibility between releases for the meta syncs

the-other-tim-brown · 2026-01-08T14:20:40Z

hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java

+   * Gets full schema (user + metadata) for a hoodie table.
   *
-   * @return HoodieSchema for this table
+   * @return Avro schema for this table


Update comment

the-other-tim-brown · 2026-01-08T14:23:16Z

hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java

          .getFields()
          .stream()
-          .map(f -> new FieldSchema(f.name(), f.schema().getType().getName(), f.doc()))
+          .map(f -> new FieldSchema(f.name(), f.schema().getType().name(), f.doc()))


Same question here

github-actions bot added the size:XL PR with lines of changes > 1000 label Dec 21, 2025

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch 3 times, most recently from d39bdd2 to c393e06 Compare December 26, 2025 09:54

voonhous changed the base branch from master to phase-18-HoodieAvroUtils-removal-p3 December 26, 2025 09:55

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels Dec 26, 2025

voonhous requested a review from the-other-tim-brown December 26, 2025 09:57

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch 2 times, most recently from 0325df7 to 4b72160 Compare December 27, 2025 08:57

voonhous mentioned this pull request Dec 27, 2025

feat(schema): phase 18 - Remove HoodieAvroUtils usage (Part 4) #17726

Closed

3 tasks

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p3 branch from c393e06 to 2749bf0 Compare December 29, 2025 19:40

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Dec 29, 2025

voonhous changed the base branch from phase-18-HoodieAvroUtils-removal-p3 to master January 3, 2026 07:55

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch from 4b72160 to a2a1e0e Compare January 3, 2026 18:31

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels Jan 3, 2026

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch 4 times, most recently from 56f15a5 to a6a47c6 Compare January 3, 2026 18:58

voonhous linked an issue Jan 3, 2026 that may be closed by this pull request

Phase 18: HoodieAvroUtils Method Removal #14283

Closed

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch 3 times, most recently from 57f86f3 to 9ebc86c Compare January 8, 2026 02:48

voonhous mentioned this pull request Jan 8, 2026

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 4) #17801

Open

3 tasks

the-other-tim-brown reviewed Jan 8, 2026

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch from 9ebc86c to 2c7bd01 Compare January 8, 2026 09:06

feat(schema): Phase 18 - Remove Avro.Schema from TableSchemaResolver(…

314210e

…Part 3)

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p2 branch from 2c7bd01 to 314210e Compare January 8, 2026 13:18

the-other-tim-brown reviewed Jan 8, 2026

View reviewed changes

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 3) #17659

Are you sure you want to change the base?

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 3) #17659

Conversation

voonhous commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

voonhous commented Dec 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Jan 8, 2026

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the-other-tim-brown left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

voonhous commented Dec 21, 2025 •

edited

Loading