Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add TableBuilder API for creating Table instances #163

Merged
merged 31 commits into from
Oct 14, 2024

Conversation

kazdy
Copy link
Contributor

@kazdy kazdy commented Oct 11, 2024

Description

Add TableBuilder to handle Table construction.

Allow to explicitly set hudi_options and storage_options via builder in rust or via python bindings when creating new Table instance.

let hudi_table = TableBuilder::from_base_uri("/tmp/hudi_table")
  .with_hudi_options([("hoodie.read.as.of.timestamp", "0")])
  .with_storage_options([("AWS_REGION", "us-east-1")])
  .build()

similarly for python

hudi_table = HudiTableBuilder.from_base_uri(
  "/tmp/hudi_table"
  ).with_hudi_options({
    "hoodie.read.as.of.timestamp": "0"
  }).with_storage_options({
    "AWS_REGION": "us-east-1"
  }).build()

resolves: #139

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

@kazdy
Copy link
Contributor Author

kazdy commented Oct 12, 2024

@xushiyan please let me know if this is a direction you're fine with

Copy link
Member

@xushiyan xushiyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are some refactoring around this. can you check out #161 first, and adjust this PR accordingly please?

crates/core/src/table/mod.rs Show resolved Hide resolved
crates/core/src/table/builder.rs Show resolved Hide resolved
python/hudi/_internal.pyi Outdated Show resolved Hide resolved
python/src/internal.rs Outdated Show resolved Hide resolved
python/tests/test_table_read.py Show resolved Hide resolved
@xushiyan
Copy link
Member

@kazdy i'm cutting release branch within 1 day. do you think you can update this soon?

@xushiyan xushiyan added this to the release-0.2.0 milestone Oct 12, 2024
@kazdy
Copy link
Contributor Author

kazdy commented Oct 12, 2024

@kazdy i'm cutting release branch within 1 day. do you think you can update this soon?

yes, I'll give it a try

@xushiyan xushiyan changed the title introduce TableBuilder to handle storage_options explicitly feat: add TableBuilder to handle storage_options explicitly Oct 12, 2024
@xushiyan xushiyan added python Related to Python codebase rust Related to Rust codebase p1 labels Oct 12, 2024
@xushiyan
Copy link
Member

@kazdy I'll push some fixes and then get this landed for release

Copy link

codecov bot commented Oct 13, 2024

Codecov Report

Attention: Patch coverage is 90.08264% with 12 lines in your changes missing coverage. Please review.

Project coverage is 90.35%. Comparing base (d5f42f6) to head (48821ab).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/table/builder.rs 88.76% 10 Missing ⚠️
crates/core/src/table/mod.rs 71.42% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #163      +/-   ##
==========================================
+ Coverage   89.81%   90.35%   +0.53%     
==========================================
  Files          17       19       +2     
  Lines         884      933      +49     
==========================================
+ Hits          794      843      +49     
  Misses         90       90              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kazdy kazdy requested a review from xushiyan October 13, 2024 12:27
@kazdy kazdy marked this pull request as ready for review October 13, 2024 12:28
@xushiyan xushiyan changed the title feat: add TableBuilder to handle storage_options explicitly feat: add TableBuilder API for passing options Oct 14, 2024
@xushiyan xushiyan merged commit c8862e4 into apache:main Oct 14, 2024
13 checks passed
@xushiyan xushiyan changed the title feat: add TableBuilder API for passing options feat: add TableBuilder API for creating Table instances Oct 14, 2024
@kazdy kazdy deleted the storage_opsions branch October 19, 2024 19:55
@xushiyan xushiyan mentioned this pull request Nov 22, 2024
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature p1 python Related to Python codebase rust Related to Rust codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: support storage_options param when reading from table
2 participants