You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently Table::read_incremental_records() expects start and end timestamps to be in the Hudi timeline format, i.e., yyyyMMddHHmmSSSSS or the legacy yyyyMMddHHmmSS. We should support parsing more timestamp formats.
Sames goes for Table::read_snapshot_as_of() and other *as_of() APIs.
Expected behavior
To support parsing strings in the form of epoch time (including seconds, milliseconds, microseconds, nanoseconds) and convert to Hudi timeline format for further processing.
To support parsing strings in the form of ISO8601 format like
2019-01-23T12:34:56.123456789+00:00
2019-01-23T12:34:56.123456+00:00
2019-01-23T12:34:56.123+00:00
2019-01-23T12:34:56+00:00
2019-01-23T12:34:56Z and other precisions like above
2019-01-23
Additional context
The conversion to Hudi timeline format should consider the hudi table's timeline timezone config.
The text was updated successfully, but these errors were encountered:
xushiyan
changed the title
Incremental API to support more timestamp formats
Time-travel & Incremental API to support more timestamp formats
Feb 1, 2025
Description of the improvement
Currently
Table::read_incremental_records()
expects start and end timestamps to be in the Hudi timeline format, i.e.,yyyyMMddHHmmSSSSS
or the legacyyyyyMMddHHmmSS
. We should support parsing more timestamp formats.Sames goes for
Table::read_snapshot_as_of()
and other*as_of()
APIs.Expected behavior
To support parsing strings in the form of epoch time (including seconds, milliseconds, microseconds, nanoseconds) and convert to Hudi timeline format for further processing.
To support parsing strings in the form of ISO8601 format like
2019-01-23T12:34:56.123456789+00:00
2019-01-23T12:34:56.123456+00:00
2019-01-23T12:34:56.123+00:00
2019-01-23T12:34:56+00:00
2019-01-23T12:34:56Z
and other precisions like above2019-01-23
Additional context
The conversion to Hudi timeline format should consider the hudi table's timeline timezone config.
The text was updated successfully, but these errors were encountered: